tar-split/DESIGN.md

Flow of TAR stream

The underlying use of github.com/vbatts/tar-split/archive/tar is most similar to stdlib.

Packer interface

For ease of storage and usage of the raw bytes, there will be a storage interface, that accepts an io.Writer (This way you could pass it an in memory buffer or a file handle).

Having a Packer interface can allow configuration of hash.Hash for file payloads and providing your own io.Writer.

Instead of having a state directory to store all the header information for all Readers, we will leave that up to user of Reader. Because we can not assume an ID for each Reader, and keeping that information differentiated.

State Directory

Perhaps we could deduplicate the header info, by hashing the rawbytes and storing them in a directory tree like:

./ac/dc/beef

Then reference the hash of the header info, in the positional records for the tar stream. Though this could be a future feature, and not required for an initial implementation. Also, this would imply an owned state directory, rather than just writing storage info to an io.Writer.