1
0
Fork 1
mirror of https://github.com/vbatts/tar-split.git synced 2024-12-19 11:56:30 +00:00
Commit graph

196 commits

Author SHA1 Message Date
38ec4ddb06
Merge pull request #42 from cyphar/chunked-padding
tar: asm: store padding in chunks to avoid memory exhaustion
2017-11-07 10:39:49 -05:00
Aleksa Sarai
99430a8454
tar: asm: add an excess padding test case
To ensure we don't have regressions in our padding fix, add a test case
that attempts to crash the test by creating 20GB of random junk padding.

Signed-off-by: Aleksa Sarai <asarai@suse.de>
2017-11-08 02:35:01 +11:00
Aleksa Sarai
3d9db48dbe
tar: asm: store padding in chunks to avoid memory exhaustion
Previously, we would read the entire padding in a given archive into
memory in order to store it in the packer. This would cause memory
exhaustion if a malicious archive was crafted with very large amounts of
padding. Since a given SegmentType is reconstructed losslessly, we can
simply chunk up any padding into large segments to avoid this problem.
Use a reasonable default of 1MiB to avoid changing the tar-split.json of
existing archives that are not malformed.

Fixes: CVE-2017-14992
Signed-off-by: Aleksa Sarai <asarai@suse.de>
2017-11-08 02:34:56 +11:00
Aleksa Sarai
b9775006bf
*: move tar_benchmark to cmd/tar-split/
This fixes a new go-vet(1) error which has surfaced in Go HEAD.

  $ go vet ./...
  go build github.com/vbatts/tar-split: no non-test Go files in
  /home/travis/gopath/src/github.com/vbatts/tar-split

Signed-off-by: Aleksa Sarai <asarai@suse.de>
2017-11-08 02:20:30 +11:00
ed1c08584d Merge pull request #40 from EdwardBetts/spelling
correct spelling mistake
2017-09-01 06:05:25 -04:00
Edward Betts
22999e8f96 correct spelling mistake 2017-09-01 09:45:09 +01:00
b9127a1393 Merge pull request #38 from vbatts/travis
travis: test more go versions
2017-03-14 11:24:38 -04:00
c6dd42815a
archive/tar: monotonic clock adjustment
commit 0e3355903d2ebcf5ee9e76096f51ac9a116a9dbb upstream

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2017-03-14 11:04:10 -04:00
245403c324
travis: test more go versions
Thanks to @tianon, for pointing to
5e3ef60b0d/lib/travis/build/config.rb (L54-L70)

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2017-03-14 08:38:13 -04:00
7560005f21
README: adding a golang report card
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2017-03-13 18:28:54 -04:00
bd4c5d64c3
main: switch import paths to urfave
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-09-27 02:54:18 +00:00
d3f1b54304
version: bump to v0.10.1
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-09-26 19:53:52 -04:00
f28028292a Merge branch 'master' of github.com:vbatts/tar-split 2016-09-26 19:52:55 -04:00
416fa5dcfe Merge pull request #36 from dmcgowan/fix-extra-nil-accounting
archive/tar: fix writing too many raw bytes
2016-09-26 18:31:47 -04:00
Derek McGowan
6b59e6942e archive/tar: fix writing too many raw bytes
When an EOF is read, only the part of the header buffer which
was read should be accounted for.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2016-09-26 14:01:48 -07:00
7410961e75 tar/asm: failing test for lack of EOF nils
Reported-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-09-26 13:39:03 -07:00
eb3808673d
version: bump to v0.10.0
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-09-23 11:01:58 -04:00
ae8540dc47 Merge pull request #34 from dmcgowan/fix-panic-issue-33
Fix panic in Next
2016-09-23 09:41:12 -04:00
Derek McGowan
e527e70d25 Fix panic in Next
readHeader should never return nil with a tr.err also nil.
To correct this, ensure tr.err never gets reset to nil followed
by a nil return.
2016-09-22 17:38:18 -07:00
6810cedb21 benchmark: add a comparison of 'archive/tar'
Since this project has forked logic of upstream 'archive/tar', this does
a brief comparison including the RawBytes usage.

```bash
$ go test -run="XXX" -bench=.
testing: warning: no tests to run
BenchmarkUpstreamTar-4                      2000            700809 ns/op
BenchmarkOurTarNoAccounting-4               2000            692055 ns/op
BenchmarkOurTarYesAccounting-4              2000            723184 ns/op
PASS
ok      vb/tar-split    4.461s
```

From this, the difference is negligible.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-07-26 09:50:08 -04:00
28bc4c32f9 Merge pull request #32 from vbatts/fix-travis
travis: update golang versions
2016-06-26 15:00:37 -04:00
beaeceb06f travis: update golang versions
This is not saying that tar-split no longer works on go1.3 or go1.4, but
rather that the headache of `go vet` having a version dependent ability
to install it, makes it a headache in travis.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-06-26 14:56:04 -04:00
54e3a92a60 Merge branch 'master' of github.com:vbatts/tar-split 2016-06-26 14:43:38 -04:00
354fd6cf34 cmd: add a disasm --no-stdout flag
Since sometimes you just need to > /dev/null

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-06-26 10:15:12 -04:00
226f7c7490 README: update archive/tar version reference
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-03-30 16:38:51 -04:00
e2a62d6b0d README.md: fix thumbnail
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-02-29 11:40:38 -05:00
24fe0a94fe version: bump to v0.9.13
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-02-15 09:44:28 -05:00
862ccd05bc Merge pull request #31 from vbatts/tar-go1.6
Tar go1.6
2016-02-15 09:41:56 -05:00
c32966b9e8 archive/tar: go1.3 and go1.4 compatibility
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-02-15 09:38:46 -05:00
Joe Tsai
10db8408f6 archive/tar: document how Reader.Read handles header-only files
Commit dd5e14a7511465d20c6e95bf54c9b8f999abbbf6 ensured that no data
could be read for header-only files regardless of what the Header.Size
said. We should document this fact in Reader.Read.

Updates #13647

Change-Id: I4df9a2892bc66b49e0279693d08454bf696cfa31
Reviewed-on: https://go-review.googlesource.com/17913
Reviewed-by: Russ Cox <rsc@golang.org>
2016-02-03 07:01:09 -05:00
Joe Tsai
962540fec3 archive/tar: spell license correctly in example
Change-Id: Ice85d161f026a991953bd63ecc6ec80f8d06dfbd
Reviewed-on: https://go-review.googlesource.com/17901
Run-TryBot: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-03 07:01:09 -05:00
Joe Tsai
a04b4ddba4 archive/tar: properly parse GNU base-256 encoding
Motivation:
* Previous implementation did not detect integer overflow when
parsing a base-256 encoded field.
* Previous implementation did not treat the integer as a two's
complement value as specified by GNU.

The relevant GNU specification says:
<<<
GNU format uses two's-complement base-256 notation to store values
that do not fit into standard ustar range.
>>>

Fixes #12435

Change-Id: I4639bcffac8d12e1cb040b76bd05c9d7bc6c23a8
Reviewed-on: https://go-review.googlesource.com/17424
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-03 07:01:09 -05:00
Joe Tsai
ce5aac17f9 archive/tar: properly format GNU base-256 encoding
Motivation:
* Previous implementation silently failed when an integer overflow
occurred. Now, we report an ErrFieldTooLong.
* Previous implementation did not encode in two's complement format and was
unable to encode negative numbers.

The relevant GNU specification says:
<<<
GNU format uses two's-complement base-256 notation to store values
that do not fit into standard ustar range.
>>>

Fixes #12436

Change-Id: I09c20602eabf8ae3a7e0db35b79440a64bfaf807
Reviewed-on: https://go-review.googlesource.com/17425
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-03 06:58:30 -05:00
Joe Tsai
be9ac88117 archive/tar: convert Reader.Next to be loop based
Motivation for change:
* Recursive logic is hard to follow, since it tends to apply
things in reverse. On the other hand, the tar formats tend to
describe meta headers as affecting the next entry.
* Recursion also applies changes in the wrong order. Two test
files are attached that use multiple headers. The previous Go
behavior differs from what GNU and BSD tar do.

Change-Id: Ic1557256fc1363c5cb26570e5d0b9f65a9e57341
Reviewed-on: https://go-review.googlesource.com/14624
Run-TryBot: Joe Tsai <joetsai@digital-static.net>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-03 06:58:30 -05:00
Joe Tsai
64935a5f0f archive/tar: move parse/format methods to standalone receiver
Motivations for this change:
* It allows these functions to be used outside of Reader/Writer.
* It allows these functions to be more easily unit tested.

Change-Id: Iebe2b70bdb8744371c9ffa87c24316cbbf025b59
Reviewed-on: https://go-review.googlesource.com/15113
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Joe Tsai <joetsai@digital-static.net>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-02 14:32:27 -05:00
Joe Tsai
b598ba3ee7 archive/tar: fix issues with readGNUSparseMap1x0
Motivations:
* Use of strconv.ParseInt does not properly treat integers as 64bit,
preventing this function from working properly on 32bit machines.
* Use of io.ReadFull does not properly detect truncated streams
when the file suddenly ends on a block boundary.
* The function blindly trusts user input for numEntries and allocates
memory accordingly.
* The function does not validate that numEntries is not negative,
allowing a malicious sparse file to cause a panic during make.

In general, this function was overly complicated for what it was
accomplishing and it was hard to reason that it was free from
bounds errors. Instead, it has been rewritten and relies on
bytes.Buffer.ReadString to do the main work. So long as invariants
about the number of '\n' in the buffer are maintained, it is much
easier to see why this approach is correct.

Change-Id: Ibb12c4126c26e0ea460ea063cd17af68e3cf609e
Reviewed-on: https://go-review.googlesource.com/15174
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-02 14:17:35 -05:00
Joe Tsai
7500c932c7 archive/tar: properly handle header-only "files" in Reader
Certain special type-flags, specifically 1, 2, 3, 4, 5, 6,
do not have a data section. Thus, regardless of what the size field
says, we should not attempt to read any data for these special types.

The relevant PAX and USTAR specification says:
<<<
If the typeflag field is set to specify a file to be of type 1 (a link)
or 2 (a symbolic link), the size field shall be specified as zero.
If the typeflag field is set to specify a file of type 5 (directory),
the size field shall be interpreted as described under the definition
of that record type. No data logical records are stored for types 1, 2, or 5.
If the typeflag field is set to 3 (character special file),
4 (block special file), or 6 (FIFO), the meaning of the size field is
unspecified by this volume of POSIX.1-2008, and no data logical records shall
be stored on the medium.
Additionally, for type 6, the size field shall be ignored when reading.
If the typeflag field is set to any other value, the number of logical
records written following the header shall be (size+511)/512, ignoring
any fraction in the result of the division.
>>>

Contrary to the specification, we do not assert that the size field
is zero for type 1 and 2 since we liberally accept non-conforming formats.

Change-Id: I666b601597cb9d7a50caa081813d90ca9cfc52ed
Reviewed-on: https://go-review.googlesource.com/16614
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-02 14:10:38 -05:00
Matt Layher
2424f4e367 archive/tar: make output deterministic
Replaces PID in PaxHeaders with 0.  Sorts PAX header keys before writing
them to the archive.

Fixes #12358

Change-Id: If239f89c85f1c9d9895a253fb06a47ad44960124
Reviewed-on: https://go-review.googlesource.com/13975
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Joe Tsai <joetsai@digital-static.net>
2016-02-02 14:10:11 -05:00
Joe Tsai
bffda594f7 archive/tar: detect truncated files
Motivation:
* Reader.skipUnread never reports io.ErrUnexpectedEOF. This is strange
given that io.ErrUnexpectedEOF is given through Reader.Read if the
user manually reads the file.
* Reader.skipUnread fails to detect truncated files since io.Seeker
is lazy about reporting errors. Thus, the behavior of Reader differs
whether the input io.Reader also satisfies io.Seeker or not.

To solve this, we seek to one before the end of the data section and
always rely on at least one call to io.CopyN. If the tr.r satisfies
io.Seeker, this is guarunteed to never read more than blockSize.

Fixes #12557

Change-Id: I0ddddfc6bed0d74465cb7e7a02b26f1de7a7a279
Reviewed-on: https://go-review.googlesource.com/15175
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-02 14:09:30 -05:00
Joe Tsai
cf83c95de8 archive/tar: fix numeric overflow issues in readGNUSparseMap0x1
Motivation:
* The logic to verify the numEntries can overflow and incorrectly
pass, allowing a malicious file to allocate arbitrary memory.
* The use of strconv.ParseInt does not set the integer precision
to 64bit, causing this code to work incorrectly on 32bit machines.

Change-Id: I1b1571a750a84f2dde97cc329ed04fe2342aaa60
Reviewed-on: https://go-review.googlesource.com/15173
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-02 14:09:04 -05:00
Joe Tsai
cb423795eb archive/tar: add missing error checks to Reader.Next
A recursive call to Reader.Next did not check the error before
trying to use the result, leading to a nil pointer panic.
This specific CL addresses the immediate issue, which is the panic,
but does not solve the root issue, which is due to an integer
overflow in the base-256 parser.

Updates #12435

Change-Id: Ia908671f0f411a409a35e24f2ebf740d46734072
Reviewed-on: https://go-review.googlesource.com/15437
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-02 14:08:38 -05:00
Joe Tsai
4ad443d166 archive/tar: expand abilities of TestReader
Motivation:
* There are an increasing number of "one-off" corrupt files added
to make sure that package does not succeed or crash on them.
Instead, allow for the test to specify the error that is expected
to occur (if any).
* Also, fold in the logic to check the MD5 checksum into this
function.

The following tests are being removed:
* TestIncrementalRead: Done by TestReader by using io.CopyBuffer
with a buffer of 8. This achieves the same behavior as this test.
* TestSparseEndToEnd: Since TestReader checks the MD5 checksums
if the input corpus provides them, then this is redundant.
* TestSparseIncrementalRead: Redundant for the same reasons that
TestIncrementalRead is now redundant
* TestNegativeHdrSize: Added to TestReader corpus
* TestIssue10968: Added to TestReader corpus
* TestIssue11169: Added to TestReader corpus

With this change, code coverage did not change: 85.3%

Change-Id: I8550d48657d4dbb8f47dfc3dc280758ef73b47ec
Reviewed-on: https://go-review.googlesource.com/15176
Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-02-02 14:06:30 -05:00
Joe Tsai
f0fc67b3a8 archive/tar: make Reader.Read errors persistent
If the stream is in an inconsistent state, it does not make sense
that Reader.Read can be called and possibly succeed.

Change-Id: I9d1c5a1300b2c2b45232188aa7999e350809dcf2
Reviewed-on: https://go-review.googlesource.com/15177
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-02 14:06:30 -05:00
Joe Tsai
af15385a0d archive/tar: fix bugs with sparseFileReader
The sparseFileReader is prone to two different forms of
denial-of-service attacks:
* A malicious tar file can cause an infinite loop
* A malicious tar file can cause arbitrary panics

This results because of poor error checking/handling, which this
CL fixes. While we are at it, add a plethora of unit tests to
test for possible malicious inputs.

Change-Id: I2f9446539d189f3c1738a1608b0ad4859c1be929
Reviewed-on: https://go-review.googlesource.com/15115
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Andrew Gerrand <adg@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-02 14:06:30 -05:00
Joe Tsai
440ba9e519 archive/tar: remove dead code with USTAR path splitting
Convert splitUSTARPath to return a bool rather than an error since
the caller never ever uses the error other than to check if it is
nil. Thus, we can remove errNameTooLong as well.

Also, fold the checking of the length <= fileNameSize and whether
the string is ASCII into the split function itself.

Lastly, remove logic to set the MAGIC since that's already done on
L200. Thus, setting the magic is redundant.

There is no overall logic change.

Updates #12638

Change-Id: I26b6992578199abad723c2a2af7f4fc078af9c17
Reviewed-on: https://go-review.googlesource.com/14723
Reviewed-by: David Symonds <dsymonds@golang.org>
Run-TryBot: David Symonds <dsymonds@golang.org>
2016-02-02 14:06:30 -05:00
b87f81631a version: mark 0.9.12 2016-01-31 01:39:10 -05:00
d50e5c9283 LICENSE: update LICENSE to BSD 3-clause
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-12-03 15:45:57 -05:00
0de4e9db0c Merge pull request #27 from vbatts/bench_asm
tar/asm: basic benchmark on disasm/asm of testdata
2015-12-02 14:09:21 -06:00
1501fe6002 Merge pull request #22 from tonistiigi/stream-opt
Optimize tar stream generation
2015-12-02 14:09:08 -06:00
19b7e22058 tar/asm: basic benchmark on disasm/asm of testdata
```
PASS
BenchmarkAsm-4         5         238968475 ns/op        66841059 B/op       2449 allocs/op
ok      _/home/vbatts/src/vb/tar-split/tar/asm  2.267s
```

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-12-02 14:36:02 -05:00