1
0
Fork 1
mirror of https://github.com/vbatts/tar-split.git synced 2024-12-18 19:46:29 +00:00
Commit graph

154 commits

Author SHA1 Message Date
Joe Tsai
4ad443d166 archive/tar: expand abilities of TestReader
Motivation:
* There are an increasing number of "one-off" corrupt files added
to make sure that package does not succeed or crash on them.
Instead, allow for the test to specify the error that is expected
to occur (if any).
* Also, fold in the logic to check the MD5 checksum into this
function.

The following tests are being removed:
* TestIncrementalRead: Done by TestReader by using io.CopyBuffer
with a buffer of 8. This achieves the same behavior as this test.
* TestSparseEndToEnd: Since TestReader checks the MD5 checksums
if the input corpus provides them, then this is redundant.
* TestSparseIncrementalRead: Redundant for the same reasons that
TestIncrementalRead is now redundant
* TestNegativeHdrSize: Added to TestReader corpus
* TestIssue10968: Added to TestReader corpus
* TestIssue11169: Added to TestReader corpus

With this change, code coverage did not change: 85.3%

Change-Id: I8550d48657d4dbb8f47dfc3dc280758ef73b47ec
Reviewed-on: https://go-review.googlesource.com/15176
Reviewed-by: Andrew Gerrand <adg@golang.org>
2016-02-02 14:06:30 -05:00
Joe Tsai
f0fc67b3a8 archive/tar: make Reader.Read errors persistent
If the stream is in an inconsistent state, it does not make sense
that Reader.Read can be called and possibly succeed.

Change-Id: I9d1c5a1300b2c2b45232188aa7999e350809dcf2
Reviewed-on: https://go-review.googlesource.com/15177
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2016-02-02 14:06:30 -05:00
Joe Tsai
af15385a0d archive/tar: fix bugs with sparseFileReader
The sparseFileReader is prone to two different forms of
denial-of-service attacks:
* A malicious tar file can cause an infinite loop
* A malicious tar file can cause arbitrary panics

This results because of poor error checking/handling, which this
CL fixes. While we are at it, add a plethora of unit tests to
test for possible malicious inputs.

Change-Id: I2f9446539d189f3c1738a1608b0ad4859c1be929
Reviewed-on: https://go-review.googlesource.com/15115
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Andrew Gerrand <adg@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2016-02-02 14:06:30 -05:00
Joe Tsai
440ba9e519 archive/tar: remove dead code with USTAR path splitting
Convert splitUSTARPath to return a bool rather than an error since
the caller never ever uses the error other than to check if it is
nil. Thus, we can remove errNameTooLong as well.

Also, fold the checking of the length <= fileNameSize and whether
the string is ASCII into the split function itself.

Lastly, remove logic to set the MAGIC since that's already done on
L200. Thus, setting the magic is redundant.

There is no overall logic change.

Updates #12638

Change-Id: I26b6992578199abad723c2a2af7f4fc078af9c17
Reviewed-on: https://go-review.googlesource.com/14723
Reviewed-by: David Symonds <dsymonds@golang.org>
Run-TryBot: David Symonds <dsymonds@golang.org>
2016-02-02 14:06:30 -05:00
d50e5c9283 LICENSE: update LICENSE to BSD 3-clause
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-12-03 15:45:57 -05:00
0de4e9db0c Merge pull request #27 from vbatts/bench_asm
tar/asm: basic benchmark on disasm/asm of testdata
2015-12-02 14:09:21 -06:00
1501fe6002 Merge pull request #22 from tonistiigi/stream-opt
Optimize tar stream generation
2015-12-02 14:09:08 -06:00
19b7e22058 tar/asm: basic benchmark on disasm/asm of testdata
```
PASS
BenchmarkAsm-4         5         238968475 ns/op        66841059 B/op       2449 allocs/op
ok      _/home/vbatts/src/vb/tar-split/tar/asm  2.267s
```

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-12-02 14:36:02 -05:00
026e78012b Merge pull request #26 from vbatts/better_discard_in_test
tar/asm: remove unneeded Tee
2015-12-02 12:00:26 -06:00
2efe34695a tar/asm: remove unneeded Tee
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-12-02 12:56:52 -05:00
Tonis Tiigi
23b6435e6b Optimize tar stream generation
- New writeTo method allows to avoid creating extra pipe.
- Copy with a pooled buffer instead of allocating new buffer for each file.
- Avoid extra object allocations inside the loop.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2015-12-01 14:08:53 -08:00
93666d5824 Merge pull request #25 from vbatts/bench
tar/storage: adding Getter Putter benchmark
2015-12-01 14:37:10 -06:00
11281e8c09 tar/storage: adding Getter Putter benchmark
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-12-01 15:31:48 -05:00
fc1e47e71d Merge pull request #24 from vbatts/drop_go1.2
travis: drop go1.2
2015-12-01 14:31:13 -06:00
d80c6b3bb1 travis: drop go1.2
seems overly reasonable to support go1.3 and greater. :-)

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-12-01 15:26:30 -05:00
Tonis Tiigi
8b20f9161d Optimize JSON decoding
This allows to avoid extra allocations on `ReadBytes` and
decoding buffers.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2015-11-30 09:52:44 -08:00
bece0c7009 demo: docker layer checksums
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-10-16 17:05:18 -04:00
7ea74e1c31 demo: basic command
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-10-16 16:41:09 -04:00
c955161e57 Merge pull request #20 from vbatts/unicode-utf8
remove common in favor of stdlib `unicode/utf8`
2015-09-25 14:38:49 -04:00
10250c25e0 tar/asm: remove useless test
The iso-8859-1 archive is already tested round trip, and this test did
not do anything really.
2015-09-25 14:35:12 -04:00
7e38cefd4b common: remove in favor of stdlib unicode/utf8 2015-09-25 14:33:24 -04:00
7ef16e6f67 Merge pull request #19 from LK4D4/go_143
Update travis to go1.4.3
2015-09-24 16:02:39 -04:00
Alexander Morozov
27876e49c2 Update travis to go1.4.3
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2015-09-24 12:24:31 -07:00
8a361ef0d8 tar/storage: Sprintf is unnecessary
fmt.Sprintf() vs string() for this []byte conversion is too much and
does not provide any further safety.

https://gist.github.com/vbatts/ab17181086aed558dd3a
2015-09-24 09:51:58 -04:00
7f56c08c48 Merge pull request #18 from vbatts/iso-8859-1
Iso 8859 1
2015-09-23 15:47:23 -04:00
cde639172f tar/asm: work with non-utf8 entry names 2015-09-23 15:27:33 -04:00
032efafc29 tar/storage: work with raw (invalid utf8) names
When the entry name is not UTF-8, for example ISO-8859-1, then store the
raw bytes.
To accommodate this, we will have getters and setters for the entry's
name now. Since this most heavily affects the json marshalling, we'll
double check the sanity of the name before storing it in the JSONPacker.
2015-09-23 15:27:33 -04:00
39d06b9dc4 tar/common: get index of first invalid utf-8 char 2015-09-23 15:27:15 -04:00
2865353200 common: add a UTF-8 check helper 2015-09-23 15:27:13 -04:00
7384cf1827 Merge pull request #16 from LK4D4/go_15
Add go 1.5.1 to CI
2015-09-11 12:50:40 -04:00
Alexander Morozov
1148e7ee3b Add go 1.5.1 to CI
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2015-09-11 08:49:52 -07:00
414a687f83 README: usage 2015-09-03 15:01:25 -04:00
b4d27b5426 Merge pull request #15 from vbatts/go1.5
Go1.5 updates for archive/tar
2015-08-20 21:25:41 -07:00
4d4b53c78b archive/tar: don't treat multiple file system links as a tar hardlink
Do not assume that if stat shows multiple links that we should mark the
file as a hardlink in the tar format.  If the hardlink link was not
referenced, this caused a link to "/".  On an overlay file system, all
files have multiple links.

The caller must keep the inode references and set TypeLink, Size = 0,
and LinkName themselves.

Change-Id: I873b8a235bc8f8fbb271db74ee54232da36ca013
Reviewed-on: https://go-review.googlesource.com/13045
Reviewed-by: Ian Lance Taylor <iant@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
Alex Brainman
3b34dbd368 archive/tar: move round-trip reading into common os file
Fixes #11426

Change-Id: I77368b0e852149ed4533e139cc43887508ac7f78
Reviewed-on: https://go-review.googlesource.com/11662
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
Brad Fitzpatrick
27e18409b9 archive/tar: also skip header roundtrip test on nacl
Update #11426

Change-Id: I7abc4ed2241a7a3af6d57c934786f36de4f97b77
Reviewed-on: https://go-review.googlesource.com/11592
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
Brad Fitzpatrick
8eee43d0df archive/tar: disable new failing test on windows and plan9
Update #11426

Change-Id: If406d2efcc81965825a63c76f5448d544ba2a740
Reviewed-on: https://go-review.googlesource.com/11590
Reviewed-by: Austin Clements <austin@google.com>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
b48c28014e archive/tar: fix round-trip attributes
The issue was identified while
working with round trip FileInfo of the headers of hardlinks. Also,
additional test cases for hard link handling.
(review carried over from http://golang.org/cl/165860043)

Fixes #9027

Change-Id: I9e3a724c8de72eb1b0fbe0751a7b488894911b76
Reviewed-on: https://go-review.googlesource.com/6790
Reviewed-by: Russ Cox <rsc@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
Michael Gehring
2e5698249c archive/tar: add missing error checks
Check for errors when reading the headers following the pax headers.

Fixes #11169.

Change-Id: Ifec4a949ec8df8b49fa7cb7a67eb826fe2282ad8
Reviewed-on: https://go-review.googlesource.com/11031
Reviewed-by: Russ Cox <rsc@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
Michael Gehring
69de764807 archive/tar: fix slice bounds out of range
Sanity check the pax-header size field before using it.

Fixes #11167.

Change-Id: I9d5d0210c3990e6fb9434c3fe333be0d507d5962
Reviewed-on: https://go-review.googlesource.com/10954
Reviewed-by: David Symonds <dsymonds@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
Håvard Haugen
55dceefe42 archive/tar: terminate when reading malformed sparse files
Fixes #10968.

Change-Id: I027bc571a71629ac49c2a0ff101b2950af6e7531
Reviewed-on: https://go-review.googlesource.com/10482
Reviewed-by: David Symonds <dsymonds@golang.org>
Run-TryBot: David Symonds <dsymonds@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
Håvard Haugen
576b273762 archive/tar: don't panic on negative file size
Fixes #10959.
Fixes #10960.

Change-Id: I9a81a0e2b8275338d0d1c3f7f7265e0fd91f3de2
Reviewed-on: https://go-review.googlesource.com/10402
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Symonds <dsymonds@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
David du Colombier
6e38573de2 archive/tar: fix error message
Write should return ErrWriteAfterClose instead
of ErrWriteTooLong when called after Close.

Change-Id: If5ec4ef924e4c56489e0d426976f7e5fad79be9b
Reviewed-on: https://go-review.googlesource.com/9259
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2015-08-21 00:15:22 -04:00
bf82db1f0d README: updates 2015-08-18 14:54:32 -04:00
ea4426eee9 Merge pull request #13 from vbatts/travis-go-versions
travis: adding older and newer golang
2015-08-14 10:18:38 -04:00
3a88af2866 travis: adding older and newer golang 2015-08-14 10:15:26 -04:00
4f81319c22 *: adding some version magic 2015-08-14 10:02:46 -04:00
c76e42010e tar/asm: additional GNU LongLink testcase
Adding a minimal test case for GNU @LongLink.
Tested that it fails on v0.9.5, but now passes on v0.9.6 and master.
2015-08-14 07:55:18 -04:00
44d93178df Merge pull request #11 from LK4D4/add_vet
Add vet check to travis
2015-08-13 15:37:30 -04:00
8f81a50860 Merge pull request #10 from LK4D4/fix_pipe_close
asm: Remove unreachable code
2015-08-13 15:36:42 -04:00