Commit graph

5 commits

Author SHA1 Message Date
Joseph Schorr
a20d758642 Add a zero chunk test to cloud storage 2017-06-16 13:38:38 -04:00
Joseph Schorr
5f99448adc Add a chunk cleanup queue for async GC of empty chunks
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
2016-11-15 15:07:41 -05:00
Joseph Schorr
bfe2646a50 Make sure we don't generate chunk sizes larger than 5 GB.
Amazon S3 does not allow for chunk sizes larger than 5 GB; we currently don't handle that case at all, which is why large uploads are failing. This change ensures that if a storage engine specifies a *maximum* chunk size, we write multiple chunks no larger than that size.
2016-10-25 13:57:49 -04:00
Joseph Schorr
06718d1237 Fix S3 tests for IAM 2016-08-15 20:34:17 -04:00
Joseph Schorr
eab6af2b87 Add mocked unit tests for cloud storage engine 2016-03-23 12:13:54 -04:00