vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9048 commits 2 branches 62 tags 205 MiB
Commit graph

164 commits

Author SHA1 Message Date
Joseph Schorr
ebc57da843 Tiny fixes found in review 2018-11-01 17:45:39 -04:00
Joseph Schorr
6a9634dffb Harden the storage replication worker to failures by explicitly catching certain errors, having better backoff and explicitly ensuring a layer has been replicated to a region before adding the placement into the database 
Also adds tests for the various failure cases
 2018-10-31 14:13:25 -04:00
Joseph Schorr
3a0adfcb11 Fix stream_write to properly raise an exception on failure, instead of just silently failing 
This was causing problems for customers using georeplication over unstable storage engines

Also adds tests for stream_write and copy,  to ensure we detect failure
 2018-10-31 12:46:49 -04:00
Joseph Schorr
b1411d4d52 Make sure empty segments in Swift are deleted on upload cancelation 2018-09-27 15:12:04 -04:00
Joseph Schorr
ba39737c3a Add blob uploader helper class, for easy management of all the state associated with uploading blobs 2018-09-24 12:54:56 -04:00
Joseph Schorr
a4f1475de4
Merge pull request #3234 from quay/joseph.schorr/QUAY-1030/interfacing-part-8 
Change verbs endpoint to use the new registry data model interface
 2018-09-13 12:50:02 -04:00
Kenny Lee Sin Cheong
b6336393de Make IPResolver run the update in a separate thread 
A separate thread will cache the results of parsing the range
file, and the IPResolver will hit the cache instead of blocking while recomputing the
ranges everytime. The thread updates every 600s, and retry every 60s on
failures.
 2018-08-31 14:00:53 -04:00
Joseph Schorr
f252b0b16f Move verbs endpoint to use new registry data model 2018-08-29 16:38:54 -04:00
Brad Ison
d3d9cca182 Upgrade Peewee to latest 3.x 
This requires a number of small changes in the data model code, as well as additional testing.
 2018-07-23 12:43:23 -04:00
Joseph Schorr
be58605f44 Move storage proxy test into the storage directory 2018-07-18 17:33:15 -04:00
Joseph Schorr
4aee811611 Move cloud storage tests to pytest 2018-07-17 16:14:23 -04:00
Joseph Schorr
7426f9e93a Change cloudfront storage to report non-cloudfront (i.e. S3) URLs for all requests missing IP information 
This ensures the security scanner gets S3 URLs

Fixes https://jira.coreos.com/browse/QUAY-954
 2018-07-05 16:16:20 +03:00
Sam Chow
14fefea38f Revert local validation context, extract another util 2018-06-29 15:09:33 -04:00
Joseph Schorr
57523d22de Have Swift storage delete segments when deleting dynamic large objects 
This ensures that we reclaim the space, rather than simply deleting the manifest

Fixes https://jira.coreos.com/browse/QUAY-942
 2018-05-16 16:01:49 -04:00
Joseph Schorr
143ca8653d Fix bugs in Azure storage engine to make it actually work 
Tested against a real Azure account
 2018-02-09 16:46:50 -05:00
Joseph Schorr
5a2f6f6faf Add exception if relative path is given to Azure storage engine 2018-02-06 13:48:40 -05:00
Joseph Schorr
d488517b36 Implement storage driver for Azure Blob Storage 2018-02-06 13:48:40 -05:00
Joseph Schorr
f4f51934ef Linter fixes for cloudfront storage 2018-01-10 13:16:41 -05:00
Joseph Schorr
6cd8140c34 Pad out the segment identifier for Swift segments and change test back to string comparison 
Makes us follow the docs closer
 2017-11-28 09:46:40 +02:00
Joseph Schorr
3bbcb93977 Add additional Swift chunking tests 2017-11-28 09:46:40 +02:00
josephschorr
a6e3686c58
Merge pull request #2891 from coreos-inc/joseph.schorr/QS-38/refactor-storage 
Small storage refactoring
 2017-10-30 17:40:07 -04:00
Joseph Schorr
ffaff5a27e Add caching of get_auth call in Swift 
Should make calls significantly faster for our Swift customers

Fixes https://jira.prod.coreos.systems/browse/QS-39
 2017-10-19 11:34:13 -04:00
Joseph Schorr
8194f5cf72 Switch ipresolver to always be defined in the storage context 
We now use a no-op IP resolver instead of an IF check

Fixes https://jira.prod.coreos.systems/browse/QS-38
 2017-10-17 14:29:40 -04:00
Joseph Schorr
5dfa46ed56 Fix storage indentation 2017-10-17 14:12:14 -04:00
josephschorr
7cfe66336a Merge pull request #2874 from coreos-inc/joseph.schorr/QS-22/swift-copy-to 
Add missing `copy_to` method to Swift
 2017-10-09 17:54:02 -04:00
Joseph Schorr
ab886226f3 Add missing copy_to method to Swift 
Along with tests

Fixes https://coreosdev.atlassian.net/browse/QS-22
 2017-10-09 17:17:36 -04:00
Joseph Schorr
f67e2baeba Fix verbs for recent storage change 
1) Initialize the storage class in verbs with the extra needed args
2) Make the CloudFrontedS3Storage resilient to those extra args being missing
 2017-10-07 00:17:10 -04:00
Joseph Schorr
61f22621d6 Fix unbound variable in new cloud fronted storage 
Fixes https://sentry.io/coreos/backend-staging/issues/363196446/
 2017-10-06 13:54:49 -04:00
Joseph Schorr
96827e2b60 Switch to defaulting to CloudFront 2017-09-28 17:29:00 -04:00
Joseph Schorr
05b4a7d457 Add worker to update ipresolver data files every few hours 2017-09-28 14:40:59 -04:00
Joseph Schorr
010dda2c52 Add CloudFrontedS3Storage, which redirects to CloudFront for non-S3 ips 2017-09-28 14:40:58 -04:00
Joseph Schorr
56fbbcf7cf Add request IP to get_direct_download_url method 2017-09-25 17:14:28 -04:00
Joseph Schorr
cdee387d3f Remove unnecessary bucket validation call in storage 2017-07-26 18:14:53 -04:00
Joseph Schorr
688312bb29 Check the returned ETag in Swift when streaming data 
This ensures that if Swift mis-writes the data, we know immediately and can fail
 2017-06-27 16:21:37 +03:00
Joseph Schorr
6b272cf7e6 Move Swift tests to pytest 2017-06-27 16:05:32 +03:00
Joseph Schorr
9979db0ce3 Make sure to compare connection kwargs in storage copy fast-path 
This ensures that if storage is on different hosts, then we use the slower but correct code path .
 2017-03-20 13:37:28 -04:00
Charlton Austin
3fd8c8a60d feature(app.py): adding queue_metrics to queues 
publishing queue metrics for SRE

[none]
 2017-02-14 16:01:28 -05:00
Joseph Schorr
b407f88a26 Remove unnecessary CloudWatch metrics 
They are spamming the API and costing us a lot of money
 2017-02-01 13:08:21 -05:00
Joseph Schorr
f4bcf68928 Add support for custom ports on RADOS and S3 storage engines 2016-12-01 14:23:18 -05:00
Charlton Austin
e7b029c993 Fixing how we encode some JSON. 2016-11-28 11:37:26 -05:00
Joseph Schorr
5f99448adc Add a chunk cleanup queue for async GC of empty chunks 
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
 2016-11-15 15:07:41 -05:00
Joseph Schorr
59cb6bd216 Make sure to not log exceptions if Swift deletes fail 2016-11-11 14:17:32 -05:00
Joseph Schorr
ce0e3e0e8d Add missing parameter on RADOS storage 
Fixes Python error that results from the missing parameter
 2016-10-31 12:48:05 -04:00
Joseph Schorr
bfe2646a50 Make sure we don't generate chunk sizes larger than 5 GB. 
Amazon S3 does not allow for chunk sizes larger than 5 GB; we currently don't handle that case at all, which is why large uploads are failing. This change ensures that if a storage engine specifies a *maximum* chunk size, we write multiple chunks no larger than that size.
 2016-10-25 13:57:49 -04:00
Joseph Schorr
29b8905051 Add retry and ignore around Swift deleting empty chunks 
Fixes #1979
 2016-10-13 12:53:59 -04:00
Joseph Schorr
38415065e6 Add S3 retry to all possible operations around the multipart upload 
Fixes #1933
 2016-10-04 21:54:23 +03:00
Joseph Schorr
6ae3faf7fc Add explicit config parameter to the JWT auth methods 2016-09-29 11:15:20 +02:00
Joseph Schorr
dd2e086a20 Add feature flag to force all direct download URLs to be proxied 
Fixes #1667
 2016-09-29 11:13:41 +02:00
Joseph Schorr
4f0ddc8d6e Don't exception log for expected 404s in Swift storage 
Fixes #1850
 2016-09-20 14:07:29 -04:00
Joseph Schorr
8c67ed14e8 Delete empty swift chunks 
Fixes #1844
 2016-09-19 11:41:20 -04:00