Joseph Schorr
bb4927470a
cherrypick
2018-11-01 13:39:57 -04:00
Joseph Schorr
143ca8653d
Fix bugs in Azure storage engine to make it actually work
...
Tested against a real Azure account
2018-02-09 16:46:50 -05:00
Joseph Schorr
5a2f6f6faf
Add exception if relative path is given to Azure storage engine
2018-02-06 13:48:40 -05:00
Joseph Schorr
d488517b36
Implement storage driver for Azure Blob Storage
2018-02-06 13:48:40 -05:00
Joseph Schorr
f4f51934ef
Linter fixes for cloudfront storage
2018-01-10 13:16:41 -05:00
Joseph Schorr
6cd8140c34
Pad out the segment identifier for Swift segments and change test back to string comparison
...
Makes us follow the docs closer
2017-11-28 09:46:40 +02:00
Joseph Schorr
3bbcb93977
Add additional Swift chunking tests
2017-11-28 09:46:40 +02:00
josephschorr
a6e3686c58
Merge pull request #2891 from coreos-inc/joseph.schorr/QS-38/refactor-storage
...
Small storage refactoring
2017-10-30 17:40:07 -04:00
Joseph Schorr
ffaff5a27e
Add caching of get_auth call in Swift
...
Should make calls significantly faster for our Swift customers
Fixes https://jira.prod.coreos.systems/browse/QS-39
2017-10-19 11:34:13 -04:00
Joseph Schorr
8194f5cf72
Switch ipresolver to always be defined in the storage context
...
We now use a no-op IP resolver instead of an IF check
Fixes https://jira.prod.coreos.systems/browse/QS-38
2017-10-17 14:29:40 -04:00
Joseph Schorr
5dfa46ed56
Fix storage indentation
2017-10-17 14:12:14 -04:00
josephschorr
7cfe66336a
Merge pull request #2874 from coreos-inc/joseph.schorr/QS-22/swift-copy-to
...
Add missing `copy_to` method to Swift
2017-10-09 17:54:02 -04:00
Joseph Schorr
ab886226f3
Add missing copy_to method to Swift
...
Along with tests
Fixes https://coreosdev.atlassian.net/browse/QS-22
2017-10-09 17:17:36 -04:00
Joseph Schorr
f67e2baeba
Fix verbs for recent storage change
...
1) Initialize the storage class in verbs with the extra needed args
2) Make the CloudFrontedS3Storage resilient to those extra args being missing
2017-10-07 00:17:10 -04:00
Joseph Schorr
61f22621d6
Fix unbound variable in new cloud fronted storage
...
Fixes https://sentry.io/coreos/backend-staging/issues/363196446/
2017-10-06 13:54:49 -04:00
Joseph Schorr
96827e2b60
Switch to defaulting to CloudFront
2017-09-28 17:29:00 -04:00
Joseph Schorr
05b4a7d457
Add worker to update ipresolver data files every few hours
2017-09-28 14:40:59 -04:00
Joseph Schorr
010dda2c52
Add CloudFrontedS3Storage, which redirects to CloudFront for non-S3 ips
2017-09-28 14:40:58 -04:00
Joseph Schorr
56fbbcf7cf
Add request IP to get_direct_download_url method
2017-09-25 17:14:28 -04:00
Joseph Schorr
cdee387d3f
Remove unnecessary bucket validation call in storage
2017-07-26 18:14:53 -04:00
Joseph Schorr
688312bb29
Check the returned ETag in Swift when streaming data
...
This ensures that if Swift mis-writes the data, we know immediately and can fail
2017-06-27 16:21:37 +03:00
Joseph Schorr
6b272cf7e6
Move Swift tests to pytest
2017-06-27 16:05:32 +03:00
Joseph Schorr
9979db0ce3
Make sure to compare connection kwargs in storage copy fast-path
...
This ensures that if storage is on different hosts, then we use the slower but correct code path .
2017-03-20 13:37:28 -04:00
Charlton Austin
3fd8c8a60d
feature(app.py): adding queue_metrics to queues
...
publishing queue metrics for SRE
[none]
2017-02-14 16:01:28 -05:00
Joseph Schorr
b407f88a26
Remove unnecessary CloudWatch metrics
...
They are spamming the API and costing us a lot of money
2017-02-01 13:08:21 -05:00
Joseph Schorr
f4bcf68928
Add support for custom ports on RADOS and S3 storage engines
2016-12-01 14:23:18 -05:00
Charlton Austin
e7b029c993
Fixing how we encode some JSON.
2016-11-28 11:37:26 -05:00
Joseph Schorr
5f99448adc
Add a chunk cleanup queue for async GC of empty chunks
...
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
2016-11-15 15:07:41 -05:00
Joseph Schorr
59cb6bd216
Make sure to not log exceptions if Swift deletes fail
2016-11-11 14:17:32 -05:00
Joseph Schorr
ce0e3e0e8d
Add missing parameter on RADOS storage
...
Fixes Python error that results from the missing parameter
2016-10-31 12:48:05 -04:00
Joseph Schorr
bfe2646a50
Make sure we don't generate chunk sizes larger than 5 GB.
...
Amazon S3 does not allow for chunk sizes larger than 5 GB; we currently don't handle that case at all, which is why large uploads are failing. This change ensures that if a storage engine specifies a *maximum* chunk size, we write multiple chunks no larger than that size.
2016-10-25 13:57:49 -04:00
Joseph Schorr
29b8905051
Add retry and ignore around Swift deleting empty chunks
...
Fixes #1979
2016-10-13 12:53:59 -04:00
Joseph Schorr
38415065e6
Add S3 retry to all possible operations around the multipart upload
...
Fixes #1933
2016-10-04 21:54:23 +03:00
Joseph Schorr
6ae3faf7fc
Add explicit config parameter to the JWT auth methods
2016-09-29 11:15:20 +02:00
Joseph Schorr
dd2e086a20
Add feature flag to force all direct download URLs to be proxied
...
Fixes #1667
2016-09-29 11:13:41 +02:00
Joseph Schorr
4f0ddc8d6e
Don't exception log for expected 404s in Swift storage
...
Fixes #1850
2016-09-20 14:07:29 -04:00
Joseph Schorr
8c67ed14e8
Delete empty swift chunks
...
Fixes #1844
2016-09-19 11:41:20 -04:00
Joseph Schorr
41cffe33f0
Fix borkened storage call
2016-09-02 14:29:53 -04:00
Joseph Schorr
233f55829e
Add multipart upload retry to chunk uploads as well
2016-09-02 12:00:18 -04:00
Joseph Schorr
0bc90ea45b
Add retry attempts for internal error on multipart upload
...
Fixes #1740
2016-08-18 12:04:36 -04:00
Joseph Schorr
14b93f72ff
Make S3 access key and secret key optional, enabling IAM.
...
If not specified, then boto will fallback to reading the credentials from IAM if on an EC2 machine. This should be safe as the validator will still ensure the credentials work if not specified.
Fixes #1707
2016-08-11 17:17:36 -04:00
josephschorr
630d022b56
Merge pull request #1675 from coreos-inc/fix-storage-validation
...
Add missing arg to validate call and add logging
2016-08-03 14:05:37 -04:00
Joseph Schorr
0b47fce1cb
Add logging when Swift deletion fails
2016-08-03 12:16:06 -04:00
Joseph Schorr
974ab6c42c
Add missing arg to validate call and add logging
2016-08-03 11:13:27 -04:00
josephschorr
83849f4320
Merge pull request #1660 from coreos-inc/storage-status
...
Add storage validation to the status endpoint
2016-08-03 11:09:48 -04:00
Joseph Schorr
770ac0016e
Change validate method to work for all storages
2016-08-02 15:01:37 -04:00
Joseph Schorr
c30b8dd1ad
Add storage validation to the status endpoint
...
Fixes #1659
2016-08-01 13:02:26 -04:00
Joseph Schorr
a41ccf0356
Add retry support to Swift
...
Fixes #1636
2016-07-26 09:56:00 -07:00
Joseph Schorr
cbf7c2bf44
Add better logging to blob uploads
...
Fixes #1635
2016-07-20 17:53:43 -04:00
Joseph Schorr
13a30d45b1
Add swift configuration options and default connection timeout
...
Fixes #1627
2016-07-19 17:19:11 -04:00
