Matt Jibson
b3c2388618
Allow setting of boto's S3 host for SIGv4
...
The problem only happens when a user has configured the new AWS Frankfurt
region for their S3 backend. It is the only region to require the new
v4 signature. All other regions support both v2 and v4. I'm not sure
which version is used by default on US Standard.
We could attempt to figure out where the bucket is hosted based on its
DNS resolution and auto-populate the host field that way. But I think
the amount of effort to have that work correctly outweighs its benefit
for such a simple solution.
fixes #863
fixes #764
2015-11-18 17:19:33 -05:00
Jake Moshenko
52125bbfed
Fix gc by using the v1/v2 storage location helper everywhere
2015-11-16 14:13:37 -05:00
Jake Moshenko
ab340e20ea
Merge remote-tracking branch 'upstream/master' into python-registry-v2
2015-11-11 16:41:40 -05:00
Silas Sewell
5000b1621c
superuser: add storage replication config
2015-11-09 17:34:22 -05:00
Jake Moshenko
9da64f3aba
Stop writing to deprecated columns for image data.
2015-10-24 14:45:15 -04:00
Jake Moshenko
e7a6176594
Merge remote-tracking branch 'upstream/v2-phase4' into python-registry-v2
2015-10-22 16:59:28 -04:00
Jake Moshenko
ce94931540
Stop writing to deprecated columns for image data.
2015-10-22 12:14:39 -04:00
Jimmy Zelinskie
9ddad4a1a9
client-side join chunks for GCS
...
Boto does not implement GCS's custom multipart API and so we're left to
join them client-side until it does.
2015-10-02 14:57:39 -04:00
Jimmy Zelinskie
6ed5087a3c
add client side chunk join method
2015-10-01 12:28:56 -04:00
Jimmy Zelinskie
abe43a0e07
override upload_chunk_complete for RadosGW
...
RadosGW doesn't support server-side copy of keys into multipart, so we
have to always join it on the local side.
2015-09-30 17:46:59 -04:00
Joseph Schorr
a3ebb9028d
Add full unit tests for the file-like objects and fix them
...
Fixes #568
2015-09-30 14:19:25 -04:00
josephschorr
41bfe2ffde
Merge pull request #551 from coreos-inc/python-registry-v2-swift
...
Add V2 storage methods to Swift storage engine
2015-09-28 17:09:34 -04:00
Joseph Schorr
6c59161527
Add V2 storage methods to Swift storage engine
...
Fixes #508
2015-09-28 16:46:19 -04:00
Jimmy Zelinskie
c5aa3ca4f0
make registry v2 tests pass for GCS
...
Fixes #509 .
2015-09-28 15:42:48 -04:00
Jake Moshenko
26cea9a07c
Merge remote-tracking branch 'upstream/master' into python-registry-v2
2015-09-17 16:16:27 -04:00
Joseph Schorr
6f2271d0ae
Add support for direct download in Swift storage engine
...
Fixes #483
2015-09-14 18:00:03 -04:00
Joseph Schorr
cccb1651f5
Fixes for direct cloud storage copying
2015-09-08 16:55:47 -04:00
Jake Moshenko
210ed7cf02
Merge remote-tracking branch 'upstream/master' into python-registry-v2
2015-09-04 16:32:01 -04:00
Jake Moshenko
8269d4ac90
Checkpoint implementing PATCH according to Docker
2015-09-03 16:26:02 -04:00
Joseph Schorr
42dba8655c
Fix auth and add V2 tests!
2015-09-03 12:21:21 -04:00
Joseph Schorr
587ef85c7f
Allow users to choose the version of Swift to use
...
Fixes #442
2015-09-02 17:46:14 -04:00
Matt Jibson
a821ad2b01
Return an error on failed S3 uploads
...
The previous change to this file didn't raise the error up to stream_write,
and so the complete_upload function still ran because the loop was only
broken. It errored because the data was already canceled. This is better
than what we had before, which was to silently fail but report success
(even internally to ourselves!) on bad image upload.
This means we discovered a bug where a user could have failed during image
upload, but quay would write that image to the repository, potentially
writing broken images to S3.
2015-09-01 15:53:32 -04:00
Joseph Schorr
724b1607d7
Add automatic storage replication
...
Adds a worker to automatically replicate data between storages and update the database accordingly
2015-09-01 14:53:32 -04:00
Matt Jibson
ab25542bd7
Measure multipart uploads
...
see #304
2015-08-31 13:48:52 -04:00
Matt Jibson
9aedfc8d2c
Cancel failed multipart uploads
2015-08-31 03:07:44 -04:00
Jake Moshenko
398202e6fc
Implement some new methods on the storage engines.
2015-08-27 11:29:19 -04:00
Joseph Schorr
c07dec4d39
File reader fixes for verbs
...
- Fix local file reader to always read in chunks
- Have gzip stream raise an exception if the full data is requested
2015-08-25 13:49:21 -04:00
Jake Moshenko
e1b3e9e6ae
Another huge batch of registry v2 changes
...
Add patch support and resumeable sha
Implement all actual registry methods
Add a simple database generation option
2015-08-12 16:41:12 -04:00
Joseph Schorr
5ea028c8be
Loosen the check for mounted volumes
...
Docker apparently does not always use the same mounted volume type
Fixes #352
2015-08-12 13:16:58 -04:00
Joseph Schorr
9798f871ec
Always create a new connection to Swift
...
Fixes #335
2015-08-07 13:41:40 -04:00
Jake Moshenko
18100be481
Refactor the util directory to use subpackages.
2015-08-03 16:04:19 -04:00
Joseph Schorr
26ae629189
Prevent local storage setup on non-mounted paths
...
Fixes #269
2015-07-27 14:32:02 -04:00
Jake Moshenko
3efaa255e8
Accidental refactor, split out legacy.py into separate sumodules and update all call sites.
2015-07-17 11:56:15 -04:00
Jake Moshenko
bea8b9ac53
More changes for registry-v2 in python.
...
Implement the minimal changes to the local filesystem storage driver and feed them through the distributed storage driver.
Create a digest package which contains digest_tools and checksums.
Fix the tests to use the new v1 endpoint locations.
Fix repository.delete_instance to properly filter the generated queries to avoid most subquery deletes, but still generate them when not explicitly filtered.
2015-07-17 11:50:41 -04:00
Joseph Schorr
4333bb9e14
Implement stream_read_file for the Swift storage engine
...
Note that Swift doesn't seem to have a file-like interface, so we need to wrap the generator we get back from it.
Fixes #210
2015-07-02 17:52:43 +03:00
Joseph Schorr
9585e2a765
End-to-end registry tests
2015-06-01 16:35:30 -04:00
Joseph Schorr
375d7670a8
Explain why we re-raise ClientException in the swift storage engine
2015-05-26 16:35:12 -04:00
Joseph Schorr
7001fb05bf
Add further comments on the TODO in get_direct_download_url
2015-05-26 16:34:59 -04:00
Joseph Schorr
5845e37e32
Add Swift storage library
2015-05-26 16:05:21 -04:00
Joseph Schorr
53e5fc6265
Have the config setup tool automatically prepare the S3 or GCS storage with CORS config
2015-01-16 16:10:40 -05:00
Joseph Schorr
63504c87fb
Get end-to-end configuration setup working, including verification (except for Github, which is in progress)
2015-01-07 16:20:51 -05:00
Jimmy Zelinskie
716d7a737b
Strip whitespace from ALL the things.
2014-11-24 16:07:38 -05:00
Joseph Schorr
43555af63d
Address review comments
2014-09-17 17:49:46 -04:00
Joseph Schorr
e3c52fa0eb
Work in progress. This is currently broken!
2014-09-16 00:18:57 -04:00
Joseph Schorr
e8ad01cb41
Lots of small NPE and other exception fixes
2014-09-15 11:27:33 -04:00
Jake Moshenko
8b3a3178b0
Finish the build logs archiver, add handlers for cloud and local that handle gzip encoded archived content.
2014-09-11 15:33:10 -04:00
Jake Moshenko
548f855f71
Use the pure python io module to avoid some interaction between gunicorn, wsgi, and bufferedreader that prevents gunicorn from properly sending the files.
2014-09-09 22:28:25 -04:00
Jake Moshenko
c9e1648781
Small fixes to bugs in the streaming handler for use with magic and radosgw.
2014-09-09 18:30:14 -04:00
Jake Moshenko
756e8ec848
Send the content type through to the cloud engines.
2014-09-09 16:52:53 -04:00
Jake Moshenko
29d40db5ea
Add a new RadosGW storage engine. Allow engines to distinguish not only between those that can support direct uploads and downloads, but those that support doing it through the browser. Rename resumeable->resumable.
2014-09-09 15:54:03 -04:00
Jake Moshenko
29f1b048a3
Add support for Google Cloud Storage.
2014-08-12 02:06:44 -04:00
Joseph Schorr
bcbea37fce
Change distributed config format to make it easier for the setup tool
2014-08-07 13:45:15 -04:00
Joseph Schorr
af46d3d455
Remove path from get_supports_resumeable_download
2014-07-07 16:21:27 -04:00
Joseph Schorr
0ac6312c8a
Get resumable downloads working in a way that the Docker CLI will actually understand. Also rename the method to conform the the name used in the Docker source code.
2014-07-02 21:18:46 -04:00
Joseph Schorr
0d89fd7f26
Add code for resumable downloads from S3
2014-07-02 00:39:59 -04:00
Jake Moshenko
bf0e01fba5
Fix bugs with pushing, pulling, and deleting.
2014-06-17 16:37:48 -04:00
Jake Moshenko
bf98575feb
Add the basics of geographic data distribution and get the tests to work.
2014-06-17 16:03:43 -04:00
Jake Moshenko
fb465fd66f
Eradicate the s3 versions of the checksum and the uploading flag.
2014-06-12 19:27:43 -04:00
Jake Moshenko
78c5aec5b9
Switch the checksums to use the registry computed value, remove all assumptions of namespaced paths for legacy storage, fix an upload race condition in the registry code.
2014-06-11 15:37:45 -04:00
Jake Moshenko
ef9fe871fc
Move the upload flag to the database, and use the database stored image size rather than going to s3.
2014-05-02 16:59:46 -04:00
jakedt
e87ffa20cf
First attempt at making config loadable through string config overrides in an env variable.
2014-04-03 17:31:46 -04:00
jakedt
177a21e996
Remove an extraneous slash in image file path computations. Add a script that will audit and fix image ancestry.
2014-02-21 12:05:03 -05:00
jakedt
b619356907
Get the base image stuff working. Checkpoint before fixing the tests.
2014-02-16 17:38:47 -05:00
jakedt
5742e6ea4e
First set of changes toward shared base images.
2014-02-14 16:06:30 -05:00
yackob03
518cd1be85
Actually delete the image data when deleting the tag.
2014-01-09 17:13:26 -05:00
yackob03
de230428cb
Clobbered the s3_bucket variable, oops.
2013-12-03 16:43:56 -08:00
yackob03
bf85013ef6
Try to use a 301 redirect to download directly from s3. Allow the s3 and userfiles to run without internet access.
2013-12-03 16:39:07 -08:00
yackob03
2e9f7cd19d
Restore the temp_store_handler.
2013-11-11 16:41:33 -05:00
yackob03
2cd98fc58e
Make the app config more powerful in terms of injecting fake dependencies. Refactor the tests to use metaclasses and to actually all run.
2013-11-06 23:21:12 -05:00
yackob03
3698e494a1
Clean up an old TODO.
2013-11-04 16:14:37 -05:00
yackob03
b96f678df8
User the secure s3 endpoint and store our files encrypted.
2013-10-31 11:32:08 -04:00
yackob03
cccfe29c77
Adapt the s3 key object to the python stream file interface, one that is compatible with tarfile.
2013-10-20 02:39:23 -04:00
yackob03
a1164269be
Complete the diff generating functionality.
2013-10-18 14:31:14 -04:00
yackob03
93b856bdb3
First few changes for the image diffs feature.
2013-10-17 18:25:19 -04:00
yackob03
cbe0590ce5
Fix s3 delete.
2013-10-01 14:46:44 -04:00
yackob03
4b9c6d9aae
Add api function to fully delete a repository.
2013-10-01 12:13:25 -04:00
yackob03
84adf680b9
Switch the registry and index to use real s3 and rds.
2013-09-30 19:10:27 -04:00
yackob03
44255421df
Namespace the storage in the registry to prevent leaking images if one acquires the image id.
2013-09-25 20:00:22 -04:00
yackob03
deee70d53b
Add the registry code.
2013-09-25 17:50:03 -04:00
