vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9187 commits 2 branches 62 tags 205 MiB
Commit graph

56 commits

Author SHA1 Message Date
Joseph Schorr
bacf074219 Fix backfills of super large manifests by stripping metadata from all but the final layer 
This is semantically valid because Docker only uses the leaf layer as the image config when reading a V2_1 manifest

Fixes https://jira.coreos.com/browse/QUAY-1351
 2019-02-20 12:08:20 -05:00
Joseph Schorr
097311f360 Handle additional error cases in the manifest schema 1 implementation 2019-02-06 11:41:50 -05:00
Joseph Schorr
c1ac12ff8d Some small cleanup to the tag backfill worker 2019-01-21 13:37:11 -05:00
Joseph Schorr
570d974067 Add support for retargeting a tag to all schema 1 manifests 
Schema version 1 manifests contain the tag name, and we have a check to ensure we don't point a tag at a manifest with the wrong name embedded. However, this also means that we cannot retarget to that manifest, which will break the UI once we get rid of legacy images.

This change means we can retarget to those manifests, and the OCI model does the work of rewriting the manifest when necessary.
 2019-01-15 16:00:06 -05:00
Joseph Schorr
eb9ca8e8a8 Fix handling of four byte utf8 manifests 
- Adds the charset: utf-8 to all the manifest responses
- Makes sure we connect to MySQL in utf8mb4 mode, to ensure we can properly read and write 4-byte utf8 strings
- Adds tests for all of the above
 2019-01-10 16:34:56 -05:00
Joseph Schorr
171c7e5238 Further fixes for unicode handling in manifests 
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
 2019-01-09 15:14:41 -05:00
Joseph Schorr
feee49be9e Ensure we encode the config in manifest schema 2 via the canonical JSON format 2018-12-18 14:52:19 -05:00
Joseph Schorr
48e584905a Make sure to decode manifests into utf-8 when necessary 
This fixes a decoding error
 2018-12-17 16:31:24 -05:00
Joseph Schorr
71b7a2b3a2 Make command optional in schema 2 manifests (as per OCI spec) and pull out additional information 
Also updates the manifest view page to show the comment or digest if there is no command defined
 2018-12-11 17:23:39 -05:00
Joseph Schorr
3c2e050593 Support pulling of schema2 manifests directly via a manifest list tag 
This change ensures that if a manifest list is requested with an accepts header for a *schema 2* manifest, the legacy manifest (if any) is returned as schema 2 if it was pushed as a schema 2 manifest (rather than being auto-converted to schema 1)
 2018-12-07 11:57:13 -05:00
Joseph Schorr
63f9043312 Code review small fixes 2018-11-28 12:56:16 +02:00
Joseph Schorr
d03eb12a77 Remove unused function 2018-11-26 21:29:53 +02:00
Joseph Schorr
4e1ff90cb2 Unify the get_layers calls across all implements of manifest schemas to ensure we have a common type returned 
Also renames some methods to make it more clear what kind of information they return
 2018-11-26 17:58:48 +02:00
Joseph Schorr
3e4f4ab7f1 Add conversion test for the latest ubuntu image 2018-11-26 12:45:05 +02:00
Joseph Schorr
38f68f81f2 Add caching of retrieved config in manifest class 2018-11-25 20:03:57 +02:00
Joseph Schorr
adccdd30ca Read blobs from new manifest blob table where relevant 2018-11-25 17:31:09 +02:00
Joseph Schorr
4985040d31 Properly handle the empty layer when pushing schema 2 manifests 
Docker doesn't send us the contents of this layer, so we are forced to synthesize it ourselves
 2018-11-25 16:16:59 +02:00
Joseph Schorr
947c029afa Fix history layer check off-by-one error and add additional conversion tests 
This also moves the conversion golden data into files to make it easier to add additional tests
 2018-11-21 21:49:01 +02:00
Joseph Schorr
c233760007 Fix conversion of schema 2 manifests to schema 1 manifests 
Also adds a number of conversion tests and clarify the interfaces a bit more
 2018-11-21 18:06:16 +02:00
Joseph Schorr
1f03fdb27e Fix pulling of squashed versions of the legacy image in a manifest lists 2018-11-20 16:37:15 +02:00
Joseph Schorr
dcf312d94d Fix exception bases for manifest exceptions 2018-11-20 12:55:45 +02:00
Joseph Schorr
e972e4088b Fix lookup of manifests referenced solely by a manifest list 
We need to ensure we can find them if there is an active tag pointing to the parent list
 2018-11-19 23:31:41 +02:00
Joseph Schorr
82ee21bfbd Change OCI data model to use a content retriever for the various manifest operations 2018-11-19 12:24:04 +02:00
Joseph Schorr
cbfb6054e5 Switch content retrieval in manifests to be behind an interface 
This allows for easy separation of retrieval of config blobs vs manifests
 2018-11-19 11:55:52 +02:00
Joseph Schorr
1eaf5b18dd Adjustments based on code review feedback 2018-11-15 13:51:48 +02:00
Joseph Schorr
c46b11bac1 Implement UI support for manifest lists 2018-11-14 18:40:57 +02:00
Joseph Schorr
37b20010aa Add support for pushing and pulling schema 2 manifests with remote layers 
This is required for windows image support
 2018-11-14 13:21:50 +02:00
Joseph Schorr
d97055e2ba Code review fixes 2018-11-14 09:15:58 +02:00
Joseph Schorr
7a794e29c0 Add tests for manifest lists and fix some issues encountered while testing 2018-11-13 21:03:20 +02:00
Joseph Schorr
7b9f56eff3 Fixes to ensuring existing code can process schema 2 manifests 2018-11-13 17:13:51 +02:00
Joseph Schorr
1b3daac3c3 Add method for retrieving a V1-compatible manifest for any manifest or manifest list 
This is used to serve older clients that don't support the V2 schema format
 2018-11-13 10:40:57 +02:00
Joseph Schorr
e344d4a5cf Enhancements for Docker schema implementations in preparing for supporting schema 2 in the OCI model 
This adds additional required properties and methods to the Docker schema interface to allow us to treat both schema1 and schema2 manifests and lists logically equivalent from the OCI mode perspective
 2018-11-12 23:27:01 +02:00
Joseph Schorr
0ae062be62 Add manifest creation to new registry data model interface 2018-10-18 14:25:39 -04:00
Joseph Schorr
f252b0b16f Move verbs endpoint to use new registry data model 2018-08-29 16:38:54 -04:00
Joseph Schorr
bafab2e734 Change tar stream formatters to be based on tag and manifest, instead of legacy images 2018-08-29 16:38:54 -04:00
Joseph Schorr
c92294c218 Add compressed_size to both schema formats on their layers 2018-08-29 16:38:54 -04:00
Joseph Schorr
7fa3506723 Add created_datetime to schema1 format 2018-08-27 18:03:58 -04:00
Joseph Schorr
fa58f3b1d2 Fix handling of manifests with unicode in the backfill 
Also adds a bunch of tests around manifests to ensure we get the same information in and out
 2018-08-15 11:41:15 -04:00
Joseph Schorr
36c7482385 Define a formal manifest interface and implement in the schema1 and schema2 manifests 
This will allow us to pass arbitrary manifests to the model
 2018-08-06 13:01:11 -04:00
Joseph Schorr
a73cd9170a Add schema2 list support 2018-06-15 16:00:39 -04:00
Joseph Schorr
52b12131f7 Add schema2 manifest and schema2 config, along with tests 2018-06-15 16:00:39 -04:00
Joseph Schorr
4bd70eab3c Add basic tests for schema1 2018-06-15 16:00:39 -04:00
Joseph Schorr
dd470bdc9d Add a reporter for verbs to track number of storage streams are needed 2018-03-26 17:02:39 -04:00
Joseph Schorr
110366f656 Retarget hard links pointing to deleted files by emitting the deleted file contents under the first hard link instance. This fixes a breakage in the squashed TAR where we were pointing hard links to missing data. 
Fixes https://jira.coreos.com/browse/QUAY-885
 2018-03-23 14:00:46 -04:00
Joseph Schorr
9e16a989f5 Audit the number of SQL queries we make in writing manifests, and significantly reduce in the common case 
Instead of 41 queries now for the simple manifest, we are down to 14.

The biggest changes:
  - Only synthesize the V1 image rows if we haven't already found them in the database
  - Thread the repository object through to the other model method calls, and use it instead of loading again and again
 2018-01-25 11:10:43 -05:00
Joseph Schorr
f9bd7ef42b Add validation of Docker V2_1 schemas and add a test for PUTing an invalid schema 2017-12-20 11:43:03 -05:00
Joseph Schorr
11e3724919 Return an http 415 (manifest version not supported) for OCI manifest content types 
This was breaking skopeo, as it first tries to send the *OCI* manifest type, which we didn't say we didn't support, thus breaking the tool
 2017-12-20 11:02:34 -05:00
Jake Moshenko
d58a1ca35a Fix port mapping for ACI conversion from newer Docker manifests. 2016-12-20 14:01:06 -05:00
Joseph Schorr
f4e1748bb7 Fix parent rewrite bug in schema1 manifest code and add a bunch more tests 
Before this change, if you ended up writing a middle layer whose parent is not in the database, the manifest would fail to rewrite. We now just lookup the parent image in the manifest given to us, ignoring whether it is in the database or not (as it doesn't actually matter if not present; it'll be created if necessary).
 2016-10-04 09:15:27 -04:00
Jimmy Zelinskie
ca883e5662 port label support to refactored v2 registry 2016-09-26 14:49:58 -04:00