Joseph Schorr
96072a44c0
Fix ordering of tags in the OCI model
...
Before this change, we were neglecting to sort the tags by ID, which meant that pagination was broken
2019-03-07 12:53:02 -05:00
Joseph Schorr
77fa28a749
Always create a new tag manifest row when backfilling in the old data model
...
This is temporary anyway, but it will allow us to finish the backfill without having to worry about newer tags being displayed with just their V1IDs
Fixes https://jira.coreos.com/browse/QUAY-1353
2019-02-20 16:04:31 -05:00
Joseph Schorr
bacf074219
Fix backfills of super large manifests by stripping metadata from all but the final layer
...
This is semantically valid because Docker only uses the leaf layer as the image config when reading a V2_1 manifest
Fixes https://jira.coreos.com/browse/QUAY-1351
2019-02-20 12:08:20 -05:00
Joseph Schorr
77d7600b64
Add ability for proportional rollout of the OCI data model
2019-02-18 16:34:43 -05:00
Joseph Schorr
4a22a52aee
Make sure to delete the image placements in the manifest builder cleanup code
2019-02-16 00:28:27 -05:00
Joseph Schorr
d878232070
Make sure the manifest builder only deletes temporary storage rows it created
...
We were accidentally deleting other image storage rows in the repository, which is really, really bad
2019-02-15 21:18:02 -05:00
Joseph Schorr
3c526c957a
Catch exceptions if trying to build an invalid manifest in the manifest backfill code
2019-02-15 21:17:33 -05:00
Joseph Schorr
7688a3e348
Make sure _lookup_repo_storages_by_content_checksum operates over a set
2019-02-14 14:43:23 -05:00
Joseph Schorr
f75f315037
Optimize lookup of shared global blobs
...
Currently, we only have one (the shared empty layer), but this should make the blob lookups for repositories significantly faster, as we won't need to do the massive join.
2019-02-14 12:46:42 -05:00
Joseph Schorr
0b706bc7c4
Make sure images in the process of being uploaded are not scanned by Clair
2019-02-08 16:20:44 -05:00
Joseph Schorr
0bfbccdd44
Add a handler for broken tags in the tag backfill system
...
This will generate a tag pointing to an empty manifest; the tag will be broken, but as it is *already* broken, at least the backfill can complete
2019-02-07 13:36:47 -05:00
Joseph Schorr
0d1ac7a4bb
Small optimization in manifest loading in the OCI data model
2019-01-29 13:11:48 -05:00
Joseph Schorr
bbfb0211dc
Fix bug where we weren't properly caching enum values and we were always looking up the media type when constructing manifests
2019-01-22 14:46:38 -05:00
Joseph Schorr
0035a1ec6d
Fix a bug in manifest backfill that made it only work on active tags
2019-01-21 16:59:37 -05:00
Joseph Schorr
7c241a93f6
Allow namespaces to be whitelisted for OCI, but not V22
2019-01-21 14:40:26 -05:00
Joseph Schorr
570d974067
Add support for retargeting a tag to all schema 1 manifests
...
Schema version 1 manifests contain the tag name, and we have a check to ensure we don't point a tag at a manifest with the wrong name embedded. However, this also means that we cannot retarget to that manifest, which will break the UI once we get rid of legacy images.
This change means we can retarget to those manifests, and the OCI model does the work of rewriting the manifest when necessary.
2019-01-15 16:00:06 -05:00
Joseph Schorr
fe45797490
Optimize the list active tags query
...
Some customers are hitting this endpoint rapidly for repositories with many, many tags. This change drops the unnecessary joins, which should reduce database load somewhat.
2019-01-14 13:34:25 -05:00
Joseph Schorr
eb9ca8e8a8
Fix handling of four byte utf8 manifests
...
- Adds the charset: utf-8 to all the manifest responses
- Makes sure we connect to MySQL in utf8mb4 mode, to ensure we can properly read and write 4-byte utf8 strings
- Adds tests for all of the above
2019-01-10 16:34:56 -05:00
Joseph Schorr
b1dd053b02
Fix an NPE when trying to pull a manifest without a legacy image via V1
2019-01-10 13:59:35 -05:00
Joseph Schorr
171c7e5238
Further fixes for unicode handling in manifests
...
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
2019-01-09 15:14:41 -05:00
Joseph Schorr
b7b64d84d3
Disable verification of schemas temporarily while we determine why some older schemas don't validate
2019-01-08 14:25:35 -05:00
Joseph Schorr
fc691cefb4
Merge pull request #3306 from quay/fix-manifest-ui
...
Fix manifest UI page to properly show the layers of manifests and show manifest lists
2018-12-11 16:53:36 -05:00
Joseph Schorr
a9b8586423
Make sure list_manifest_layers is not callable for manifest lists
2018-12-11 13:10:24 -05:00
Joseph Schorr
7fa60d5802
Fix manifest UI display for manifests with empty layers
...
Also fixes the command for schema 2 manifests
2018-12-10 17:10:54 -05:00
Joseph Schorr
eb7591183d
Add a tag backfill worker to fully backfill the new-style Tag's in the background
2018-12-10 15:36:25 -05:00
Joseph Schorr
4106f5ce51
Fix manifest UI page to properly show the layers of manifests and show manifest lists
2018-12-10 15:33:59 -05:00
Joseph Schorr
3c2e050593
Support pulling of schema2 manifests directly via a manifest list tag
...
This change ensures that if a manifest list is requested with an accepts header for a *schema 2* manifest, the legacy manifest (if any) is returned as schema 2 if it was pushed as a schema 2 manifest (rather than being auto-converted to schema 1)
2018-12-07 11:57:13 -05:00
Joseph Schorr
c3710a6a5e
Add ability for specific geographic regions to be blocked from pulling images within a namespace
2018-12-06 17:14:45 -05:00
Joseph Schorr
d59bea3569
Enable a configurable whitelist of namespaces for V22
...
If a namespace is present in the whitelist, all calls are sent to the OCI model instead of the Pre OCI model
Note that this does increase overhead for registry calls (since we need to lookup the namespace for every single call), but it should only be temporary until we've migrated all users over to the OCI data model
2018-12-03 17:37:28 -05:00
Joseph Schorr
4e1ff90cb2
Unify the get_layers calls across all implements of manifest schemas to ensure we have a common type returned
...
Also renames some methods to make it more clear what kind of information they return
2018-11-26 17:58:48 +02:00
Joseph Schorr
180d8847db
Fix image replication for images with remote layers
2018-11-26 16:15:48 +02:00
Joseph Schorr
adccdd30ca
Read blobs from new manifest blob table where relevant
2018-11-25 17:31:09 +02:00
Joseph Schorr
ffd567e2c2
Rename accessed items in the data model
2018-11-21 18:06:16 +02:00
Joseph Schorr
1f03fdb27e
Fix pulling of squashed versions of the legacy image in a manifest lists
2018-11-20 16:37:15 +02:00
Joseph Schorr
001768c043
Make security status for manifests without legacy images more clear in the UI
2018-11-20 16:37:15 +02:00
Joseph Schorr
e6c2ddfa93
Add support for direct pushing of schema 2 manifests without tags
...
This is required for manifest lists
2018-11-19 14:01:41 +02:00
Joseph Schorr
8a3427e55a
Change registry data model to use a content retriever where applicable
2018-11-19 14:01:41 +02:00
Joseph Schorr
c46b11bac1
Implement UI support for manifest lists
2018-11-14 18:40:57 +02:00
Joseph Schorr
276d0d571d
Fix support for pulling manifest lists via Docker V1 protocol where applicable
2018-11-14 14:05:06 +02:00
Joseph Schorr
37b20010aa
Add support for pushing and pulling schema 2 manifests with remote layers
...
This is required for windows image support
2018-11-14 13:21:50 +02:00
Joseph Schorr
7a794e29c0
Add tests for manifest lists and fix some issues encountered while testing
2018-11-13 21:03:20 +02:00
Joseph Schorr
7b9f56eff3
Fixes to ensuring existing code can process schema 2 manifests
2018-11-13 17:13:51 +02:00
Joseph Schorr
9474fb7833
Fix bug with json.dumps
changing the order of keys
...
We write the literal received JSON string into the image data, to ensure the checksums computed will always match
2018-11-13 16:50:13 +02:00
Joseph Schorr
849e613386
Implement support for schema 2 manifests
2018-11-13 11:49:12 +02:00
Joseph Schorr
30f072aeff
Add support for creating schema 2 manifests and manifest lists via the OCI model
2018-11-12 23:27:49 +02:00
Joseph Schorr
fdcb8bad23
Implement the new OCI-based registry data model
...
Note that this change does *not* enable the new data model by default, but does allow it to be used when a special environment variable is specified.
2018-11-07 22:07:58 -05:00
Joseph Schorr
e8ed43b46f
Fix name of reversion field in new Tag table and add a test
2018-11-01 17:59:10 -04:00
Joseph Schorr
114e2c3bf2
Have all tag code add, modify and delete both old and new style tags
...
This preps us for being able to use the new data model with existing repositories
2018-11-01 17:59:10 -04:00
Joseph Schorr
36db133b86
Pull out common shared code in the data model interface
2018-11-01 17:59:10 -04:00
Joseph Schorr
053d918d67
Rename pre oci model tests in preparation for the new model (the tests will be applicable to both models)
2018-11-01 17:59:09 -04:00