Joseph Schorr
b1dd053b02
Fix an NPE when trying to pull a manifest without a legacy image via V1
2019-01-10 13:59:35 -05:00
Joseph Schorr
171c7e5238
Further fixes for unicode handling in manifests
...
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
2019-01-09 15:14:41 -05:00
Joseph Schorr
b7b64d84d3
Disable verification of schemas temporarily while we determine why some older schemas don't validate
2019-01-08 14:25:35 -05:00
Joseph Schorr
c510759df3
Remove the foreign key constraint from the kind field on the logentry3 table
2019-01-04 13:35:14 -05:00
Joseph Schorr
e309508776
Change lookup of blobs to not use a subquery, which is murder on MySQL
2019-01-03 15:41:21 -05:00
Joseph Schorr
cdb49dbfd3
Add LogEntry3 table without the extra indexes and switch to writing to it
2019-01-03 13:50:43 -05:00
Joseph Schorr
b6db002729
Fix references to LogEntry model used and add support for a maximum page of results on the logs API
2019-01-02 16:04:15 -05:00
Joseph Schorr
4ba4d9141b
Put user last accessed updating behind a feature flag
2019-01-02 16:04:15 -05:00
Joseph Schorr
a6b5b4d6e3
Switch to writing logs to LogEntry2 table
2019-01-02 16:04:15 -05:00
Joseph Schorr
58d7dd07b9
Revert the changes to drop LogEntry2 table definition
2019-01-02 13:29:35 -05:00
Joseph Schorr
442312402f
Merge pull request #3299 from quay/joseph.schorr/QUAY-1233/batch-logs-download
...
Batch download of logs
2018-12-19 15:38:22 -05:00
Joseph Schorr
9190c046c7
Remove unusable sitemap endpoint
...
It calls a data model method which is extremely slow and heavy, and it isn't even used anyway
2018-12-18 16:37:09 -05:00
Joseph Schorr
8a212728a3
Implement a worker for batch exporting of usage logs
...
This will allow customers to request their usage logs for a repository or an entire namespace, and we can export the logs in a manner that doesn't absolutely destroy the database, with every step along the way timed.
2018-12-18 15:33:03 -05:00
Joseph Schorr
9e2c5417b8
Merge pull request #3310 from quay/joseph.schorr/QUAY-1235/drop-logentry2
...
Remove all references to LogEntry2 from the code
2018-12-18 15:18:22 -05:00
Joseph Schorr
fc691cefb4
Merge pull request #3306 from quay/fix-manifest-ui
...
Fix manifest UI page to properly show the layers of manifests and show manifest lists
2018-12-11 16:53:36 -05:00
Joseph Schorr
6a94eba1a2
Remove all references to LogEntry2 from the code
...
The migration to actually drop the table will go in after this code has been pushed to production
2018-12-11 15:54:16 -05:00
Joseph Schorr
a9b8586423
Make sure list_manifest_layers is not callable for manifest lists
2018-12-11 13:10:24 -05:00
Joseph Schorr
7fa60d5802
Fix manifest UI display for manifests with empty layers
...
Also fixes the command for schema 2 manifests
2018-12-10 17:10:54 -05:00
Joseph Schorr
eb7591183d
Add a tag backfill worker to fully backfill the new-style Tag's in the background
2018-12-10 15:36:25 -05:00
Joseph Schorr
4106f5ce51
Fix manifest UI page to properly show the layers of manifests and show manifest lists
2018-12-10 15:33:59 -05:00
Joseph Schorr
3c2e050593
Support pulling of schema2 manifests directly via a manifest list tag
...
This change ensures that if a manifest list is requested with an accepts header for a *schema 2* manifest, the legacy manifest (if any) is returned as schema 2 if it was pushed as a schema 2 manifest (rather than being auto-converted to schema 1)
2018-12-07 11:57:13 -05:00
Joseph Schorr
c3710a6a5e
Add ability for specific geographic regions to be blocked from pulling images within a namespace
2018-12-06 17:14:45 -05:00
Joseph Schorr
d59bea3569
Enable a configurable whitelist of namespaces for V22
...
If a namespace is present in the whitelist, all calls are sent to the OCI model instead of the Pre OCI model
Note that this does increase overhead for registry calls (since we need to lookup the namespace for every single call), but it should only be temporary until we've migrated all users over to the OCI data model
2018-12-03 17:37:28 -05:00
Joseph Schorr
492934de3c
Remove redundant call
2018-11-27 18:44:34 +02:00
Joseph Schorr
4e1ff90cb2
Unify the get_layers calls across all implements of manifest schemas to ensure we have a common type returned
...
Also renames some methods to make it more clear what kind of information they return
2018-11-26 17:58:48 +02:00
Joseph Schorr
180d8847db
Fix image replication for images with remote layers
2018-11-26 16:15:48 +02:00
Joseph Schorr
adccdd30ca
Read blobs from new manifest blob table where relevant
2018-11-25 17:31:09 +02:00
Joseph Schorr
4985040d31
Properly handle the empty layer when pushing schema 2 manifests
...
Docker doesn't send us the contents of this layer, so we are forced to synthesize it ourselves
2018-11-25 16:16:59 +02:00
Joseph Schorr
ffd567e2c2
Rename accessed items in the data model
2018-11-21 18:06:16 +02:00
Joseph Schorr
1f03fdb27e
Fix pulling of squashed versions of the legacy image in a manifest lists
2018-11-20 16:37:15 +02:00
Joseph Schorr
001768c043
Make security status for manifests without legacy images more clear in the UI
2018-11-20 16:37:15 +02:00
Joseph Schorr
e972e4088b
Fix lookup of manifests referenced solely by a manifest list
...
We need to ensure we can find them if there is an active tag pointing to the parent list
2018-11-19 23:31:41 +02:00
Joseph Schorr
54904cfd6e
Fix bug around pushing manifest lists that refer to the same manifest twice as children
2018-11-19 18:26:22 +02:00
Joseph Schorr
e6c2ddfa93
Add support for direct pushing of schema 2 manifests without tags
...
This is required for manifest lists
2018-11-19 14:01:41 +02:00
Joseph Schorr
8a3427e55a
Change registry data model to use a content retriever where applicable
2018-11-19 14:01:41 +02:00
Joseph Schorr
82ee21bfbd
Change OCI data model to use a content retriever for the various manifest operations
2018-11-19 12:24:04 +02:00
Joseph Schorr
1eaf5b18dd
Adjustments based on code review feedback
2018-11-15 13:51:48 +02:00
Joseph Schorr
c46b11bac1
Implement UI support for manifest lists
2018-11-14 18:40:57 +02:00
Joseph Schorr
276d0d571d
Fix support for pulling manifest lists via Docker V1 protocol where applicable
2018-11-14 14:05:06 +02:00
Joseph Schorr
37b20010aa
Add support for pushing and pulling schema 2 manifests with remote layers
...
This is required for windows image support
2018-11-14 13:21:50 +02:00
Joseph Schorr
d97055e2ba
Code review fixes
2018-11-14 09:15:58 +02:00
Joseph Schorr
7a794e29c0
Add tests for manifest lists and fix some issues encountered while testing
2018-11-13 21:03:20 +02:00
Joseph Schorr
9994f0ae61
Disable GC for repositories with manifest lists
...
This is a temporary measure so we can roll V2_2 out for internal customer testing without having to worry about GC interacting oddly with it
2018-11-13 21:03:20 +02:00
Joseph Schorr
7b9f56eff3
Fixes to ensuring existing code can process schema 2 manifests
2018-11-13 17:13:51 +02:00
Joseph Schorr
9474fb7833
Fix bug with json.dumps changing the order of keys
...
We write the literal received JSON string into the image data, to ensure the checksums computed will always match
2018-11-13 16:50:13 +02:00
Joseph Schorr
849e613386
Implement support for schema 2 manifests
2018-11-13 11:49:12 +02:00
Joseph Schorr
d77d383e46
Add schema2 media types
2018-11-13 09:24:04 +02:00
Joseph Schorr
30f072aeff
Add support for creating schema 2 manifests and manifest lists via the OCI model
2018-11-12 23:27:49 +02:00
Joseph Schorr
6b86b87a16
Merge pull request #3290 from quay/joseph.schorr/QUAY-1124/implement-new-data-model
...
Implement the new OCI-based registry data model
2018-11-08 13:39:29 -05:00
Joseph Schorr
fdcb8bad23
Implement the new OCI-based registry data model
...
Note that this change does *not* enable the new data model by default, but does allow it to be used when a special environment variable is specified.
2018-11-07 22:07:58 -05:00
