vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9129 commits 2 branches 62 tags 205 MiB
Commit graph

1662 commits

Author SHA1 Message Date
Joseph Schorr
54904cfd6e Fix bug around pushing manifest lists that refer to the same manifest twice as children 2018-11-19 18:26:22 +02:00
Joseph Schorr
e6c2ddfa93 Add support for direct pushing of schema 2 manifests without tags 
This is required for manifest lists
 2018-11-19 14:01:41 +02:00
Joseph Schorr
8a3427e55a Change registry data model to use a content retriever where applicable 2018-11-19 14:01:41 +02:00
Joseph Schorr
82ee21bfbd Change OCI data model to use a content retriever for the various manifest operations 2018-11-19 12:24:04 +02:00
Joseph Schorr
1eaf5b18dd Adjustments based on code review feedback 2018-11-15 13:51:48 +02:00
Joseph Schorr
c46b11bac1 Implement UI support for manifest lists 2018-11-14 18:40:57 +02:00
Joseph Schorr
276d0d571d Fix support for pulling manifest lists via Docker V1 protocol where applicable 2018-11-14 14:05:06 +02:00
Joseph Schorr
37b20010aa Add support for pushing and pulling schema 2 manifests with remote layers 
This is required for windows image support
 2018-11-14 13:21:50 +02:00
Joseph Schorr
d97055e2ba Code review fixes 2018-11-14 09:15:58 +02:00
Joseph Schorr
7a794e29c0 Add tests for manifest lists and fix some issues encountered while testing 2018-11-13 21:03:20 +02:00
Joseph Schorr
9994f0ae61 Disable GC for repositories with manifest lists 
This is a temporary measure so we can roll V2_2 out for internal customer testing without having to worry about GC interacting oddly with it
 2018-11-13 21:03:20 +02:00
Joseph Schorr
7b9f56eff3 Fixes to ensuring existing code can process schema 2 manifests 2018-11-13 17:13:51 +02:00
Joseph Schorr
9474fb7833 Fix bug with json.dumps changing the order of keys 
We write the literal received JSON string into the image data, to ensure the checksums computed will always match
 2018-11-13 16:50:13 +02:00
Joseph Schorr
849e613386 Implement support for schema 2 manifests 2018-11-13 11:49:12 +02:00
Joseph Schorr
d77d383e46 Add schema2 media types 2018-11-13 09:24:04 +02:00
Joseph Schorr
30f072aeff Add support for creating schema 2 manifests and manifest lists via the OCI model 2018-11-12 23:27:49 +02:00
Joseph Schorr
6b86b87a16
Merge pull request #3290 from quay/joseph.schorr/QUAY-1124/implement-new-data-model 
Implement the new OCI-based registry data model
 2018-11-08 13:39:29 -05:00
Joseph Schorr
fdcb8bad23 Implement the new OCI-based registry data model 
Note that this change does *not* enable the new data model by default, but does allow it to be used when a special environment variable is specified.
 2018-11-07 22:07:58 -05:00
Joseph Schorr
3e63b08731
Merge pull request #3288 from quay/faster-gc 
Change garbage collection queries to be far smaller by GCing per tag and per image
 2018-11-07 14:35:38 -05:00
Joseph Schorr
5124422332 Change garbage collection queries to be far smaller by GCing per tag and per image 
While this will require far more iterations and queries, each query itself will be quite small, thus preventing us from locking up the database
 2018-11-07 00:50:34 -05:00
Joseph Schorr
5b089653d1 Add an extra check in garbage collection to ensure we don't delete manifests referenced by other new-style tags 2018-11-02 15:56:18 -04:00
Joseph Schorr
4779e05f55 Fix NPE in changing the expiration of a tag 2018-11-02 14:54:13 -04:00
Joseph Schorr
0c83e1272b Switch Tag name field back to ASCII 
Because we run MySQL 5.6 in production, making this field a utf-8 went outside the 767 byte max size for indexes on the `tag_repository_id_name_lifetime_end_ms` index.
 2018-11-02 14:35:39 -04:00
Joseph Schorr
e8ed43b46f Fix name of reversion field in new Tag table and add a test 2018-11-01 17:59:10 -04:00
Joseph Schorr
114e2c3bf2 Have all tag code add, modify and delete both old and new style tags 
This preps us for being able to use the new data model with existing repositories
 2018-11-01 17:59:10 -04:00
Joseph Schorr
36db133b86 Pull out common shared code in the data model interface 2018-11-01 17:59:10 -04:00
Joseph Schorr
c0653ef2ad Add Tag, TagKind and ManifestChild tables in prep for new data model 2018-11-01 17:59:09 -04:00
Joseph Schorr
053d918d67 Rename pre oci model tests in preparation for the new model (the tests will be applicable to both models) 2018-11-01 17:59:09 -04:00
Joseph Schorr
366bddd20a Fix NPE bug in torrentinfo set call 
Fixes https://sentry.io/coreos/backend-production/issues/747494304
 2018-10-30 16:47:59 -04:00
Joseph Schorr
88f19ee0b9 Make blob upload errors more specific 2018-10-29 12:21:18 -04:00
Joseph Schorr
4a7b4ad06a Fix disabled namespace check 2018-10-18 14:25:40 -04:00
Joseph Schorr
cbf0edb164 Address remaining usage of the old data interface 2018-10-18 14:25:40 -04:00
Joseph Schorr
e91ba98e1b Convert V2's tag endpoints to use the new data model interface 2018-10-18 14:25:40 -04:00
Joseph Schorr
6b5064aba4 Convert V2's manifest endpoints to use the new data model interface 2018-10-18 14:25:40 -04:00
Joseph Schorr
a172de4fdc Reimplement cache support for blobs in the registry data model 2018-10-18 14:25:40 -04:00
Joseph Schorr
7a68c41f1c Convert V2 to use the blob uploader interface 2018-10-18 14:25:40 -04:00
Joseph Schorr
0ae062be62 Add manifest creation to new registry data model interface 2018-10-18 14:25:39 -04:00
Joseph Schorr
818ed32f87 Add function in data interface for mounting blobs into other repositories 2018-10-18 14:25:39 -04:00
Joseph Schorr
03789b2210 Add interface for batch creation of labels on a manifest 
This cannot be a true batch operation right now because of the current mapping table entries needed, but we can create and use the interface now and change the underlying implementation later
 2018-10-18 14:25:39 -04:00
Joseph Schorr
8cfb3f4fe8 Add interface function for deleting tags pointing to a manifest 2018-10-18 14:25:39 -04:00
Joseph Schorr
0f1d415c49
Merge pull request #3263 from quay/new-tier 
Add new 2000 repo pricing tier
 2018-10-02 22:44:29 -04:00
Joseph Schorr
4520f9e842 Change V1 to use the manifest builder and new registry data model 2018-09-28 17:30:04 -04:00
Joseph Schorr
65d5be23c7 Implement a manifest builder, to allow for easier management of state around constructing manifests 2018-09-28 15:52:43 -04:00
Joseph Schorr
91afeefebe Add new 2000 repo pricing tier 2018-09-28 15:42:57 -04:00
Joseph Schorr
468e5a8fc2
Merge pull request #3250 from quay/joseph.schorr/QUAY-1030/interfacing-part-9 
Implement blob uploader and change V1 to use it
 2018-09-24 16:03:41 -04:00
Joseph Schorr
ba39737c3a Add blob uploader helper class, for easy management of all the state associated with uploading blobs 2018-09-24 12:54:56 -04:00
Joseph Schorr
5d2fe535b4 Add ability to commit blob uploads into blobs 2018-09-14 17:15:25 -04:00
Joseph Schorr
f68fbb8028 Add blob upload to new registry data model 2018-09-14 17:15:25 -04:00
Joseph Schorr
1bbe41bb36 Add blob support to get_legacy_image 2018-09-14 17:15:24 -04:00
Kenny Lee Sin Cheong
8e643ce5d9
Repository endpoint tags pagination (#3238) 
* endpoint/api/repository: limit the number of tags returned

- Limit the number of tags returned by /api/v1/repository/<ns:repo> to 500.
- Uses the tag history endpoint instead, with an active tag filte.
- Update UI to use tag history endpoint instead.
 2018-09-14 15:30:54 -04:00
Joseph Schorr
a4f1475de4
Merge pull request #3234 from quay/joseph.schorr/QUAY-1030/interfacing-part-8 
Change verbs endpoint to use the new registry data model interface
 2018-09-13 12:50:02 -04:00
Joseph Schorr
f252b0b16f Move verbs endpoint to use new registry data model 2018-08-29 16:38:54 -04:00
Joseph Schorr
6c5c2f1a75 Add new methods to registry data model interface in prep for moving verbs to using it 2018-08-29 16:38:54 -04:00
Joseph Schorr
ce61ec6668 Fix bug where GC attempts to delete manifests that are shared by multiple tag manifests 
Also adds a test for this case

All of this will be moot once we get rid of tag manifests, but for now, its causing exceptions
 2018-08-29 11:11:52 -04:00
Joseph Schorr
70e233ab5c Disallow access in the new registry model to hidden tags 
This was causing problems with certain build cache pulls
 2018-08-28 13:34:58 -04:00
Joseph Schorr
581ec47798 Handle the cases where an existing tag manifest or manifest has already been created 2018-08-27 15:02:03 -04:00
Joseph Schorr
f297249100 Move manifest backfill for V1 tags into the new registry model interface 2018-08-27 15:01:27 -04:00
Joseph Schorr
55afa2ca08 Reverting a tag doesn't mean it always has an existing image, so remove that check 2018-08-27 12:52:27 -04:00
Sam Chow
a40e1e939a Merge branch 'master' into delete-setup-page 2018-08-27 11:26:51 -04:00
Joseph Schorr
a96c5a7f64 Optimize the new registry data model to avoid unnecessary queries 2018-08-24 11:25:24 -04:00
Joseph Schorr
46edebe6b0 Change secscan API endpoints to use new registry model interface 2018-08-23 16:36:04 -04:00
Sam Chow
cd6b0a6f46 Merge branch 'master' into delete-setup-page 2018-08-23 12:46:08 -04:00
Joseph Schorr
8225c61a1f Implement tag functions in new registry model interface 2018-08-22 18:03:57 -04:00
Joseph Schorr
a0a6a3d67d Change manifest API endpoints to use new registry data interface 2018-08-21 17:26:32 -04:00
Sam Chow
d9f7c07f42 Add db configuration to modifying config 
Move more tests over
 2018-08-21 15:19:24 -04:00
Joseph Schorr
5a997eb4e3 Change label creation function to return the label created 2018-08-21 14:27:10 -04:00
Joseph Schorr
584a18c0e1 Change datatype base to raise AttributeError on missing keys 2018-08-20 15:49:51 -04:00
Joseph Schorr
254f06e634 Implement legacy image portion of the data model 
This also makes use of the newly created input system
 2018-08-20 15:49:51 -04:00
Joseph Schorr
8aafbf8b8c Switch the registry data model types to our own class constructor 
This allows us to hide the DB ID from external-to-the-package users of the types and will allow us to add model-level caching as well
 2018-08-20 15:49:51 -04:00
Joseph Schorr
9f96e595ac Change build component labeling to use new registry interface 2018-08-20 15:49:51 -04:00
Joseph Schorr
301532279c Fix broken registry data interface tests 2018-08-20 13:01:12 -04:00
Joseph Schorr
e30b746aef Fix TagManifests with shared digests under the same repository. 
TagManifests can (apparently, in very rare scenarios) share manifests with the exact same digests, so we need to support that case in the backfill worker. We also need to remove a unique constraint on the manifest column in the mapping table to support this case.
 2018-08-20 11:32:59 -04:00
Joseph Schorr
6414e580f6
Merge pull request #3204 from quay/joseph.schorr/QUAY-1030/registry-interface 
Start on a basic registry_model interface
 2018-08-17 15:31:11 -04:00
Sam Chow
ec14007268
Merge pull request #3207 from quay/project/gen-sec-key 
Add the service key creation to config tool
 2018-08-16 16:48:15 -04:00
Sam Chow
cc9bedbeb9 refactor approval service key to not need approver 2018-08-15 17:18:41 -04:00
Joseph Schorr
2b4d52a357 Migrate the manifest bytes column to UTF8 2018-08-15 11:41:26 -04:00
Joseph Schorr
70df10dade Change manifest backfill to lookup missing content checksums in the repository 
After discussion, we decided the best solution for the missing content checksum problem was to lookup the proper blobs in the repository and, if not present, mark the manifest as broken, as this would reflect the actual issue the user faces if they pull the repository tag today via V2
 2018-08-14 14:36:19 -04:00
Joseph Schorr
c30214c7a8 Start on a basic registry_model interface and change a single module to use it. This will allow us to completely abstract out how we deal with registry-related tables and ensure that transitioning to the new OCI-like model will be easier to do. 2018-08-14 13:48:59 -04:00
Joseph Schorr
96e0fc4ad6 Fix manifest backfill for manifests pointing to V1 images 
V1 images don't have checksums, so we just always use the full storage set for the manifest, rather than a checksum map
 2018-08-13 15:51:18 -04:00
Joseph Schorr
03ea3a3250 Add worker to backfill the new manifest tables from the tagmanifest table 2018-08-13 14:56:31 -04:00
Joseph Schorr
2ab7ca30d0 Change ManifestBlob row creation to be a single batch operation 
Should reduce DB roundtrip times and the overall time we spend under the transaction
 2018-08-09 17:04:16 -04:00
Joseph Schorr
1f04b3ea03
Merge pull request #3197 from quay/remove-transaction 
Remove the transaction around creating the tag and the tag manifest
 2018-08-09 16:34:31 -04:00
Joseph Schorr
8cdddd365b Remove the transaction around creating the tag and the tag manifest 
The tag manifest creation operation is now quite a bit heavier (because it is populating a bunch of new-model tables as well), and it is not necessary for the transaction to include this operation, because tags are valid without manifests currently.
 2018-08-09 16:32:07 -04:00
Joseph Schorr
590f8f65a3 Allow lookup of "dead" manifests so manifest links can be clicked in the tag history 
Fixes https://jira.coreos.com/browse/QUAY-1021
 2018-08-09 16:05:10 -04:00
Joseph Schorr
56222f95dc Change manifest creation to take in the map of blobs that form the manifest 
We need to lookup the blobs *specific to the images in that manifest*, so we now pass them in from the locations in which we know that information
 2018-08-07 16:28:50 -04:00
Joseph Schorr
e3da522d26 Rename store_tag_manifest to indicate clearly it is only to be used for testing 2018-08-07 14:52:59 -04:00
Joseph Schorr
9669320df2 Fix pushing of manifests whose layers share blobs 
If a blob was repeated previously, we would get a constraint error from the ManifestBlob table
 2018-08-07 13:06:30 -04:00
Joseph Schorr
89582438cd Fix the V22 phase 1 migrations to use new tables for mapping rather than editing existing tables 
The ALTER TABLE operations previously used were causing the DB to die when run on the production TagManifest table which has 7 million rows. We instead now use new mapping tables, which is less nice, but these are temporary anyway, so hopefully we only have to deal with their ugliness for a short duration.
 2018-08-06 16:58:27 -04:00
Joseph Schorr
7068010751
Merge pull request #3186 from quay/joseph.schorr/QUAY-1013-1015/manifest-migration 
First phase of V2_2 data model migrations
 2018-08-06 13:56:14 -04:00
Joseph Schorr
a46660a06f Add new Manifest, ManifestLabel, ManifestLegacyImage and ManifestBlob tables and start writing and GCing to/from them 
This change also starts passing in the manifest interface, rather than the raw data, to the model for writing.

Note that this change does *not* backfill the existing rows in to the new tables; that will occur in a followup PR. The new columns in `tagmanifest` and `tagmanifestlabel` will be used to track the backfill, as it will occur in a worker.
 2018-08-06 13:01:11 -04:00
Sida Chen
6efe262217
Merge pull request #3188 from KeyboardNerd/sidchen/notification_datetime_bug 
Fixes empty "created" field in notification
 2018-08-06 12:58:38 -04:00
Sida Chen
415631153a Fixes empty "created" field in notification 
Fixes https://jira.coreos.com/browse/QUAY-1022
 2018-08-01 17:17:47 -04:00
Joseph Schorr
7325b22c90 Change id column in LogEntry to a BigInt and migrate back to using that table 2018-07-30 12:42:56 -04:00
Joseph Schorr
8e425ee559 Fix for merge 2018-07-23 12:43:23 -04:00
Joseph Schorr
9a40e99a8f Enhancements for validation of DB urls 
We now turn off retries and DB pooling, and make sure to always close the connection
 2018-07-23 12:43:23 -04:00
Joseph Schorr
d15dcae505 Reenable retry wrapper on database operations 2018-07-23 12:43:23 -04:00
Brad Ison
d3d9cca182 Upgrade Peewee to latest 3.x 
This requires a number of small changes in the data model code, as well as additional testing.
 2018-07-23 12:43:23 -04:00
Joseph Schorr
487edf0ba1 Phase 3 of Appr migration 
Deletes the old models and their code
 2018-07-21 15:43:00 -04:00
Joseph Schorr
acb7d626a4 Phase 2 of Appr migration 
Backfills the new tables from the old ones, and switches all code to read and write from the new tables
 2018-07-21 15:25:04 -04:00
Joseph Schorr
113bb96f29 Phase 1 of migrating APPR-specific tables to tables with the Appr prefix 
Fixes https://jira.coreos.com/browse/QUAY-950
 2018-07-20 13:19:01 -04:00