Commit graph

766 commits

Author SHA1 Message Date
Kenny Lee Sin Cheong
8e643ce5d9
Repository endpoint tags pagination (#3238)
* endpoint/api/repository: limit the number of tags returned

- Limit the number of tags returned by /api/v1/repository/<ns:repo> to 500.
- Uses the tag history endpoint instead, with an active tag filte.
- Update UI to use tag history endpoint instead.
2018-09-14 15:30:54 -04:00
Joseph Schorr
a4f1475de4
Merge pull request #3234 from quay/joseph.schorr/QUAY-1030/interfacing-part-8
Change verbs endpoint to use the new registry data model interface
2018-09-13 12:50:02 -04:00
Joseph Schorr
f252b0b16f Move verbs endpoint to use new registry data model 2018-08-29 16:38:54 -04:00
Joseph Schorr
6c5c2f1a75 Add new methods to registry data model interface in prep for moving verbs to using it 2018-08-29 16:38:54 -04:00
Joseph Schorr
ce61ec6668 Fix bug where GC attempts to delete manifests that are shared by multiple tag manifests
Also adds a test for this case

All of this will be moot once we get rid of tag manifests, but for now, its causing exceptions
2018-08-29 11:11:52 -04:00
Joseph Schorr
70e233ab5c Disallow access in the new registry model to hidden tags
This was causing problems with certain build cache pulls
2018-08-28 13:34:58 -04:00
Joseph Schorr
581ec47798 Handle the cases where an existing tag manifest or manifest has already been created 2018-08-27 15:02:03 -04:00
Joseph Schorr
a96c5a7f64 Optimize the new registry data model to avoid unnecessary queries 2018-08-24 11:25:24 -04:00
Joseph Schorr
8225c61a1f Implement tag functions in new registry model interface 2018-08-22 18:03:57 -04:00
Joseph Schorr
9f96e595ac Change build component labeling to use new registry interface 2018-08-20 15:49:51 -04:00
Joseph Schorr
6414e580f6
Merge pull request #3204 from quay/joseph.schorr/QUAY-1030/registry-interface
Start on a basic registry_model interface
2018-08-17 15:31:11 -04:00
Sam Chow
cc9bedbeb9 refactor approval service key to not need approver 2018-08-15 17:18:41 -04:00
Joseph Schorr
70df10dade Change manifest backfill to lookup missing content checksums in the repository
After discussion, we decided the best solution for the missing content checksum problem was to lookup the proper blobs in the repository and, if not present, mark the manifest as broken, as this would reflect the actual issue the user faces if they pull the repository tag today via V2
2018-08-14 14:36:19 -04:00
Joseph Schorr
c30214c7a8 Start on a basic registry_model interface and change a single module to use it. This will allow us to completely abstract out how we deal with registry-related tables and ensure that transitioning to the new OCI-like model will be easier to do. 2018-08-14 13:48:59 -04:00
Joseph Schorr
96e0fc4ad6 Fix manifest backfill for manifests pointing to V1 images
V1 images don't have checksums, so we just always use the full storage set for the manifest, rather than a checksum map
2018-08-13 15:51:18 -04:00
Joseph Schorr
03ea3a3250 Add worker to backfill the new manifest tables from the tagmanifest table 2018-08-13 14:56:31 -04:00
Joseph Schorr
2ab7ca30d0 Change ManifestBlob row creation to be a single batch operation
Should reduce DB roundtrip times and the overall time we spend under the transaction
2018-08-09 17:04:16 -04:00
Joseph Schorr
1f04b3ea03
Merge pull request #3197 from quay/remove-transaction
Remove the transaction around creating the tag and the tag manifest
2018-08-09 16:34:31 -04:00
Joseph Schorr
8cdddd365b Remove the transaction around creating the tag and the tag manifest
The tag manifest creation operation is now quite a bit heavier (because it is populating a bunch of new-model tables as well), and it is not necessary for the transaction to include this operation, because tags are valid without manifests currently.
2018-08-09 16:32:07 -04:00
Joseph Schorr
590f8f65a3 Allow lookup of "dead" manifests so manifest links can be clicked in the tag history
Fixes https://jira.coreos.com/browse/QUAY-1021
2018-08-09 16:05:10 -04:00
Joseph Schorr
56222f95dc Change manifest creation to take in the map of blobs that form the manifest
We need to lookup the blobs *specific to the images in that manifest*, so we now pass them in from the locations in which we know that information
2018-08-07 16:28:50 -04:00
Joseph Schorr
e3da522d26 Rename store_tag_manifest to indicate clearly it is only to be used for testing 2018-08-07 14:52:59 -04:00
Joseph Schorr
9669320df2 Fix pushing of manifests whose layers share blobs
If a blob was repeated previously, we would get a constraint error from the ManifestBlob table
2018-08-07 13:06:30 -04:00
Joseph Schorr
89582438cd Fix the V22 phase 1 migrations to use new tables for mapping rather than editing existing tables
The ALTER TABLE operations previously used were causing the DB to die when run on the production TagManifest table which has 7 million rows. We instead now use new mapping tables, which is less nice, but these are temporary anyway, so hopefully we only have to deal with their ugliness for a short duration.
2018-08-06 16:58:27 -04:00
Joseph Schorr
7068010751
Merge pull request #3186 from quay/joseph.schorr/QUAY-1013-1015/manifest-migration
First phase of V2_2 data model migrations
2018-08-06 13:56:14 -04:00
Joseph Schorr
a46660a06f Add new Manifest, ManifestLabel, ManifestLegacyImage and ManifestBlob tables and start writing and GCing to/from them
This change also starts passing in the manifest interface, rather than the raw data, to the model for writing.

Note that this change does *not* backfill the existing rows in to the new tables; that will occur in a followup PR. The new columns in `tagmanifest` and `tagmanifestlabel` will be used to track the backfill, as it will occur in a worker.
2018-08-06 13:01:11 -04:00
Sida Chen
6efe262217
Merge pull request #3188 from KeyboardNerd/sidchen/notification_datetime_bug
Fixes empty "created" field in notification
2018-08-06 12:58:38 -04:00
Sida Chen
415631153a Fixes empty "created" field in notification
Fixes https://jira.coreos.com/browse/QUAY-1022
2018-08-01 17:17:47 -04:00
Joseph Schorr
7325b22c90 Change id column in LogEntry to a BigInt and migrate back to using that table 2018-07-30 12:42:56 -04:00
Joseph Schorr
9a40e99a8f Enhancements for validation of DB urls
We now turn off retries and DB pooling, and make sure to always close the connection
2018-07-23 12:43:23 -04:00
Brad Ison
d3d9cca182 Upgrade Peewee to latest 3.x
This requires a number of small changes in the data model code, as well as additional testing.
2018-07-23 12:43:23 -04:00
Joseph Schorr
487edf0ba1 Phase 3 of Appr migration
Deletes the old models and their code
2018-07-21 15:43:00 -04:00
Joseph Schorr
113bb96f29 Phase 1 of migrating APPR-specific tables to tables with the Appr prefix
Fixes https://jira.coreos.com/browse/QUAY-950
2018-07-20 13:19:01 -04:00
Joseph Schorr
3f3a0946f6 Move repo visibility tests into pytest 2018-07-19 11:23:11 -04:00
Joseph Schorr
2d7b4dd1bf Move public repo test to pytest 2018-07-18 17:22:05 -04:00
Joseph Schorr
928a5d604a Move build query tests to pytest 2018-07-18 11:26:18 -04:00
Joseph Schorr
eb8e325f94 Move image sharing model tests to pytest 2018-07-16 09:58:21 -04:00
Joseph Schorr
dda083abb1 Move blob model tests to pytest 2018-07-13 13:19:46 +03:00
Brad Ison
73cb7f3228
endpoints/api: Allow null fields in user metadata
The user metadata fields are nullable in the database, but were not in
the json sechema.  This prevented users from updating some of their
information on the site if they hadn't set the metadata fields.
2018-06-27 15:34:55 -04:00
Joseph Schorr
7604e9842b Change repo filtering for users to use a user ID reference, rather than the username
While this means we need an additional query for initial lookup, it makes the *filtering* query (which is the heavy part) require far fewer joins, thus making it more efficient.

Also adds a new unit test to verify that our filter filters to the correct set of repositories.
2018-06-19 10:51:30 -04:00
Joseph Schorr
0fa1a1d5fd Implement support for blob mounting via the mount parameter on blob uploads
Fixes https://jira.coreos.com/browse/QUAY-893
2018-06-06 11:31:28 -04:00
Joseph Schorr
b2262eaf46 Add feature flag to disable username confirmation
Fixes https://jira.coreos.com/browse/QUAY-914
2018-06-01 13:30:50 -04:00
Joseph Schorr
8ee98c1c48 Show starred public repositories in the starred repos list
We forgot to check if the repository was public in the filter

Fixes https://jira.coreos.com/browse/QUAY-857
2018-05-31 17:34:17 -04:00
Joseph Schorr
a007332d4c Temporarily change to storing logs in a new LogEntry2 table
This will prevent us from running out of auto-incrementing ID values until such time as we can upgrade to peewee 3 and change the field type to a BigInt

Fixes https://jira.coreos.com/browse/QUAY-943
2018-05-18 20:15:16 -04:00
Joseph Schorr
f5fafc480f Notifications can target robots, but the model did not reflect that
This fixes a breakage when deleting robots that have notifications targeting them.
2018-05-15 16:19:46 -04:00
josephschorr
58612955d6
Merge pull request #3079 from quay/joseph.schorr/QUAY-927/robots-limit
Limit robots displayed in entity search
2018-05-15 13:59:48 -04:00
josephschorr
6780a1f06b
Merge pull request #3077 from quay/joseph.schorr/QUAY-931/log-slowdown
Logs slowdown fixes
2018-05-15 13:28:05 -04:00
Joseph Schorr
5c50161d85 Limit robots displayed in entity search
Before, we'd load *all* the robots, which can be a huge issue in namespaces with a large number of robots. Now, we only load the top-20 robots (as per recency in login), and we also limit the information returned to the entity search to save some bandwidth.

Fixes https://jira.coreos.com/browse/QUAY-927
2018-05-15 11:00:57 -04:00
Joseph Schorr
b98c65b3a3 Filter out deleted users and organizations from the superuser list
Superusers were getting confused because the users/orgs were being disabled and renamed, but still appeared in the list until they were GCed by the background worker. Now we just hide them.

Fixes https://jira.coreos.com/browse/QUAY-936
2018-05-15 10:42:41 -04:00
Joseph Schorr
e3248bde47 Small fixes to make loading of logs faster
Removes filtering of log types where not necessary, removes filtering based on namespace when filtering based on repository (superfluous check that was causing issues in MySQL preventing the use of the correct index) and fix some other small issues around the API

Fixes https://jira.coreos.com/browse/QUAY-931
2018-05-14 11:41:49 -04:00