Joseph Schorr
0035a1ec6d
Fix a bug in manifest backfill that made it only work on active tags
2019-01-21 16:59:37 -05:00
Joseph Schorr
fe45797490
Optimize the list active tags query
...
Some customers are hitting this endpoint rapidly for repositories with many, many tags. This change drops the unnecessary joins, which should reduce database load somewhat.
2019-01-14 13:34:25 -05:00
Joseph Schorr
171c7e5238
Further fixes for unicode handling in manifests
...
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
2019-01-09 15:14:41 -05:00
Joseph Schorr
37b20010aa
Add support for pushing and pulling schema 2 manifests with remote layers
...
This is required for windows image support
2018-11-14 13:21:50 +02:00
Joseph Schorr
6b86b87a16
Merge pull request #3290 from quay/joseph.schorr/QUAY-1124/implement-new-data-model
...
Implement the new OCI-based registry data model
2018-11-08 13:39:29 -05:00
Joseph Schorr
fdcb8bad23
Implement the new OCI-based registry data model
...
Note that this change does *not* enable the new data model by default, but does allow it to be used when a special environment variable is specified.
2018-11-07 22:07:58 -05:00
Joseph Schorr
3e63b08731
Merge pull request #3288 from quay/faster-gc
...
Change garbage collection queries to be far smaller by GCing per tag and per image
2018-11-07 14:35:38 -05:00
Joseph Schorr
5124422332
Change garbage collection queries to be far smaller by GCing per tag and per image
...
While this will require far more iterations and queries, each query itself will be quite small, thus preventing us from locking up the database
2018-11-07 00:50:34 -05:00
Joseph Schorr
5b089653d1
Add an extra check in garbage collection to ensure we don't delete manifests referenced by other new-style tags
2018-11-02 15:56:18 -04:00
Joseph Schorr
4779e05f55
Fix NPE in changing the expiration of a tag
2018-11-02 14:54:13 -04:00
Joseph Schorr
114e2c3bf2
Have all tag code add, modify and delete both old and new style tags
...
This preps us for being able to use the new data model with existing repositories
2018-11-01 17:59:10 -04:00
Joseph Schorr
4a7b4ad06a
Fix disabled namespace check
2018-10-18 14:25:40 -04:00
Joseph Schorr
e91ba98e1b
Convert V2's tag endpoints to use the new data model interface
2018-10-18 14:25:40 -04:00
Joseph Schorr
6b5064aba4
Convert V2's manifest endpoints to use the new data model interface
2018-10-18 14:25:40 -04:00
Joseph Schorr
0ae062be62
Add manifest creation to new registry data model interface
2018-10-18 14:25:39 -04:00
Kenny Lee Sin Cheong
8e643ce5d9
Repository endpoint tags pagination ( #3238 )
...
* endpoint/api/repository: limit the number of tags returned
- Limit the number of tags returned by /api/v1/repository/<ns:repo> to 500.
- Uses the tag history endpoint instead, with an active tag filte.
- Update UI to use tag history endpoint instead.
2018-09-14 15:30:54 -04:00
Joseph Schorr
ce61ec6668
Fix bug where GC attempts to delete manifests that are shared by multiple tag manifests
...
Also adds a test for this case
All of this will be moot once we get rid of tag manifests, but for now, its causing exceptions
2018-08-29 11:11:52 -04:00
Joseph Schorr
70e233ab5c
Disallow access in the new registry model to hidden tags
...
This was causing problems with certain build cache pulls
2018-08-28 13:34:58 -04:00
Joseph Schorr
581ec47798
Handle the cases where an existing tag manifest or manifest has already been created
2018-08-27 15:02:03 -04:00
Joseph Schorr
a96c5a7f64
Optimize the new registry data model to avoid unnecessary queries
2018-08-24 11:25:24 -04:00
Joseph Schorr
8225c61a1f
Implement tag functions in new registry model interface
2018-08-22 18:03:57 -04:00
Joseph Schorr
6414e580f6
Merge pull request #3204 from quay/joseph.schorr/QUAY-1030/registry-interface
...
Start on a basic registry_model interface
2018-08-17 15:31:11 -04:00
Joseph Schorr
70df10dade
Change manifest backfill to lookup missing content checksums in the repository
...
After discussion, we decided the best solution for the missing content checksum problem was to lookup the proper blobs in the repository and, if not present, mark the manifest as broken, as this would reflect the actual issue the user faces if they pull the repository tag today via V2
2018-08-14 14:36:19 -04:00
Joseph Schorr
c30214c7a8
Start on a basic registry_model interface and change a single module to use it. This will allow us to completely abstract out how we deal with registry-related tables and ensure that transitioning to the new OCI-like model will be easier to do.
2018-08-14 13:48:59 -04:00
Joseph Schorr
96e0fc4ad6
Fix manifest backfill for manifests pointing to V1 images
...
V1 images don't have checksums, so we just always use the full storage set for the manifest, rather than a checksum map
2018-08-13 15:51:18 -04:00
Joseph Schorr
03ea3a3250
Add worker to backfill the new manifest tables from the tagmanifest table
2018-08-13 14:56:31 -04:00
Joseph Schorr
2ab7ca30d0
Change ManifestBlob row creation to be a single batch operation
...
Should reduce DB roundtrip times and the overall time we spend under the transaction
2018-08-09 17:04:16 -04:00
Joseph Schorr
1f04b3ea03
Merge pull request #3197 from quay/remove-transaction
...
Remove the transaction around creating the tag and the tag manifest
2018-08-09 16:34:31 -04:00
Joseph Schorr
8cdddd365b
Remove the transaction around creating the tag and the tag manifest
...
The tag manifest creation operation is now quite a bit heavier (because it is populating a bunch of new-model tables as well), and it is not necessary for the transaction to include this operation, because tags are valid without manifests currently.
2018-08-09 16:32:07 -04:00
Joseph Schorr
590f8f65a3
Allow lookup of "dead" manifests so manifest links can be clicked in the tag history
...
Fixes https://jira.coreos.com/browse/QUAY-1021
2018-08-09 16:05:10 -04:00
Joseph Schorr
56222f95dc
Change manifest creation to take in the map of blobs that form the manifest
...
We need to lookup the blobs *specific to the images in that manifest*, so we now pass them in from the locations in which we know that information
2018-08-07 16:28:50 -04:00
Joseph Schorr
e3da522d26
Rename store_tag_manifest to indicate clearly it is only to be used for testing
2018-08-07 14:52:59 -04:00
Joseph Schorr
9669320df2
Fix pushing of manifests whose layers share blobs
...
If a blob was repeated previously, we would get a constraint error from the ManifestBlob table
2018-08-07 13:06:30 -04:00
Joseph Schorr
89582438cd
Fix the V22 phase 1 migrations to use new tables for mapping rather than editing existing tables
...
The ALTER TABLE operations previously used were causing the DB to die when run on the production TagManifest table which has 7 million rows. We instead now use new mapping tables, which is less nice, but these are temporary anyway, so hopefully we only have to deal with their ugliness for a short duration.
2018-08-06 16:58:27 -04:00
Joseph Schorr
a46660a06f
Add new Manifest, ManifestLabel, ManifestLegacyImage and ManifestBlob tables and start writing and GCing to/from them
...
This change also starts passing in the manifest interface, rather than the raw data, to the model for writing.
Note that this change does *not* backfill the existing rows in to the new tables; that will occur in a followup PR. The new columns in `tagmanifest` and `tagmanifestlabel` will be used to track the backfill, as it will occur in a worker.
2018-08-06 13:01:11 -04:00
Brad Ison
d3d9cca182
Upgrade Peewee to latest 3.x
...
This requires a number of small changes in the data model code, as well as additional testing.
2018-07-23 12:43:23 -04:00
Joseph Schorr
9e16a989f5
Audit the number of SQL queries we make in writing manifests, and significantly reduce in the common case
...
Instead of 41 queries now for the simple manifest, we are down to 14.
The biggest changes:
- Only synthesize the V1 image rows if we haven't already found them in the database
- Thread the repository object through to the other model method calls, and use it instead of loading again and again
2018-01-25 11:10:43 -05:00
Joseph Schorr
7d4fed6892
Change error message when trying to pull a deleted or expired tag
...
Will let the users know they can recover the tag via time machine
Note: This was tested with the Docker protocol, but the new error code is *technically* out of spec; we should make sure its okay.
2017-07-19 17:13:48 -04:00
Joseph Schorr
99d7fde8ee
Add UI for viewing and changing the expiration of tags
2017-07-19 17:13:48 -04:00
Joseph Schorr
68d473bff4
Clarify comment and make pairs code a little nicer
2017-05-03 11:41:39 -04:00
Joseph Schorr
977bbc20a2
Add filtering onto the images query in get_matching_tags_for_images
...
Should make the query even faster in the security notification case
2017-05-02 18:29:14 -04:00
Joseph Schorr
d77463cc8b
Add sharing to the images lookup in get_matching_tags_for_images
...
Should prevent an issue if we ever get a request with thousands and thousands of images where we would exceed the DB's max packet size
2017-05-02 15:50:31 -04:00
Joseph Schorr
74dd0ef8e8
Add a batch get_matching_tags_for_images
method
...
This will be used in the security notification worker to retrieving the tags needed in a set of batch calls, rather than multiple calls per image
2017-05-02 15:38:25 -04:00
Joseph Schorr
e583be3914
Remove inner query for ancestors lookup on get_matching_tags
2017-04-28 20:10:54 -04:00
Joseph Schorr
626f306283
Add tests for list_active_tags
2017-04-14 17:53:31 -04:00
Joseph Schorr
4a1fad7520
Make sure to filter hidden tags from the active tags query
2017-04-13 19:29:38 -04:00
Joseph Schorr
ab2f044331
Switch get repo API to use a single list tags query
...
Should make things faster since the join occurs on the database side
2017-04-13 18:06:58 -04:00
Joseph Schorr
e90cab4d77
Change revert tag into restore tag and add manifest support
2017-03-14 11:34:42 -04:00
Joseph Schorr
af743b156b
Show manifest digests in place of V1 ids in the tag view when possible
2017-03-14 11:34:41 -04:00
Jimmy Zelinskie
53eb579459
data.model.tag: find min *alive* tag
2017-03-10 13:15:35 -05:00