Joseph Schorr
eb7591183d
Add a tag backfill worker to fully backfill the new-style Tag's in the background
2018-12-10 15:36:25 -05:00
Joseph Schorr
57e93a82c9
Remove manifest backfill worker
2018-12-10 15:36:25 -05:00
Joseph Schorr
3c2e050593
Support pulling of schema2 manifests directly via a manifest list tag
...
This change ensures that if a manifest list is requested with an accepts header for a *schema 2* manifest, the legacy manifest (if any) is returned as schema 2 if it was pushed as a schema 2 manifest (rather than being auto-converted to schema 1)
2018-12-07 11:57:13 -05:00
Joseph Schorr
cdfaca8e78
Disable GC worker in its entirety
2018-12-03 10:42:53 -05:00
Joseph Schorr
4e1ff90cb2
Unify the get_layers calls across all implements of manifest schemas to ensure we have a common type returned
...
Also renames some methods to make it more clear what kind of information they return
2018-11-26 17:58:48 +02:00
Joseph Schorr
adccdd30ca
Read blobs from new manifest blob table where relevant
2018-11-25 17:31:09 +02:00
Joseph Schorr
c233760007
Fix conversion of schema 2 manifests to schema 1 manifests
...
Also adds a number of conversion tests and clarify the interfaces a bit more
2018-11-21 18:06:16 +02:00
Joseph Schorr
1eaf5b18dd
Adjustments based on code review feedback
2018-11-15 13:51:48 +02:00
Joseph Schorr
c46b11bac1
Implement UI support for manifest lists
2018-11-14 18:40:57 +02:00
Joseph Schorr
37b20010aa
Add support for pushing and pulling schema 2 manifests with remote layers
...
This is required for windows image support
2018-11-14 13:21:50 +02:00
Joseph Schorr
d97055e2ba
Code review fixes
2018-11-14 09:15:58 +02:00
Joseph Schorr
e344d4a5cf
Enhancements for Docker schema implementations in preparing for supporting schema 2 in the OCI model
...
This adds additional required properties and methods to the Docker schema interface to allow us to treat both schema1 and schema2 manifests and lists logically equivalent from the OCI mode perspective
2018-11-12 23:27:01 +02:00
Joseph Schorr
114e2c3bf2
Have all tag code add, modify and delete both old and new style tags
...
This preps us for being able to use the new data model with existing repositories
2018-11-01 17:59:10 -04:00
Joseph Schorr
6a9634dffb
Harden the storage replication worker to failures by explicitly catching certain errors, having better backoff and explicitly ensuring a layer has been replicated to a region before adding the placement into the database
...
Also adds tests for the various failure cases
2018-10-31 14:13:25 -04:00
Joseph Schorr
0ae062be62
Add manifest creation to new registry data model interface
2018-10-18 14:25:39 -04:00
Joseph Schorr
aeceea0f97
Add a worker for backfilling labels on manifests that have already been backfilled
2018-09-26 14:55:14 -04:00
Joseph Schorr
a4fff886a6
Handle the case where the log metadata JSON cannot be parsed
...
This can happen if the JSON overflowed the text field in the table, for example
Fixes https://sentry.io/coreos/backend-production/issues/670957592/
2018-09-07 13:34:22 -04:00
Joseph Schorr
a04658085b
Add configuration flag to control whether we backfill manifests
...
This will allow us to easily turn off the backfill once it is initially complete, but also reenable it if necessary
2018-08-27 17:22:43 -04:00
Joseph Schorr
ed897626a2
Handle data model exceptions when loading parent images in manifest backfill
2018-08-20 11:32:59 -04:00
Joseph Schorr
e30b746aef
Fix TagManifests with shared digests under the same repository.
...
TagManifests can (apparently, in very rare scenarios) share manifests with the exact same digests, so we need to support that case in the backfill worker. We also need to remove a unique constraint on the manifest column in the mapping table to support this case.
2018-08-20 11:32:59 -04:00
Joseph Schorr
fa58f3b1d2
Fix handling of manifests with unicode in the backfill
...
Also adds a bunch of tests around manifests to ensure we get the same information in and out
2018-08-15 11:41:15 -04:00
Joseph Schorr
70df10dade
Change manifest backfill to lookup missing content checksums in the repository
...
After discussion, we decided the best solution for the missing content checksum problem was to lookup the proper blobs in the repository and, if not present, mark the manifest as broken, as this would reflect the actual issue the user faces if they pull the repository tag today via V2
2018-08-14 14:36:19 -04:00
Joseph Schorr
96e0fc4ad6
Fix manifest backfill for manifests pointing to V1 images
...
V1 images don't have checksums, so we just always use the full storage set for the manifest, rather than a checksum map
2018-08-13 15:51:18 -04:00
Joseph Schorr
f2d50b3f8e
Add run commands for backfill worker
2018-08-13 14:56:32 -04:00
Joseph Schorr
03ea3a3250
Add worker to backfill the new manifest tables from the tagmanifest table
2018-08-13 14:56:31 -04:00
Joseph Schorr
1a2e2f54e1
Move storagereplication test under the workers test directory
2018-07-19 15:56:20 -04:00
Sam Chow
84f604739f
Add log rotation threshold configuration
2018-06-29 17:16:44 -04:00
Sam Chow
7f21d0da58
Merge pull request #3096 from quay/refactor/secscan-simple
...
Refactor validators to decouple app
2018-06-01 15:44:18 -04:00
Sam Chow
7df8ed4a60
Add a security scanner api config object for params
...
Change SecScanAPI to use a uri creation func instead of test context
Pass config provider through validator context
Remove app config dependency for validators
2018-06-01 15:06:50 -04:00
Joseph Schorr
52ddf6a308
Fix exception in storage replication worker
...
Fixes https://jira.coreos.com/browse/QUAY-957
2018-05-31 13:58:04 -04:00
josephschorr
58c2ddac98
Merge pull request #3087 from quay/joseph.schorr/QUAY-943/temp-logs-migrate
...
Temporarily change to storing logs in a new LogEntry2 table
2018-05-18 20:35:25 -04:00
Joseph Schorr
a007332d4c
Temporarily change to storing logs in a new LogEntry2 table
...
This will prevent us from running out of auto-incrementing ID values until such time as we can upgrade to peewee 3 and change the field type to a BigInt
Fixes https://jira.coreos.com/browse/QUAY-943
2018-05-18 20:15:16 -04:00
Joseph Schorr
57523d22de
Have Swift storage delete segments when deleting dynamic large objects
...
This ensures that we reclaim the space, rather than simply deleting the manifest
Fixes https://jira.coreos.com/browse/QUAY-942
2018-05-16 16:01:49 -04:00
Joseph Schorr
8bc55a5676
Make namespace deletion asynchronous
...
Instead of deleting a namespace synchronously as before, we now mark the namespace for deletion, disable it, and rename it. A worker then comes along and deletes the namespace in the background. This results in a *significantly* better user experience, as the namespace deletion operation now "completes" in under a second, where before it could take 10s of minutes at the worse.
Fixes https://jira.coreos.com/browse/QUAY-838
2018-02-27 13:12:51 -05:00
Joseph Schorr
d45161b120
Add a worker to automatically GC expired app specific tokens
...
Fixes https://jira.coreos.com/browse/QUAY-822
2018-02-12 14:56:01 -05:00
Joseph Schorr
bbdf9e074c
Add metrics for tracking when instance key renewal succeeds and fails, as well as when instance key *lookup* fails
2018-02-02 11:14:42 -05:00
Joseph Schorr
05b4a7d457
Add worker to update ipresolver data files every few hours
2017-09-28 14:40:59 -04:00
Joseph Schorr
8a4d583f65
Disable default debug logs for workers
...
https://coreosdev.atlassian.net/browse/QUAY-771
2017-08-24 14:25:51 -04:00
Joseph Schorr
74e8bc296e
Fix bug in service key rotation and fix associated flaky test
...
We were using `datetime.now` in both the rotation code and the test, but the model uses `utcnow`.
2017-07-28 14:20:11 -04:00
Joseph Schorr
e7d6e60d97
Update for merge and make additional interface improvements
2017-07-25 17:00:08 -04:00
Joseph Schorr
543cba352b
Add end-to-end notification worker tests for all notification methods
2017-07-25 17:00:07 -04:00
Joseph Schorr
ce56031846
Move notifications into its own package
2017-07-25 17:00:06 -04:00
josephschorr
78652de3ee
Merge pull request #2766 from coreos-inc/joseph.schorr/QUAY-634/buildlogsarchiver-data-interface
...
Change buildlogsarchiver to use a data model interface
2017-07-19 16:40:05 -04:00
josephschorr
9bd4cee029
Merge pull request #2765 from coreos-inc/joseph.schorr/QUAY-629/globalprom-data-interface
...
Switch globalpromstats worker to use a data interface
2017-07-19 16:39:36 -04:00
Joseph Schorr
89fad7568d
Fix temp fix PR for notifications
2017-07-16 12:09:30 +03:00
Joseph Schorr
026a0d28df
Temporary fix for empty event config JSON
2017-07-15 14:01:27 +03:00
Joseph Schorr
bf3e941d7f
Fix notification system to use the new tuple correctly
2017-07-13 22:43:26 +03:00
josephschorr
fdb21aa5dc
Merge pull request #2777 from coreos-inc/joseph.schorr/QUAY-618/notificationworker-data-interface
...
Change notificationworker to use data interface
2017-07-13 00:23:15 +03:00
josephschorr
2206c81a95
Merge pull request #2776 from coreos-inc/joseph.schorr/QUAY-652/servicekeyworker-data-interface
...
Change service key worker to use a data interface
2017-07-13 00:22:49 +03:00
Joseph Schorr
4ed73d247b
yapf
2017-07-12 18:11:51 +03:00
