vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
8810 commits 2 branches 62 tags 205 MiB
Commit graph

354 commits

Author SHA1 Message Date
Joseph Schorr
a04658085b Add configuration flag to control whether we backfill manifests 
This will allow us to easily turn off the backfill once it is initially complete, but also reenable it if necessary
 2018-08-27 17:22:43 -04:00
Joseph Schorr
ed897626a2 Handle data model exceptions when loading parent images in manifest backfill 2018-08-20 11:32:59 -04:00
Joseph Schorr
e30b746aef Fix TagManifests with shared digests under the same repository. 
TagManifests can (apparently, in very rare scenarios) share manifests with the exact same digests, so we need to support that case in the backfill worker. We also need to remove a unique constraint on the manifest column in the mapping table to support this case.
 2018-08-20 11:32:59 -04:00
Joseph Schorr
fa58f3b1d2 Fix handling of manifests with unicode in the backfill 
Also adds a bunch of tests around manifests to ensure we get the same information in and out
 2018-08-15 11:41:15 -04:00
Joseph Schorr
70df10dade Change manifest backfill to lookup missing content checksums in the repository 
After discussion, we decided the best solution for the missing content checksum problem was to lookup the proper blobs in the repository and, if not present, mark the manifest as broken, as this would reflect the actual issue the user faces if they pull the repository tag today via V2
 2018-08-14 14:36:19 -04:00
Joseph Schorr
96e0fc4ad6 Fix manifest backfill for manifests pointing to V1 images 
V1 images don't have checksums, so we just always use the full storage set for the manifest, rather than a checksum map
 2018-08-13 15:51:18 -04:00
Joseph Schorr
f2d50b3f8e Add run commands for backfill worker 2018-08-13 14:56:32 -04:00
Joseph Schorr
03ea3a3250 Add worker to backfill the new manifest tables from the tagmanifest table 2018-08-13 14:56:31 -04:00
Joseph Schorr
1a2e2f54e1 Move storagereplication test under the workers test directory 2018-07-19 15:56:20 -04:00
Sam Chow
84f604739f Add log rotation threshold configuration 2018-06-29 17:16:44 -04:00
Sam Chow
7f21d0da58
Merge pull request #3096 from quay/refactor/secscan-simple 
Refactor validators to decouple app
 2018-06-01 15:44:18 -04:00
Sam Chow
7df8ed4a60 Add a security scanner api config object for params 
Change SecScanAPI to use a uri creation func instead of test context

Pass config provider through validator context

Remove app config dependency for validators
 2018-06-01 15:06:50 -04:00
Joseph Schorr
52ddf6a308 Fix exception in storage replication worker 
Fixes https://jira.coreos.com/browse/QUAY-957
 2018-05-31 13:58:04 -04:00
josephschorr
58c2ddac98
Merge pull request #3087 from quay/joseph.schorr/QUAY-943/temp-logs-migrate 
Temporarily change to storing logs in a new LogEntry2 table
 2018-05-18 20:35:25 -04:00
Joseph Schorr
a007332d4c Temporarily change to storing logs in a new LogEntry2 table 
This will prevent us from running out of auto-incrementing ID values until such time as we can upgrade to peewee 3 and change the field type to a BigInt

Fixes https://jira.coreos.com/browse/QUAY-943
 2018-05-18 20:15:16 -04:00
Joseph Schorr
57523d22de Have Swift storage delete segments when deleting dynamic large objects 
This ensures that we reclaim the space, rather than simply deleting the manifest

Fixes https://jira.coreos.com/browse/QUAY-942
 2018-05-16 16:01:49 -04:00
Joseph Schorr
8bc55a5676 Make namespace deletion asynchronous 
Instead of deleting a namespace synchronously as before, we now mark the namespace for deletion, disable it, and rename it. A worker then comes along and deletes the namespace in the background. This results in a *significantly* better user experience, as the namespace deletion operation now "completes" in under a second, where before it could take 10s of minutes at the worse.

Fixes https://jira.coreos.com/browse/QUAY-838
 2018-02-27 13:12:51 -05:00
Joseph Schorr
d45161b120 Add a worker to automatically GC expired app specific tokens 
Fixes https://jira.coreos.com/browse/QUAY-822
 2018-02-12 14:56:01 -05:00
Joseph Schorr
bbdf9e074c Add metrics for tracking when instance key renewal succeeds and fails, as well as when instance key *lookup* fails 2018-02-02 11:14:42 -05:00
Joseph Schorr
05b4a7d457 Add worker to update ipresolver data files every few hours 2017-09-28 14:40:59 -04:00
Joseph Schorr
8a4d583f65 Disable default debug logs for workers 
https://coreosdev.atlassian.net/browse/QUAY-771
 2017-08-24 14:25:51 -04:00
Joseph Schorr
74e8bc296e Fix bug in service key rotation and fix associated flaky test 
We were using `datetime.now` in both the rotation code and the test, but the model uses `utcnow`.
 2017-07-28 14:20:11 -04:00
Joseph Schorr
e7d6e60d97 Update for merge and make additional interface improvements 2017-07-25 17:00:08 -04:00
Joseph Schorr
543cba352b Add end-to-end notification worker tests for all notification methods 2017-07-25 17:00:07 -04:00
Joseph Schorr
ce56031846 Move notifications into its own package 2017-07-25 17:00:06 -04:00
josephschorr
78652de3ee Merge pull request #2766 from coreos-inc/joseph.schorr/QUAY-634/buildlogsarchiver-data-interface 
Change buildlogsarchiver to use a data model interface
 2017-07-19 16:40:05 -04:00
josephschorr
9bd4cee029 Merge pull request #2765 from coreos-inc/joseph.schorr/QUAY-629/globalprom-data-interface 
Switch globalpromstats worker to use a data interface
 2017-07-19 16:39:36 -04:00
Joseph Schorr
89fad7568d Fix temp fix PR for notifications 2017-07-16 12:09:30 +03:00
Joseph Schorr
026a0d28df Temporary fix for empty event config JSON 2017-07-15 14:01:27 +03:00
Joseph Schorr
bf3e941d7f Fix notification system to use the new tuple correctly 2017-07-13 22:43:26 +03:00
josephschorr
fdb21aa5dc Merge pull request #2777 from coreos-inc/joseph.schorr/QUAY-618/notificationworker-data-interface 
Change notificationworker to use data interface
 2017-07-13 00:23:15 +03:00
josephschorr
2206c81a95 Merge pull request #2776 from coreos-inc/joseph.schorr/QUAY-652/servicekeyworker-data-interface 
Change service key worker to use a data interface
 2017-07-13 00:22:49 +03:00
Joseph Schorr
4ed73d247b yapf 2017-07-12 18:11:51 +03:00
Joseph Schorr
b6f1782642 Change notificationworker to use a data interface 2017-07-12 17:40:45 +03:00
Joseph Schorr
8ec198228c Change notificationworker test to pytest 2017-07-12 17:35:09 +03:00
Joseph Schorr
50c2f1fde8 Move notification worker test under its own package 2017-07-12 17:35:09 +03:00
Joseph Schorr
fbfd78532c Move notification worker to its own package 2017-07-12 17:35:09 +03:00
Joseph Schorr
932db23a5c Change servicekeyworker to use a data interface 2017-07-12 16:37:32 +03:00
Joseph Schorr
0afc222214 Add basic unit test for the servicekeyworker 2017-07-12 16:19:30 +03:00
Joseph Schorr
3b496e2759 Move serverkeyworker into its own package 2017-07-12 15:57:02 +03:00
Joseph Schorr
e2cf2d6f2b Move teamsyncworker into its own package 2017-07-12 15:53:01 +03:00
josephschorr
dc6c6b30fc Merge pull request #2768 from coreos-inc/joseph.schorr/QUAY-653/blobuploadcleanupworker-data-interface 
Change blobuploadcleanupworker to use a data interface
 2017-07-12 00:32:09 +03:00
josephschorr
96d1fd128d Merge pull request #2757 from coreos-inc/joseph.schorr/QUAY-606/logarchive-georep 
Add support for QE customers to enable log rotation
 2017-07-12 00:30:04 +03:00
Joseph Schorr
8ded8f573d yapf 2017-07-11 16:58:30 +03:00
Joseph Schorr
bdab367285 Change blobuploadcleanupworker to use a data interface 2017-07-11 16:58:09 +03:00
Joseph Schorr
b2053829f9 Add a basic test for blob upload cleanup 2017-07-11 16:35:10 +03:00
Joseph Schorr
b87415129f Move blobuploadcleanupworker into its own package 2017-07-11 15:38:10 +03:00
Joseph Schorr
c7f1944bd5 yapf 2017-07-11 15:33:48 +03:00
Joseph Schorr
8ba71f7a45 Change buildlogsarchiver to use a data interface 2017-07-11 15:33:28 +03:00
Joseph Schorr
b7a2a4390b Add a basic build logs archiver test 2017-07-11 15:12:34 +03:00