vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
6353 commits 2 branches 62 tags 205 MiB
Commit graph

253 commits

Author SHA1 Message Date
Charlton Austin
9e25fde3a0 Fixing api usage. 2016-12-07 12:53:07 -05:00
Jimmy Zelinskie
3a7119d499 Merge pull request #2209 from coreos-inc/clair-notification-read 
Clair notification read and queue fixes
 2016-12-05 19:36:59 -05:00
Joseph Schorr
9f0ce7c634 Have the security worker remove failed notifications from Clair 2016-12-05 19:08:52 -05:00
Jake Moshenko
c263772703 Do not extend processing immediately after taking queue item. 2016-12-05 18:12:14 -05:00
Jake Moshenko
709edd7eb6 Reduce the update period on queue worker metrics. 2016-12-05 18:12:14 -05:00
Quentin Machu
b990a27d50 Increase limit in securitynotificationworker 
With https://github.com/coreos/clair/pull/278 and https://github.com/coreos/clair/pull/279, performance of this API call has increased. It has been observed that querying 100 or 1000 layers page doesn't noticeably change the execution time. Therefore, doing significantly less calls will reduce the overall processing time for each notification.
 2016-12-04 13:39:34 +01:00
Charlton Austin
7b3d8e3977 Merge pull request #2183 from charltonaustin/metrics_for_unscanned_images 
Adding in some metrics around clair sec scan.
 2016-12-02 11:50:29 -05:00
Charlton Austin
edd9dcd7f6 Adding in some metrics around clair sec scan. 2016-12-01 16:50:02 -05:00
Joseph Schorr
e6ee538e15 Fix full database test script to not fail randomly 
- Switches database schema creation to alembic, which solves the MySQL issue (and makes sure we test migrations as well)
- Adds a few time.sleep(1) to work around MySQL's second-precision issue when adding items to queues and then immediately retrieving them
- Disables the storage proxy tests when running against non-SQLite databases, as it causes failures with the multiple process and multiple transactions
- Changes initdb to support only populating the database, as well as fixing a few small items around the test data when working with non-SQLite data
 2016-11-30 18:24:08 -05:00
Joseph Schorr
e29cb34336 Fix Set calls to gauges 
Fixes #2150

The proper function is `Set` (not `set`), which was causing these gauges to not report to Prometheus
 2016-11-21 15:27:17 -05:00
Joseph Schorr
5f99448adc Add a chunk cleanup queue for async GC of empty chunks 
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
 2016-11-15 15:07:41 -05:00
Jimmy Zelinskie
8b9f9478a4 pylint formatting 2016-10-28 17:12:46 -04:00
Jimmy Zelinskie
a30b358709 add staggered worker startup 
Fixes #787
 2016-10-28 17:12:39 -04:00
Jimmy Zelinskie
2bd1e76267 workers.queuecleanup: s/week/day cleanup frequency 2016-10-20 13:47:07 -04:00
Jimmy Zelinskie
20ef43d5fb workers.queuecleanup: remove direct peewee usage 2016-10-20 13:46:00 -04:00
Joseph Schorr
30af8aef1a Add a worker for reporting global stats to Prometheus 
Fixes #1789
 2016-09-12 16:19:19 -04:00
josephschorr
5c64646629 Merge pull request #1778 from coreos-inc/redlock 
Fix locking via RedLock
 2016-08-29 16:12:01 -04:00
Joseph Schorr
aa7c87d765 Fix locking via RedLock 
Fixes #1777
 2016-08-29 16:06:26 -04:00
Joseph Schorr
08a3b70b56 Extend processing before processing security notifications 
Makes sure queue items don't expire during processing

Fixes #1776
 2016-08-29 13:08:38 -04:00
Jake Moshenko
a113f548db Accidentally forgot a line in the gc worker. 2016-08-02 10:44:53 -04:00
Jake Moshenko
05e2773fa7 Get rid of remaining slow query for garbage collection. 2016-08-01 18:22:38 -04:00
Joseph Schorr
b8d2570725 Don't raise an error on duplicate placements 
This can happen if two pushes are racing on the same storage.
 2016-07-19 16:44:05 -04:00
Joseph Schorr
5cd793331e Fix storage replication for CAS and add tests 2016-07-12 13:46:06 -04:00
Joseph Schorr
3b994431eb Auto expire the build status and logs in redis 2016-06-20 13:53:13 -04:00
Jake Moshenko
a1cf12e460 Add a sitemap.txt for popular public repos 
and reference it from the robots.txt
 2016-06-17 14:34:20 -04:00
Joseph Schorr
8887f09ba8 Use the instance service key for registry JWT signing 2016-06-07 11:58:10 -04:00
Joseph Schorr
dd0dd39bf0 Fix the queue cleanup worker to delete the items that have expired, not unexpired 2016-06-03 22:14:14 -04:00
Joseph Schorr
5746b42c69 Add a cleanup worker for the queue item table 
Fixes #784
 2016-06-02 15:00:44 -04:00
josephschorr
ec492bb683 Merge pull request #1323 from coreos-inc/secworkerreturn 
Move security notification work into its own method to allow for retu…
 2016-06-02 13:59:25 -04:00
Jake Moshenko
9221a515de Use the registry API for security scanning 
when the storage engine doesn't support direct download url
 2016-05-04 18:04:06 -04:00
Joseph Schorr
73fa593d02 Various small fixes in prep for QE release 2016-05-04 15:20:27 -04:00
Jimmy Zelinskie
f842545b3e rename config values to remove "Quay" (#1431) 2016-05-03 13:11:21 -04:00
Evan Cordell
489752a0b7 Only refresh current instance service key 2016-04-29 14:10:33 -04:00
Evan Cordell
a6f6a114c2 service key worker to refresh automatic keys 2016-04-29 14:10:33 -04:00
Jimmy Zelinskie
128b0cd38c logrotateworker: archive every 24 hours 2016-04-18 13:02:30 -04:00
Jimmy Zelinskie
ef65822410 logrotateworker: perf optimizations 
This removes our needless transaction, only calculates the cutoff date
once, removes the logs generator, and uses a tested optimal
MIN_LOGS_PER_ROTATION.
 2016-04-15 16:51:17 -04:00
Jimmy Zelinskie
3d190b786f userfiles: make handler optional 2016-04-15 13:56:07 -04:00
Jimmy Zelinskie
c7c52e6c74 logrotateworker: save to storage via userfiles 2016-04-14 13:29:29 -04:00
Joseph Schorr
d62ec22fc9 Move security notification work into its own method to allow for return values 
Fixes #1302
Fixes #1304
 2016-03-31 14:08:33 -04:00
Joseph Schorr
dc8f9713f8 Change logs worker to use a global lock in the inner loop and move storage out of the transaction 2016-03-24 14:09:48 -04:00
Joseph Schorr
aa5587c93c Fixes and added tests for the security notification worker 
Fixes #1301

- Ensures that the worker uses pagination properly
- Ensures that the worker handles failure as expected
- Moves marking the notification as read to after the worker processes it
- Increases the number of layers requested to 100
 2016-03-18 20:28:06 -04:00
Quentin Machu
5b7d6b0638 Merge pull request #1275 from Quentin-M/min_id_once 
Compute min_id only once during securityworker's lifetime
 2016-03-04 14:02:47 -05:00
Quentin Machu
54153c9b80 Compute min_id only once during securityworker's lifetime 2016-03-04 14:02:28 -05:00
Jimmy Zelinskie
b5d904f373 Merge pull request #1218 from jzelinskie/logrotate5ever 
vastly simplify log rotation
 2016-03-04 13:48:21 -05:00
Quentin Machu
888f976e8d Use a feature flag to toggle security notifications 2016-03-01 15:54:18 -05:00
Joseph Schorr
f498e92d58 Implement against new Clair paginated notification system 2016-02-25 15:58:42 -05:00
Joseph Schorr
c0374d71c9 Refactor the security worker and API calls and add a bunch of tests 2016-02-25 12:29:41 -05:00
Quentin Machu
e5da33578c Adapt security worker for Clair v1.0 (except notifications) 2016-02-19 17:44:14 -05:00
Quentin Machu
f62a05f6d7 various securityworker fixes 2016-02-09 21:25:07 -05:00
Quentin Machu
1d2b31a581 Mark layers that Clair can't extract as failed 2016-02-09 18:24:35 -05:00