Charlton Austin
7b3d8e3977
Merge pull request #2183 from charltonaustin/metrics_for_unscanned_images
...
Adding in some metrics around clair sec scan.
2016-12-02 11:50:29 -05:00
Charlton Austin
edd9dcd7f6
Adding in some metrics around clair sec scan.
2016-12-01 16:50:02 -05:00
Joseph Schorr
e6ee538e15
Fix full database test script to not fail randomly
...
- Switches database schema creation to alembic, which solves the MySQL issue (and makes sure we test migrations as well)
- Adds a few time.sleep(1) to work around MySQL's second-precision issue when adding items to queues and then immediately retrieving them
- Disables the storage proxy tests when running against non-SQLite databases, as it causes failures with the multiple process and multiple transactions
- Changes initdb to support only populating the database, as well as fixing a few small items around the test data when working with non-SQLite data
2016-11-30 18:24:08 -05:00
Joseph Schorr
e29cb34336
Fix Set calls to gauges
...
Fixes #2150
The proper function is `Set` (not `set`), which was causing these gauges to not report to Prometheus
2016-11-21 15:27:17 -05:00
Joseph Schorr
5f99448adc
Add a chunk cleanup queue for async GC of empty chunks
...
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
2016-11-15 15:07:41 -05:00
Jimmy Zelinskie
8b9f9478a4
pylint formatting
2016-10-28 17:12:46 -04:00
Jimmy Zelinskie
a30b358709
add staggered worker startup
...
Fixes #787
2016-10-28 17:12:39 -04:00
Jimmy Zelinskie
2bd1e76267
workers.queuecleanup: s/week/day cleanup frequency
2016-10-20 13:47:07 -04:00
Jimmy Zelinskie
20ef43d5fb
workers.queuecleanup: remove direct peewee usage
2016-10-20 13:46:00 -04:00
Joseph Schorr
30af8aef1a
Add a worker for reporting global stats to Prometheus
...
Fixes #1789
2016-09-12 16:19:19 -04:00
josephschorr
5c64646629
Merge pull request #1778 from coreos-inc/redlock
...
Fix locking via RedLock
2016-08-29 16:12:01 -04:00
Joseph Schorr
aa7c87d765
Fix locking via RedLock
...
Fixes #1777
2016-08-29 16:06:26 -04:00
Joseph Schorr
08a3b70b56
Extend processing before processing security notifications
...
Makes sure queue items don't expire during processing
Fixes #1776
2016-08-29 13:08:38 -04:00
Jake Moshenko
a113f548db
Accidentally forgot a line in the gc worker.
2016-08-02 10:44:53 -04:00
Jake Moshenko
05e2773fa7
Get rid of remaining slow query for garbage collection.
2016-08-01 18:22:38 -04:00
Joseph Schorr
b8d2570725
Don't raise an error on duplicate placements
...
This can happen if two pushes are racing on the same storage.
2016-07-19 16:44:05 -04:00
Joseph Schorr
5cd793331e
Fix storage replication for CAS and add tests
2016-07-12 13:46:06 -04:00
Joseph Schorr
3b994431eb
Auto expire the build status and logs in redis
2016-06-20 13:53:13 -04:00
Jake Moshenko
a1cf12e460
Add a sitemap.txt for popular public repos
...
and reference it from the robots.txt
2016-06-17 14:34:20 -04:00
Joseph Schorr
8887f09ba8
Use the instance service key for registry JWT signing
2016-06-07 11:58:10 -04:00
Joseph Schorr
dd0dd39bf0
Fix the queue cleanup worker to delete the items that have expired, not unexpired
2016-06-03 22:14:14 -04:00
Joseph Schorr
5746b42c69
Add a cleanup worker for the queue item table
...
Fixes #784
2016-06-02 15:00:44 -04:00
josephschorr
ec492bb683
Merge pull request #1323 from coreos-inc/secworkerreturn
...
Move security notification work into its own method to allow for retu…
2016-06-02 13:59:25 -04:00
Jake Moshenko
9221a515de
Use the registry API for security scanning
...
when the storage engine doesn't support direct download url
2016-05-04 18:04:06 -04:00
Joseph Schorr
73fa593d02
Various small fixes in prep for QE release
2016-05-04 15:20:27 -04:00
Jimmy Zelinskie
f842545b3e
rename config values to remove "Quay" ( #1431 )
2016-05-03 13:11:21 -04:00
Evan Cordell
489752a0b7
Only refresh current instance service key
2016-04-29 14:10:33 -04:00
Evan Cordell
a6f6a114c2
service key worker to refresh automatic keys
2016-04-29 14:10:33 -04:00
Jimmy Zelinskie
128b0cd38c
logrotateworker: archive every 24 hours
2016-04-18 13:02:30 -04:00
Jimmy Zelinskie
ef65822410
logrotateworker: perf optimizations
...
This removes our needless transaction, only calculates the cutoff date
once, removes the logs generator, and uses a tested optimal
MIN_LOGS_PER_ROTATION.
2016-04-15 16:51:17 -04:00
Jimmy Zelinskie
3d190b786f
userfiles: make handler optional
2016-04-15 13:56:07 -04:00
Jimmy Zelinskie
c7c52e6c74
logrotateworker: save to storage via userfiles
2016-04-14 13:29:29 -04:00
Joseph Schorr
d62ec22fc9
Move security notification work into its own method to allow for return values
...
Fixes #1302
Fixes #1304
2016-03-31 14:08:33 -04:00
Joseph Schorr
dc8f9713f8
Change logs worker to use a global lock in the inner loop and move storage out of the transaction
2016-03-24 14:09:48 -04:00
Joseph Schorr
aa5587c93c
Fixes and added tests for the security notification worker
...
Fixes #1301
- Ensures that the worker uses pagination properly
- Ensures that the worker handles failure as expected
- Moves marking the notification as read to after the worker processes it
- Increases the number of layers requested to 100
2016-03-18 20:28:06 -04:00
Quentin Machu
5b7d6b0638
Merge pull request #1275 from Quentin-M/min_id_once
...
Compute min_id only once during securityworker's lifetime
2016-03-04 14:02:47 -05:00
Quentin Machu
54153c9b80
Compute min_id only once during securityworker's lifetime
2016-03-04 14:02:28 -05:00
Jimmy Zelinskie
b5d904f373
Merge pull request #1218 from jzelinskie/logrotate5ever
...
vastly simplify log rotation
2016-03-04 13:48:21 -05:00
Quentin Machu
888f976e8d
Use a feature flag to toggle security notifications
2016-03-01 15:54:18 -05:00
Joseph Schorr
f498e92d58
Implement against new Clair paginated notification system
2016-02-25 15:58:42 -05:00
Joseph Schorr
c0374d71c9
Refactor the security worker and API calls and add a bunch of tests
2016-02-25 12:29:41 -05:00
Quentin Machu
e5da33578c
Adapt security worker for Clair v1.0 (except notifications)
2016-02-19 17:44:14 -05:00
Quentin Machu
f62a05f6d7
various securityworker fixes
2016-02-09 21:25:07 -05:00
Quentin Machu
1d2b31a581
Mark layers that Clair can't extract as failed
2016-02-09 18:24:35 -05:00
Jimmy Zelinskie
ee705fe7a9
vastly simplify log rotation
2016-02-09 18:20:14 -05:00
Quentin Machu
13c10ba7b1
Double the securityworker indexing interval
2016-02-09 14:49:10 -05:00
Joseph Schorr
ab166c4448
Delete the image diff feature
...
Fixes #1077
2015-12-23 13:08:01 -05:00
Jimmy Zelinskie
f439ad7804
Merge pull request #618 from jzelinskie/logsworker
...
add a log rotation worker
2015-12-16 17:25:50 -05:00
Jimmy Zelinskie
e1f955a3f6
add a log rotation worker
...
Fixes #609 .
2015-12-16 17:22:28 -05:00
Joseph Schorr
c888a8b3be
Make GC timeout configurable
2015-12-16 15:45:02 -05:00
