Antoine Legrand
f0dd2e348b
Merge pull request #2551 from coreos-inc/structured-logs
...
Add log formatter class
2017-06-07 08:22:18 -07:00
Antoine Legrand
3c99928a27
Add log JSON formatter
2017-06-07 00:02:52 +02:00
Kenny Lee Sin Cheong
ad1a0e0840
logger.exception dumps a stack trace by default
2017-06-02 17:21:40 -04:00
Kenny Lee Sin Cheong
3302a96f88
Log the APIRequestFailure at ERROR level
2017-06-02 14:49:50 -04:00
Kenny Lee Sin Cheong
b5f8e7e24d
Returning from the method instead of calling sleep
...
Simply returning from the method will give DEFAULT_INDEXING_INTERVAL seconds
before the next scan operation.
2017-06-02 12:28:17 -04:00
Kenny Lee Sin Cheong
203c0b76e0
Raise an APIRequestFailure exception when security scanner is unavailable
...
Put worker to sleep for the duration of the default indexing interval
when an APIRequestFailure occurs, when the API request fails due to a
connection error, timeout, or other ambiguous errors, from
analyze_layer or get_layer_data .
2017-05-24 11:04:44 -04:00
Jimmy Zelinskie
c6f6204630
workers.securityworker: small fixes
...
This change adjusts our batch size to coerce to integer after all
floating point math in order to get a more accurate end result. In
addition, we handle the scenario when there are no longer any images in
the database to be scanned when finding the min id.
2017-03-13 18:22:35 -04:00
Jimmy Zelinskie
a780136337
workers.securityworker: revert to image querying
2017-03-10 17:37:40 -05:00
Jimmy Zelinskie
40636d4103
find work based on tag IDs rather than image IDs
2017-03-06 17:09:57 -05:00
Jimmy Zelinskie
904b902295
workers.securityworker: find eligible tag images
2017-03-06 14:37:34 -05:00
Jimmy Zelinskie
b9ac2b7b3b
workers.securityworker: simplify min id
2017-03-03 14:51:18 -05:00
Jimmy Zelinskie
4ed0cdda14
securityscanner: add a min image id option
...
This will enable us to force some instances of the securityworker to
scan only new images.
2017-03-03 13:55:25 -05:00
Jake Moshenko
de7a5c9959
Make the security scanning worker period configurable
2017-02-27 15:02:29 -05:00
Joseph Schorr
407341fe96
Remove images count (which is horribly slow in InnoDB) and add a max gauge
2017-02-23 17:37:28 -05:00
Jake Moshenko
27f5f14f90
Linter fixes
2017-02-22 11:45:38 -05:00
Jake Moshenko
add6b654ae
Move the total image count stat back to the prom stat worker
2017-02-22 11:45:38 -05:00
Jake Moshenko
b03e03c389
Read the number of unscanned clair images from the block allocator
2017-02-21 19:13:51 -05:00
Joseph Schorr
405eca074c
Security scanner flow changes and auto-retry
...
Changes the security scanner code to raise exceptions now for non-successful operations. One of the new exceptions raised is MissingParentLayerException, which, when raised, will cause the security worker to perform a full rescan of all parent images for the current layer, before trying once more to scan the current layer. This should allow the system to be "self-healing" in the case where the security scanner engine somehow loses or corrupts a parent layer.
2016-12-16 15:38:09 -05:00
Charlton Austin
edd9dcd7f6
Adding in some metrics around clair sec scan.
2016-12-01 16:50:02 -05:00
Jimmy Zelinskie
8b9f9478a4
pylint formatting
2016-10-28 17:12:46 -04:00
Jake Moshenko
9221a515de
Use the registry API for security scanning
...
when the storage engine doesn't support direct download url
2016-05-04 18:04:06 -04:00
Joseph Schorr
73fa593d02
Various small fixes in prep for QE release
2016-05-04 15:20:27 -04:00
Quentin Machu
54153c9b80
Compute min_id only once during securityworker's lifetime
2016-03-04 14:02:28 -05:00
Joseph Schorr
c0374d71c9
Refactor the security worker and API calls and add a bunch of tests
2016-02-25 12:29:41 -05:00
Quentin Machu
e5da33578c
Adapt security worker for Clair v1.0 (except notifications)
2016-02-19 17:44:14 -05:00
Quentin Machu
f62a05f6d7
various securityworker fixes
2016-02-09 21:25:07 -05:00
Quentin Machu
1d2b31a581
Mark layers that Clair can't extract as failed
2016-02-09 18:24:35 -05:00
Quentin Machu
13c10ba7b1
Double the securityworker indexing interval
2016-02-09 14:49:10 -05:00
Jake Moshenko
2f626f2691
Unify the database connection lifecycle across all workers
2015-12-04 15:51:53 -05:00
Silas Sewell
1162814734
securityworker: mark children we can't analyze
...
This allows us to differentiate between images that are queued and those we
can't analyze in constant time.
2015-11-19 11:22:15 -05:00
Quentin Machu
88e85cded0
Fix security worker (again?)
2015-11-18 19:45:09 -05:00
Quentin Machu
7e9faa6c54
Add missing import
2015-11-18 17:39:27 -05:00
Quentin Machu
605ed1fc77
Refactor security worker
2015-11-18 14:38:32 -05:00
Joseph Schorr
6412e145dd
Fix key error
2015-11-13 13:16:33 -05:00
Jimmy Zelinskie
09ce33e0dc
fix case where query broke on empty list
2015-11-13 12:35:18 -05:00
Joseph Schorr
927a0b639c
Add check for empty locations list
2015-11-13 12:23:02 -05:00
Joseph Schorr
030c69d7d2
Further merge fixes
2015-11-12 22:00:28 -05:00
Joseph Schorr
7816b0c657
Merge master into vulnerability-tool
2015-11-12 21:52:47 -05:00
Joseph Schorr
25b8b7590f
Fix all the things!
2015-11-12 20:55:41 -05:00
Jimmy Zelinskie
37ce84f6af
tiny fixes to securityworker
2015-11-12 17:18:04 -05:00
Jimmy Zelinskie
f6a34c5d06
refactor securityworker
...
Fixes #772 .
2015-11-12 16:03:10 -05:00
Jimmy Zelinskie
8e2868737b
rename secscan_endpoint and move db close to API
2015-11-10 15:22:31 -05:00
Jimmy Zelinskie
da31714fb5
specify securityworker skip message
2015-11-10 15:22:30 -05:00
Quentin Machu
16c364a90c
Rename secscan_endpoint where required, fix index and indentation
2015-11-09 15:18:42 -05:00
Joseph Schorr
2d2662f53f
Fix deleting repos and images under MySQL
...
MySQL doesn't handle constraints at the end of transactions, so deleting images currently fails. This removes the constraint and just leaves parent_id as an int
2015-11-09 14:42:05 -05:00
Quentin Machu
7dbe15e339
Remove checksum from Clair's worker and adjust line length
2015-11-09 14:31:24 -05:00
Joseph Schorr
b408cfd2cc
Ready for demo
2015-11-09 12:51:05 -05:00
Joseph Schorr
7fa4fe08e7
Fix worker
2015-11-09 12:50:39 -05:00
Joseph Schorr
407eaae137
WIP: Towards sec demo
2015-11-09 12:50:39 -05:00
Quentin Machu
37118423a5
Add support for Quay's vulnerability tool
2015-11-09 12:49:19 -05:00
