vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
7529 commits 2 branches 62 tags 205 MiB
Commit graph

720 commits

Author SHA1 Message Date
Evan Cordell
217b4a5ab2 Return hashes and expiration when fetching signed tags 2017-04-07 16:12:28 -04:00
Joseph Schorr
ed3da4697f Add client ID and client secret to OIDC config validator 2017-04-07 11:33:02 -04:00
Jake Moshenko
c7241911a5 Fix old-style flask imports to silence deprecation warnings. 2017-04-06 13:15:48 -04:00
Jake Moshenko
a0817bfd59 Refresh dependencies and fix tests. 2017-04-06 13:15:48 -04:00
Evan Cordell
9515f18fb6 Add tufmetadata endpoint 2017-04-05 10:03:27 -04:00
Joseph Schorr
0b6c062e32 Add superuser panel config for team syncing 2017-04-03 11:31:30 -04:00
Joseph Schorr
a6486b7823 Gitlab validation must allow unspecified endpoint 
Gitlab config validator currently requires the gitlab endpoint to be specified, even though we support leaving it unspecified for non-enterprise installs. Fix the validator to allow this case.
 2017-03-30 12:57:41 -04:00
Joseph Schorr
45179216af Have sec scan retries actually work 
Until this change, if `ping` raised an exception, we wouldn't retry properly
 2017-03-29 16:19:46 -04:00
Jimmy Zelinskie
65a17dc155 Merge pull request #2473 from coreos-inc/certs-fixes 
Fixes and improvements around custom certificate handling
 2017-03-27 15:08:36 -04:00
Evan Cordell
1016641f8d refactor jwt context building 2017-03-27 11:37:17 -04:00
Evan Cordell
abd78bce56 Use constants for TUF roots 2017-03-27 11:37:17 -04:00
Evan Cordell
6ad107709c Change build_context_and_subject to take kwargs 2017-03-27 11:37:17 -04:00
Evan Cordell
43dd974dca Determine which TUF root to show based on actual access, not requested 
access
 2017-03-27 11:37:17 -04:00
Joseph Schorr
b017133cc6 Make QSS validation errors more descriptive 2017-03-24 17:28:16 -04:00
Jimmy Zelinskie
23759a1592 util.config.db: ensure blob locations sync on boot 2017-03-22 22:57:21 -04:00
Joseph Schorr
6ab5b8be45 Have storage replication backfill tool only backfill missing storages 
Prevents overload of the queue
 2017-03-22 11:30:49 -04:00
Joseph Schorr
6476488221 Skip bitbucket pushes without any commits 
Fixes https://sentry.io/coreos/backend-production/issues/178220183/
 2017-03-20 18:23:21 -04:00
josephschorr
432b2d3fe8 Merge pull request #2392 from coreos-inc/search-optimization 
Optimize repository search by changing our lookup strategy
 2017-03-10 15:44:26 -05:00
josephschorr
6d6be63ca6 Merge pull request #2393 from coreos-inc/oidc-ui 
OIDC configuration support in superuser config panel
 2017-03-10 12:13:48 -05:00
Joseph Schorr
b5bb76cdea Optimize repository search by changing our lookup strategy 
Previous to this change, repositories were looked up unfiltered in six different queries, and then filtered using the permissions model, which issued a query per repository found, making search incredibly slow. Instead, we now lookup a chunk of repositories unfiltered and then filter them via a single query to the database. By layering the filtering on top of the lookup, each as queries, we can minimize the number of queries necessary, without (at the same time) using a super expensive join.

Other changes:
- Remove the 5 page pre-lookup on V1 search and simply return that there is one more page available, until there isn't. While technically not correct, it is much more efficient, and no one should be using pagination with V1 search anyway.
- Remove the lookup for repos without entries in the RAC table. Instead, we now add a new RAC entry when the repository is created for *the day before*, with count 0, so that it is immediately searchable
- Remove lookup of results with a matching namespace; these aren't very relevant anyway, and it overly complicates sorting
 2017-03-09 19:47:55 -05:00
Joseph Schorr
eff1827d9d Batch QSS notifications after initial scan 2017-03-01 15:42:49 -05:00
Jimmy Zelinskie
cbb2fff0e2 util.secscan.api: raise exception for !200 status 2017-03-01 00:40:47 -05:00
Jimmy Zelinskie
cba7816caf util.failover: re-raise exceptions on failure 2017-03-01 00:40:47 -05:00
Joseph Schorr
157640e696 Add config validator for OIDC logins 2017-02-28 16:18:19 -05:00
Joseph Schorr
88b808f468 Fix typo 2017-02-24 12:23:18 -05:00
Joseph Schorr
d4eb4f7f3c Pull out github trigger and login validation into validator class 2017-02-24 12:23:18 -05:00
Joseph Schorr
a31f2267e8 Pull out gitlab trigger validation into validator class 2017-02-24 12:23:18 -05:00
Joseph Schorr
7a260d81d3 Pull out bitbucket trigger validation into validator class 2017-02-24 12:23:17 -05:00
Joseph Schorr
49638b081b Pull out google login validation into validator class 2017-02-24 12:23:17 -05:00
Joseph Schorr
620e377faf Pull out ssl validation into validator class 2017-02-24 12:23:17 -05:00
Joseph Schorr
e76b95f0e6 Add S3 storage test to validator tests 2017-02-24 12:23:17 -05:00
Joseph Schorr
09b3cfd549 Pull out torrent validation into validator class 2017-02-24 12:23:17 -05:00
Joseph Schorr
2944a4e13d Pull out signing validation into validator class 2017-02-24 12:23:17 -05:00
Joseph Schorr
8844ecbb7c Fix imports 2017-02-24 12:23:16 -05:00
Joseph Schorr
dcabb36ac7 Add TODO 2017-02-24 12:23:16 -05:00
Joseph Schorr
3db4c15459 Pull out security scanner validation into validator class 2017-02-24 12:23:16 -05:00
Joseph Schorr
c0f7530b29 Pull out JWT auth validation into validator class 
Also fixes a small bug in validation (yay tests!)
 2017-02-24 12:23:16 -05:00
Joseph Schorr
678f868bc4 Pull out keystone validation into validator class 2017-02-24 12:23:15 -05:00
Joseph Schorr
c55ddf7341 Pull out ldap validation into validator class 2017-02-24 12:23:15 -05:00
Joseph Schorr
2d64cf3000 Rename config validation source files 2017-02-24 12:23:15 -05:00
Joseph Schorr
00eceb7ed5 Pull out email validation into validator class 2017-02-24 12:23:15 -05:00
Joseph Schorr
ee4f5ed5d6 Move registry storage validator to new location 2017-02-24 12:23:15 -05:00
Joseph Schorr
b2afe68632 Pull out redis validation into validator class 2017-02-24 12:23:15 -05:00
Joseph Schorr
f933b3e295 Pull out database validation into validator class 2017-02-24 12:23:14 -05:00
Joseph Schorr
484977f728 Refactor security scanner validation from single sleep to polling 2017-02-24 12:23:14 -05:00
Jimmy Zelinskie
c8034deab4 util.secscan.api: failover connection failures 2017-02-23 15:01:32 -05:00
Joseph Schorr
67c0bf6263 Fix docker versioning library to support new versioning scheme 
Fixes: https://sentry.io/coreos/backend-production/issues/222349174/
Reference: https://github.com/docker/docker/pull/31075
 2017-02-22 16:08:17 -05:00
Joseph Schorr
94be8731f3 Change Docker Version tests to pytest 2017-02-22 15:45:06 -05:00
josephschorr
f7a7d30ec2 Merge pull request #2366 from coreos-inc/alert-spam-fixes 
Small fixes for alert spam
 2017-02-22 14:18:18 -05:00
Joseph Schorr
7cc7e54945 Remove unicode before sending it to path parser 
Fixes https://sentry.io/coreos/backend-production/issues/175929456/
 2017-02-22 13:21:12 -05:00
Jake Moshenko
b03e03c389 Read the number of unscanned clair images from the block allocator 2017-02-21 19:13:51 -05:00
josephschorr
8f01cb959a Merge pull request #2354 from coreos-inc/license-sorting 
Change entitlement sorting to sort *valid* entitlements by reverse expiration time
 2017-02-15 16:24:51 -05:00
Joseph Schorr
d506279892 Change entitlement sorting to sort *valid* entitlements by reverse expiration time 
With this change, if all entitlements are valid, we sort to show the entitlement that will expire the farthest in the future, as that defines the point at which the user must act before the license becomes invalid.
 2017-02-15 14:31:24 -05:00
Charlton Austin
3fd8c8a60d feature(app.py): adding queue_metrics to queues 
publishing queue metrics for SRE

[none]
 2017-02-14 16:01:28 -05:00
Jimmy Zelinskie
1d6339e644 test.test_api_usage: fix secscan tests 2017-02-14 15:21:18 -05:00
Jimmy Zelinskie
3286566478 util.secscan.api: reorg try/catch 2017-02-14 15:21:17 -05:00
Jimmy Zelinskie
d2909c0e4d failover: store result in FailoverException 2017-02-14 14:36:36 -05:00
Jimmy Zelinskie
c2c6bc1e90 test: add qss read failover case 2017-02-03 19:20:13 -05:00
Jimmy Zelinskie
1d59095460 utils.secscan: linter fixes 2017-02-03 19:20:13 -05:00
Jimmy Zelinskie
e81926fcba util.secscan.api: init read-only failover 2017-02-03 19:20:13 -05:00
Jimmy Zelinskie
b4efa7e45b util.failover: init 2017-02-03 19:20:13 -05:00
Joseph Schorr
c9bb132339 Increase cloudwatch send timeout to reduce how often we hit the API 2017-02-01 13:09:00 -05:00
Joseph Schorr
b407f88a26 Remove unnecessary CloudWatch metrics 
They are spamming the API and costing us a lot of money
 2017-02-01 13:08:21 -05:00
josephschorr
01ec22b362 Merge pull request #2300 from coreos-inc/openid-connect 
OpenID Connect support and OAuth login refactoring
 2017-01-31 18:14:44 -05:00
Jimmy Zelinskie
7a957c94c8 image/appc: fix volume conversion and add tests 2017-01-31 15:37:16 -05:00
Joseph Schorr
f5dbc350f8 Fix missed tests and revert conftest change (breaks docker build) 2017-01-30 17:28:25 -05:00
Joseph Schorr
d63cca025a DNS name check got reversed; breaks wildcards 2017-01-29 11:51:37 -05:00
Joseph Schorr
d9003d1375 Make sure the parent dir of a file path exists before writing the file 
Fixes when the `extra_ca_certs` directory doesn't exist when using the new custom certs tool
 2017-01-26 15:15:40 -05:00
Joseph Schorr
7c1bb886db Security scanner ordered tuplize bug fix 
If only the old list is present, we still need to tuplize the entries.

Fixes https://sentry.io/coreos/backend-production/issues/207196561/
 2017-01-24 13:16:44 -05:00
Joseph Schorr
19f7acf575 Lay foundation for truly dynamic external logins 
Moves all the external login services into a set of classes that share as much code as possible. These services are then registered on both the client and server, allowing us in the followup change to dynamically register new handlers
 2017-01-20 15:21:08 -05:00
Joseph Schorr
4755d08677 Refactor and rename the standard OAuth services 2017-01-19 15:23:15 -05:00
Joseph Schorr
bee2551dc2 Temporarily remove Dex login support 
This will be added back in later in this PR as part of proper generic OIDC support
 2017-01-19 14:51:12 -05:00
Joseph Schorr
7c7a07fb5a Allow namespaces to be between 2 and 255 characters in length 
[Delivers #137924329]
 2017-01-19 13:10:26 -05:00
Joseph Schorr
462f47924e More detailed namespace validation 
Fixes namespace validation to use the proper regex for checking length, as well as showing the proper messaging if the entered namespace is invalid

[Delivers #137830461]
 2017-01-17 17:31:59 -05:00
josephschorr
aafcb592a6 Merge pull request #2257 from coreos-inc/clair-gc-take2 
feat(gc): Garbage collection for security scanning
 2017-01-17 14:49:36 -05:00
josephschorr
eb2cafacd4 Merge pull request #2249 from coreos-inc/notifier-fixes 
Security notification pagination fix
 2017-01-17 11:33:25 -05:00
josephschorr
ac8cddc5a9 Merge pull request #2274 from coreos-inc/custom-cert-management 
Custom SSL certificates config panel
 2017-01-13 16:24:47 -05:00
josephschorr
6539fa3b20 Merge pull request #2259 from coreos-inc/delete-abuse-tool 
Add tool for handling abusing users
 2017-01-13 16:22:15 -05:00
Joseph Schorr
1cbacbbb63 Add tool for handling abusing users 2017-01-13 14:42:03 -05:00
Joseph Schorr
7e0fbeb625 Custom SSL certificates config panel 
Adds a new panel to the superuser config tool, for managing custom SSL certificates in the config bundle

[Delivers #135586525]
 2017-01-13 14:34:35 -05:00
Joseph Schorr
3a24871422 Add SSL certificate utility and tests 2017-01-10 17:06:13 -05:00
Joseph Schorr
f1c9965edf Add more volume file operations and cleanup k8s provider code 2017-01-10 17:06:13 -05:00
Joseph Schorr
29d6abddb5 Linter fixes 2017-01-10 17:06:13 -05:00
EvB
a7122db250 fix(cloudwatch): randomize sleep interval 2017-01-05 11:41:12 -05:00
Jake Moshenko
6c84b9330b Merge pull request #2251 from jakedt/fixaci 
Fix port mapping for ACI conversion from newer Docker manifests.
 2016-12-27 14:13:03 -05:00
Joseph Schorr
d609e6a1c4 Security scanner garbage collection support 
Adds support for calling GC in the security scanner for any layers+storage removed by GC on the Quay side
 2016-12-22 14:55:26 -05:00
Joseph Schorr
9413e25123 Change georeplication queuing to use new batch system 2016-12-21 17:44:30 -05:00
Jake Moshenko
d58a1ca35a Fix port mapping for ACI conversion from newer Docker manifests. 2016-12-20 14:01:06 -05:00
Joseph Schorr
5b3212ea0e Change security notification code to use the new stream diff reporters 
This ensures that even if security scanner pagination sends Old and New layer IDs on different pages, they will properly be handled across the entire notification.

Fixes https://www.pivotaltracker.com/story/show/136133657
 2016-12-20 12:50:19 -05:00
Joseph Schorr
ced0149520 Implement helper classes for tracking streaming diffs, both indexed and non-indexed 
These classes will be used to handle the Layer ID paginated diffs from Clair.
 2016-12-20 12:50:18 -05:00
Joseph Schorr
405eca074c Security scanner flow changes and auto-retry 
Changes the security scanner code to raise exceptions now for non-successful operations. One of the new exceptions raised is MissingParentLayerException, which, when raised, will cause the security worker to perform a full rescan of all parent images for the current layer, before trying once more to scan the current layer. This should allow the system to be "self-healing" in the case where the security scanner engine somehow loses or corrupts a parent layer.
 2016-12-16 15:38:09 -05:00
josephschorr
9fa16679f8 Merge pull request #2238 from coreos-inc/fake-clair 
Add a fake security scanner class for easier testing
 2016-12-15 20:51:24 -05:00
Brad Ison
2730c26b2e Merge pull request #2237 from coreos-inc/metrics-labels 
Don't record size in chunk upload metrics
 2016-12-15 14:20:34 -05:00
Brad Ison
df7366eace Add chunk size metric 2016-12-15 13:20:16 -05:00
Joseph Schorr
15041ac5ed Add a fake security scanner class for easier testing 
The FakeSecurityScanner mocks out all calls that Quay is expected to make to the security scanner API, and returns faked data that can be adjusted by the calling test case
 2016-12-14 17:11:45 -05:00
Brad Ison
8f59ac1251 Don't record size in chunk upload metrics 2016-12-14 12:16:02 -05:00
Joseph Schorr
6871eb95b1 Send notifications for previously unscannable layers in QSS 
Following this change, if an image was previously indexed unsuccessfully, then we will send notifications once successfully indexed
 2016-12-14 11:25:45 -05:00
Joseph Schorr
624b2a8385 Have security scanner analyze only send notifications for *new* layers 
Following this change, anytime a layer is indexed by the security scanner, we only send notifications out if the layer previously had a security_indexed_engine value of `-1`, thus ensuring it has *never* been indexed previously. This will allow us to change to version of the security scanner upwards, and have all the images be re-indexed, without firing off notifications in a spammy manner.
 2016-12-13 23:17:11 -05:00
Evan Cordell
5686c80af1 Revert "Add GC of layers in Clair" 
This reverts 49872838ab
 2016-12-13 18:40:58 -05:00
Evan Cordell
dd5f7cbe6c Fix the ephemeral build metrics 2016-12-13 18:28:04 -05:00