vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
6978 commits 2 branches 62 tags 205 MiB
Commit graph

1237 commits

Author SHA1 Message Date
Joseph Schorr
62312e6461 Add warning when CAS paths are skipped and ensure we are under a transaction 2017-03-08 17:01:07 -05:00
Joseph Schorr
69e550d125 Fix GC handling around CAS paths 
Adds code to ensure we never GC CAS paths that are shared amongst multiple ImageStorage rows, as well as an associated pair of tests to catch the positive and negative cases.
 2017-03-07 13:48:07 -05:00
Jimmy Zelinskie
40636d4103 find work based on tag IDs rather than image IDs 2017-03-06 17:09:57 -05:00
Jimmy Zelinskie
2cead05f53 data.model.tag: filter hidden for scan eligibility 2017-03-06 15:44:01 -05:00
Jimmy Zelinskie
904b902295 workers.securityworker: find eligible tag images 2017-03-06 14:37:34 -05:00
Jimmy Zelinskie
b9ac2b7b3b workers.securityworker: simplify min id 2017-03-03 14:51:18 -05:00
Jimmy Zelinskie
4ed0cdda14 securityscanner: add a min image id option 
This will enable us to force some instances of the securityworker to
scan only new images.
 2017-03-03 13:55:25 -05:00
Joseph Schorr
8e863b8cf5 Implement new create and manager trigger UI 
Implements the new trigger setup user interface, which is now a linear workflow found on its own page, rather than a tiny modal dialog

Fixes #1187
 2017-02-28 16:51:42 -05:00
Joseph Schorr
8ec6221ca2 Fix health check 2017-02-24 12:23:18 -05:00
Joseph Schorr
c0f7530b29 Pull out JWT auth validation into validator class 
Also fixes a small bug in validation (yay tests!)
 2017-02-24 12:23:16 -05:00
josephschorr
f7a7d30ec2 Merge pull request #2366 from coreos-inc/alert-spam-fixes 
Small fixes for alert spam
 2017-02-22 14:18:18 -05:00
Joseph Schorr
478b1642b2 Eat AttributeError in peewee close database call 
Fixes https://sentry.io/coreos/backend-production/issues/104257892/
 2017-02-22 13:21:12 -05:00
Joseph Schorr
d29d2da1ca Handle IntegrityError in tag update code 
Fixes https://sentry.io/coreos/backend-production/issues/173470565/events/4938537230/
 2017-02-22 13:20:04 -05:00
Joseph Schorr
ef9cb3757d Check for missing repository on GC call 
Fixes https://sentry.io/coreos/backend-production/issues/192273882/
 2017-02-22 13:18:23 -05:00
Joseph Schorr
89b7c13da5 Catch team member invite missing exception 
Fixes https://sentry.io/coreos/backend-production/issues/195926082/
 2017-02-22 13:18:22 -05:00
Jake Moshenko
27f5f14f90 Linter fixes 2017-02-22 11:45:38 -05:00
Jake Moshenko
add6b654ae Move the total image count stat back to the prom stat worker 2017-02-22 11:45:38 -05:00
Jimmy Zelinskie
3d21af59fd data.model.image: fake QSS progress metric 2017-02-21 17:48:40 -05:00
Joseph Schorr
eece782038 Prevent peewee from loading the visibility every time 
By calling `visibility` instead of `visibility_id`, peewee was issuing a SQL Select statement for the repository, which removes the benefit of the optimization
 2017-02-17 12:09:48 -05:00
Joseph Schorr
421c5d6012 Fix bug where the login service ID doesn't exist 2017-02-16 16:27:53 -05:00
josephschorr
2a7d1fbe57 Merge pull request #2358 from coreos-inc/better-logging 
Log more information to the action logs and display the namespaces for superusers
 2017-02-14 16:38:35 -05:00
Charlton Austin
3fd8c8a60d feature(app.py): adding queue_metrics to queues 
publishing queue metrics for SRE

[none]
 2017-02-14 16:01:28 -05:00
Joseph Schorr
11c931f781 Log more information to the action logs and display the namespaces for superusers 
This helps superusers understand better what, exactly, is going on in the registry
 2017-02-14 14:55:24 -05:00
Charlton Austin
85bcb63439 update(security_test.py): moving tests to new framework 
We should be moving tests over to pytest

[none]
 2017-02-02 13:40:00 -05:00
Joseph Schorr
b407f88a26 Remove unnecessary CloudWatch metrics 
They are spamming the API and costing us a lot of money
 2017-02-01 13:08:21 -05:00
josephschorr
01ec22b362 Merge pull request #2300 from coreos-inc/openid-connect 
OpenID Connect support and OAuth login refactoring
 2017-01-31 18:14:44 -05:00
Joseph Schorr
3324743bff Fix db migration revision 2017-01-31 11:38:31 -05:00
Joseph Schorr
973a110ac7 Full text search for repository name and description 
Adds support for searching full text against the name and description of a repository

[Delivers #134867401]
 2017-01-31 11:38:31 -05:00
Joseph Schorr
d65d32b284 Convert model to use moved prefix_search method 2017-01-31 11:38:31 -05:00
Joseph Schorr
d89c79b92d Full text support in peewee 
Adds support for full text search in peewee with the creation of two new field types: `FullIndexedCharField` and `FullIndexedTextField`.

Note that this change depends upon https://github.com/zzzeek/sqlalchemy/pull/339

[Delivers #137453279]
[Delivers #137453317]
 2017-01-31 11:38:31 -05:00
Joseph Schorr
fda203e4d7 Add proper and tested OIDC support on the server 
Note that this will still not work on the client side; the followup CL for the client side is right after this one.
 2017-01-23 17:53:34 -05:00
Evan Cordell
28813159e5 fix(userevent): ignore subscribe notifications in userevents 
[Fixes #138007389]
 2017-01-20 13:38:02 -05:00
Joseph Schorr
71ec23b550 Switch QueueItem state_id to be unique after a backfill 2017-01-18 17:43:41 -05:00
josephschorr
e2748fccd9 Merge pull request #2282 from coreos-inc/motd-updates 
Severity and Markdown support in MOTD
 2017-01-18 17:41:27 -05:00
Joseph Schorr
3106504f39 Severity and Markdown support in MOTD 
[Delivers #133555165]
 2017-01-18 16:55:32 -05:00
Joseph Schorr
af23d2bedd Remove unique from queue item state_id 2017-01-18 15:04:26 -05:00
Joseph Schorr
3cf8f6c28a Cleanup user event reporting and lower its timeout 2017-01-18 11:27:00 -05:00
Joseph Schorr
462f47924e More detailed namespace validation 
Fixes namespace validation to use the proper regex for checking length, as well as showing the proper messaging if the entered namespace is invalid

[Delivers #137830461]
 2017-01-17 17:31:59 -05:00
josephschorr
aafcb592a6 Merge pull request #2257 from coreos-inc/clair-gc-take2 
feat(gc): Garbage collection for security scanning
 2017-01-17 14:49:36 -05:00
Joseph Schorr
8c4e86f48b Change queue to use state-field for claiming items 
Before this change, the queue code would check that none of the fields on the item to be claimed had changed between the time when the item was selected and the item is claimed. While this is a safe approach, it also causes quite a bit of lock contention in MySQL, because InnoDB will take a lock on *any* rows examined by the `where` clause of the `update`, even if they will ultimately thrown out due to other clauses (See: http://dev.mysql.com/doc/refman/5.7/en/innodb-locks-set.html: "A ..., an UPDATE, ... generally set record locks on every index record that is scanned in the processing of the SQL statement. It does not matter whether there are WHERE conditions in the statement that would exclude the row. InnoDB does not remember the exact WHERE condition, but only knows which index ranges were scanned").

As a result, we want to minimize the number of fields accessed in the `where` clause on an update to the QueueItem row. To do so, we introduce a new `state_id` column, which is updated on *every change* to the QueueItem rows with a unique, random value. We can then have the queue item claiming code simply check that the `state_id` column has not changed between the retrieval and claiming steps. This minimizes the number of columns being checked to two (`id` and `state_id`), and thus, should significantly reduce lock contention. Note that we can not (yet) reduce to just a single `state_id` column (which should work in theory), because we need to maintain backwards compatibility with existing items in the QueueItem table, which will be given empty `state_id` values when the migration in this change runs.

Also adds a number of tests for other queue operations that we want to make sure operate correctly following this change.

[Delivers #133632501]
 2017-01-17 13:29:26 -05:00
Joseph Schorr
19cb64df5d Remove unused class 2017-01-17 13:26:09 -05:00
Joseph Schorr
7f63cbd14f Remove FOR UPDATE in Queue cancel and complete 
We have no need for them anymore and it should reduce lock contention a bit

Fixes #776
 2017-01-17 13:26:09 -05:00
Charlton Austin
ca832df975 Adding in new indices for queueitem table. 2017-01-17 10:04:31 -05:00
Joseph Schorr
1cbacbbb63 Add tool for handling abusing users 2017-01-13 14:42:03 -05:00
Joseph Schorr
5225642850 Garbage collection image+storage callback support 
Add support to GC to invoke a callback with the image+storages removed. Only images whose storage was also removed will be sent to the callback. This will be used by security scanning for its own GC in the followup change.
 2016-12-22 14:27:42 -05:00
Joseph Schorr
e2efb6c458 Add default and configurable LDAP timeouts 
Fixes https://www.pivotaltracker.com/story/show/135885019
 2016-12-19 11:53:06 -05:00
Joseph Schorr
58b7481a63 Make sure robot accounts always show up first in entity search 
Fixes https://www.pivotaltracker.com/story/show/136277321
Fixes #2241
 2016-12-16 15:04:30 -05:00
Joseph Schorr
785c74de52 Fix attempts to confirm team invite for mismatched email address 
Currently, if a user tries to confirm an invite sent to them on an account with a mismatching email address, we simply redirect to the org (where they get a 403). This change ensures they get the proper error response message, and restyles the error page to be nicer.

Fixes #2227
Fixes https://www.pivotaltracker.com/story/show/136088507
 2016-12-15 17:15:11 -05:00
Joseph Schorr
624b2a8385 Have security scanner analyze only send notifications for *new* layers 
Following this change, anytime a layer is indexed by the security scanner, we only send notifications out if the layer previously had a security_indexed_engine value of `-1`, thus ensuring it has *never* been indexed previously. This will allow us to change to version of the security scanner upwards, and have all the images be re-indexed, without firing off notifications in a spammy manner.
 2016-12-13 23:17:11 -05:00
Evan Cordell
5686c80af1 Revert "Add GC of layers in Clair" 
This reverts 49872838ab
 2016-12-13 18:40:58 -05:00
Charlton Austin
0b8c2ef92f Removing an unused import. 2016-12-08 13:53:52 -05:00
josephschorr
410b9d74fc Merge pull request #2214 from coreos-inc/clair-gc 
Add GC of layers in Clair
 2016-12-07 17:58:21 -05:00
josephschorr
543d86ae10 Merge pull request #2221 from coreos-inc/fix-error-pages 
Have all error pages be rendered by Angular
 2016-12-07 17:53:14 -05:00
josephschorr
111b7b0788 Merge pull request #2206 from coreos-inc/ldap-user-search-fix 
Fix external auth returns for query_user calls
 2016-12-07 17:53:04 -05:00
josephschorr
c5111d7930 Merge pull request #2144 from coreos-inc/buildlogs-improvements 
Change the append build log method to execute the two calls via one pipelined connection
 2016-12-07 17:52:22 -05:00
Joseph Schorr
c06bba38de Have all error pages be rendered by Angular 
Fixes #2198

Fixes https://www.pivotaltracker.com/story/show/135724483
 2016-12-07 17:49:02 -05:00
Jimmy Zelinskie
00eafff747 Merge pull request #2204 from jzelinskie/429builds 
add rate limiting to build queues
 2016-12-07 15:03:31 -05:00
Joseph Schorr
3203fd6de1 Fix external auth returns for query_user calls 
Adds the missing field on the query_user calls, updates the external auth tests to ensure it is returned properly, and adds new end-to-end tests which call the external auth engines via the *API*, to ensure this doesn't break again
 2016-12-07 14:28:42 -05:00
Jimmy Zelinskie
ebbe58d311 replace prefix w/ canonical name list 2016-12-07 12:56:56 -05:00
Jimmy Zelinskie
c41de8ded6 build queue rate limiting: address PR comments 2016-12-06 20:40:54 -05:00
Joseph Schorr
49872838ab Add GC of layers in Clair 
Fixes https://www.pivotaltracker.com/story/show/135583207
 2016-12-06 19:52:56 -05:00
Jimmy Zelinskie
eb69abff8b build rate limiting: tests 2016-12-06 16:30:12 -05:00
Jimmy Zelinskie
57770493fa build rate limiting: use a rate 2016-12-06 16:30:12 -05:00
Jimmy Zelinskie
7877c6ab94 add rate limiting to build queues 2016-12-06 16:30:12 -05:00
Charlton Austin
0aa6e6cd58 Merge pull request #2203 from charltonaustin/fix_build_component_cleanup 
Adding in a cancel method to the build component so we can properly c…
 2016-12-06 14:13:10 -05:00
Jake Moshenko
21e3001446 Add a bulk insert for queue and notifications. 
Use it for Clair spawned notifications.
 2016-12-06 14:00:16 -05:00
Charlton Austin
c6be12e31e Adding in a cancel method to the build component so we can properly clean up the job task. 2016-12-06 13:37:49 -05:00
Jimmy Zelinskie
3a7119d499 Merge pull request #2209 from coreos-inc/clair-notification-read 
Clair notification read and queue fixes
 2016-12-05 19:36:59 -05:00
Joseph Schorr
97d150e281 Have QSS only add security scanner notifications once 2016-12-05 19:08:20 -05:00
Jake Moshenko
7c490b46c8 Only save dirty fields on Queue queries. 2016-12-05 18:12:14 -05:00
Charlton Austin
0a6322015c Fix the queue item delete. 2016-12-02 15:30:35 -05:00
Charlton Austin
7b3d8e3977 Merge pull request #2183 from charltonaustin/metrics_for_unscanned_images 
Adding in some metrics around clair sec scan.
 2016-12-02 11:50:29 -05:00
Charlton Austin
edd9dcd7f6 Adding in some metrics around clair sec scan. 2016-12-01 16:50:02 -05:00
Charlton Austin
1f03fcb146 Adding in notification type for notification kind. 2016-12-01 12:26:18 -05:00
Charlton Austin
2c637fe5ce Merge pull request #2173 from charltonaustin/adding_in_build_cancel_notifications 
Adding in cancel notifications
 2016-11-30 15:03:17 -05:00
Charlton Austin
4103a0b75f Adding in cancel notifications 2016-11-30 14:38:34 -05:00
Joseph Schorr
730a220eb0 Fix user lookup query under Postgres 
Adds a missing group_by clause
 2016-11-29 11:36:53 -05:00
Joseph Schorr
402ad25690 Change team invitation acceptance to join all invited teams under the org 
Fixes #1989
 2016-11-28 18:39:28 -05:00
Joseph Schorr
e29cb34336 Fix Set calls to gauges 
Fixes #2150

The proper function is `Set` (not `set`), which was causing these gauges to not report to Prometheus
 2016-11-21 15:27:17 -05:00
Charlton Austin
2fe74e4057 Adding in UI for cancel anytime. 2016-11-21 10:58:32 -05:00
Joseph Schorr
1b8820f2e7 Change the append build log method to execute the two calls via one pipelined connection 
Should reduce the amount of packets used by the build manager

Reference: https://github.com/andymccurdy/redis-py#pipelines
 2016-11-18 11:47:16 -05:00
Charlton Austin
fd7c566d31 Adding in cancel for a build that is building. 2016-11-16 17:40:24 -05:00
Joseph Schorr
1a61ef4e04 Report the user's name and company to Marketo 
Also fixes the API to report the other changes (username and email) as well
 2016-11-14 17:34:50 -05:00
Jake Moshenko
10255d4052 Merge pull request #2109 from jakedt/collapsemigrations 
Collapse all migrations prior to 2.0.0 into one.
 2016-11-10 17:35:07 -05:00
Jake Moshenko
b5834a8a66 Collapse all migrations prior to 2.0.0 into one. 2016-11-10 17:31:00 -05:00
Joseph Schorr
536809a992 Change LDAP errors into debug statements to reduce log clutter 
Fixes #2083
 2016-11-10 16:39:26 -05:00
Joseph Schorr
0f2eb61f4a Add collection of user metadata: name and company 2016-11-08 16:15:02 -05:00
josephschorr
233b2be5c2 Merge pull request #2066 from coreos-inc/select-username 
Add support for temp usernames and an interstitial to confirm username
 2016-11-03 16:22:16 -04:00
Joseph Schorr
1e3b354201 Add support for temp usernames and an interstitial to confirm username 
When a user now logs in for the first time for any external auth (LDAP, JWT, Keystone, Github, Google, Dex), they will be presented with a confirmation screen that affords them the opportunity to change their Quay-assigned username.

Addresses most of the user issues around #74
 2016-11-03 15:59:14 -04:00
Joseph Schorr
3fd92aef35 Fix entity search API to not IndexError 2016-11-02 16:22:35 -04:00
Joseph Schorr
d7f56350a4 Make email addresses optional in external auth if email feature is turned off 
Before this change, external auth such as Keystone would fail if a user without an email address tried to login, even if the email feature was disabled.
 2016-10-31 13:50:24 -04:00
josephschorr
934cdecbd6 Merge pull request #1905 from coreos-inc/external-auth-search 
Add support for entity search against external auth users not yet linked
 2016-10-27 16:06:42 -04:00
Joseph Schorr
b3d1d7227c Add support to Keystone Auth for external user linking 
Also adds Keystone V3 support
 2016-10-27 15:42:03 -04:00
Joseph Schorr
fbb524e34e Add support to ExternalJWT Auth for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
f9ee8d2bef Add support to LDAP for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
d145222812 Add support for linking to external users in entity search 2016-10-27 15:42:03 -04:00
Charlton Austin
2147005d2c Adding a method of cancelling a build based on etcd message. 2016-10-25 12:50:58 -04:00
Charlton Austin
dc35769396 Merge pull request #2022 from charltonaustin/refactor_for_cancel_anytime 
Making some refactors to make it easier to cancel the build at any time.
 2016-10-24 16:17:55 -04:00
Charlton Austin
1cde22e76c Making some refactors to make it easier to cancel the build at any time. 2016-10-24 15:59:33 -04:00
josephschorr
edc2bc8b93 Merge pull request #1698 from coreos-inc/delete-namespace 
Add support for deleting namespaces (users, organizations)
 2016-10-21 16:54:52 -04:00