vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9228 commits 2 branches 62 tags 205 MiB
Commit graph

1698 commits

Author SHA1 Message Date
Joseph Schorr
c5bb9abf11 Fix deleting repos when sec scan or signing is disabled 
Make sure we don't invoke the APIs to non-existent endpoints
 2017-04-19 16:57:36 -04:00
Charlton Austin
38d4af0d8b Merge pull request #2479 from charltonaustin/phase_three_config 
feat(data): remove writing of old config
 2017-04-18 10:25:50 -04:00
Antoine Legrand
599ce0de54 code-stye Yapf: 5 files updated 
data/interfaces/appr.py endpoints/appr/cnr_backend.py endpoints/appr/registry.py endpoints/appr/test/test_api.py endpoints/appr/test/test_registry.py
 2017-04-18 14:02:48 +02:00
Antoine Legrand
578f87f94c Fix login with robot to quay-appr 2017-04-18 13:59:21 +02:00
Evan Cordell
2661db7485 Add flag to enable trust per repo (#2541) 
* Add flag to enable trust per repo

* Add api for enabling/disabling trust

* Add new LogEntryKind for changing repo trust settings
Also add tests for repo trust api

* Add `set_trust` method to repository

* Expose new logkind to UI

* Fix registry tests

* Rebase migrations and regen test.db

* Raise downstreamissue if trust metadata can't be removed

* Refactor change_repo_trust

* Add show_if to change_repo_trust endpoint
 2017-04-15 08:26:33 -04:00
Joseph Schorr
626f306283 Add tests for list_active_tags 2017-04-14 17:53:31 -04:00
Joseph Schorr
4a1fad7520 Make sure to filter hidden tags from the active tags query 2017-04-13 19:29:38 -04:00
Joseph Schorr
ab2f044331 Switch get repo API to use a single list tags query 
Should make things faster since the join occurs on the database side
 2017-04-13 18:06:58 -04:00
Charlton Austin
40906afdd8 feat(data): remove writing of old config 
### Description of Changes
  Phase three of config data model change.
 2017-04-13 13:25:36 -04:00
Joseph Schorr
68331859b0 Add backfill for repository search score table 2017-04-13 12:30:44 -04:00
josephschorr
c78ec89305 Merge pull request #2531 from coreos-inc/repo-score 
Switch to using RepositorySearchScore table for search ranking
 2017-04-13 11:51:59 -04:00
Evan Cordell
ec63e495fc Add repo purge callbacks and register TUF metadata deletion as one 2017-04-12 17:33:51 -04:00
Joseph Schorr
80693d6b8c Fix NPE bug in RAC worker 
We need to return `None`, not `0` if there are no additional repositories to measure
 2017-04-11 15:42:11 -04:00
Joseph Schorr
e5a009a777 Switch to using RepositorySearchScore table for search ranking 
Should make search queries much, much faster as it contains the denormalized RAC data
 2017-04-11 14:55:20 -04:00
josephschorr
928b9915ed Merge pull request #2441 from coreos-inc/repo-score-denormalization 
Add a RepositorySearchScore table and calculation to the RAC worker
 2017-04-10 16:31:09 -04:00
Joseph Schorr
df3f47c79a Add a RepositorySearchScore table and calculation to the RAC worker 
This will be used in a followup PR to order search results instead of the RAC join. Currently, the join with the RAC table in search results in a lookup of ~600K rows, which causes searching to take ~6s. This PR denormalizes the data we need, as well as allowing us to score based on a wider band (6 months vs the current 1 week).
 2017-04-10 14:29:02 -04:00
Erica
3f79422a52 Merge pull request #2306 from coreos-inc/QUAY-2842-audit-log-strict-config-option 
feat(config.py): add setting for audit log strictness
 2017-04-07 13:43:11 -04:00
EvB
20c4d971c4 refactor(model/log): pull allowed action types into constant 2017-04-07 11:39:54 -04:00
Jake Moshenko
a8ec7865a7 Merge pull request #2511 from jakedt/fixwarnings 
Fixwarnings
 2017-04-06 16:12:19 -04:00
Jake Moshenko
c7241911a5 Fix old-style flask imports to silence deprecation warnings. 2017-04-06 13:15:48 -04:00
Joseph Schorr
f9e6110f73 Add basic user interface for application repos 
Adds support for creating app repos, viewing app repos and seeing the list of app repos in the Quay UI.
 2017-04-05 11:30:09 -04:00
EvB
d0aaaaa1ef test(data/model/log): add more log_action tests 
temp
 2017-04-05 11:26:10 -04:00
EvB
ea740d15ba fix(model/log): log exception on swallowed db errors 2017-04-05 11:26:10 -04:00
EvB
625bd66b42 test(data/model): test action logging 2017-04-05 11:26:10 -04:00
EvB
6916d82e0d feat(endpoints/trackhelper): wrap log op for silent fails 2017-04-05 11:26:10 -04:00
Joseph Schorr
b10608e277 Change teamsync config to be a UTF8 field 2017-04-03 14:16:34 -04:00
Joseph Schorr
bdd07d4f39 Fix flakiness in team sync tests 2017-04-03 11:36:42 -04:00
Joseph Schorr
bd22fb255e Rename get_federated_user to get_and_link_federated_user_info 
Better to be explicit wherever possible
 2017-04-03 11:36:42 -04:00
Joseph Schorr
1a31d98c44 Clarify variable name in Keystone auth 2017-04-03 11:36:41 -04:00
Joseph Schorr
8c07f733eb Add pagination tests for LDAP 2017-04-03 11:36:41 -04:00
Joseph Schorr
541aa722c2 Add sleeps to make test non-flaky 
Sucks, but MySQL only has second-level timing, so we need this to be sure
 2017-04-03 11:36:41 -04:00
Joseph Schorr
103186f5e8 Small renames to make team syncing code more clear 2017-04-03 11:36:41 -04:00
Joseph Schorr
7f0aa19292 Code cleanup and style improvements in team sync 2017-04-03 11:36:41 -04:00
Joseph Schorr
84e37b68ee Change if statement to be more readable 2017-04-03 11:31:30 -04:00
Joseph Schorr
71d52d45ba Add a test for same user returned twice in team sync 2017-04-03 11:31:30 -04:00
Joseph Schorr
d7825c6720 Add group iteration and syncing support to Keystone auth 2017-04-03 11:31:30 -04:00
Joseph Schorr
47278cc559 Cleanup test fixtures 2017-04-03 11:31:30 -04:00
Joseph Schorr
df603462b8 Add database migration for TeamSync 2017-04-03 11:31:29 -04:00
Joseph Schorr
96b9d6b0cd Add end-to-end test for team sync 2017-04-03 11:31:29 -04:00
Joseph Schorr
4055158fc4 Fix indentation 2017-04-03 11:31:29 -04:00
Joseph Schorr
938730c076 Move sync team into its own module and add tests 2017-04-03 11:31:29 -04:00
Joseph Schorr
eeadeb9383 Initial interfaces and support for team syncing worker 2017-04-03 11:31:29 -04:00
Joseph Schorr
94b07e6de9 Allow nulls in last_updated field to accurately report the last updated time to users for newly sync teams 2017-04-03 11:31:29 -04:00
Joseph Schorr
8ea3977140 Add ability to enable, disable and view team syncing in UI and API 
Also extracts out some common testing infrastructure to make testing APIs easier now using pytest
 2017-04-03 11:31:29 -04:00
Joseph Schorr
bb20422260 Fix pagination disabling in LDAP with mockldap 
Since mockldap doesn't support pagination, just disable it globally
 2017-04-03 11:31:28 -04:00
Joseph Schorr
ecfac81721 Add check_group_lookup_args and service_metadata to auth providers 2017-04-03 11:31:28 -04:00
Joseph Schorr
1cfc4a8341 Change max size of LDAP pages and add filtering to reduce attributes returned 2017-04-03 11:31:28 -04:00
Joseph Schorr
f5a854c189 Add TeamSync database and API support 
Teams can now have a TeamSync entry in the database, indicating how they are synced via an external group. If found, then the user membership of the team cannot be changed via the API.
 2017-04-03 11:31:28 -04:00
Joseph Schorr
d718829f5d Initial LDAP group member iteration support 
Add interface for group member iteration on internal auth providers and implement support in the LDAP interface.
 2017-04-03 11:31:28 -04:00
Charlton Austin
9ff189b16e fix(migration merge issue): missing .save() on migration 2017-03-28 15:17:51 -04:00
Charlton Austin
d559dc7b3e Fixing the migration path so we don't have incorrect branches. 2017-03-28 14:54:21 -04:00
Charlton Austin
ca99535774 Merge pull request #2449 from charltonaustin/phase_two_config 
feat(build runner): added in context, dockerfile_location
 2017-03-28 14:14:36 -04:00
Charlton Austin
e6d201e0b0 feat(build runner): added in context, dockerfile_location 
this is a new feature meant to allow people to use any file as
  a dockerfile and any folder as a context directory
 2017-03-28 13:55:31 -04:00
Antoine Legrand
d2ed37e158 Fix force push causing duplicated entries 2017-03-27 15:39:57 +02:00
Antoine Legrand
22c1a29892 fix strip_sha256 2017-03-24 19:49:52 +01:00
Antoine Legrand
bbd74eabd1 Allow force push for app 2017-03-23 22:50:07 +01:00
Joseph Schorr
ac4a79ae01 Update PR for rebase 2017-03-23 15:57:49 -04:00
Joseph Schorr
651666b60b Refactor our auth handling code to be cleaner 
Breaks out the validation code from the auth context modification calls, makes decorators easier to define and adds testing for each individual piece. Will be the basis of better error messaging in the following change.
 2017-03-23 15:42:45 -04:00
Joseph Schorr
dd9e4bf3e7 Remove transaction around OCI blobs 
Fixes https://www.pivotaltracker.com/story/show/142341399
 2017-03-23 14:51:37 -04:00
Jimmy Zelinskie
9c0cbbf57c data.oci_model: sloppily rewrite digest format 
We expect digests to be in the form 'sha256:digest'
 2017-03-23 12:37:32 -04:00
Joseph Schorr
7d66f30d52 Fix filtering of repositories in search 2017-03-23 11:35:17 -04:00
Joseph Schorr
917d5e2550 Fix typos in data model 2017-03-23 11:14:08 -04:00
Joseph Schorr
05ce571e3e Add missing return statement 2017-03-23 11:11:21 -04:00
Jimmy Zelinskie
d20ff785e6 data.model.repository: add back search fields 2017-03-23 10:46:04 -04:00
Jimmy Zelinskie
2bdd3d4fa1 data.oci_model.tag: add missing import 2017-03-23 00:58:57 -04:00
Jimmy Zelinskie
e872c310d0 data.oci_model: fix imports 2017-03-23 00:21:21 -04:00
Joseph Schorr
bdda74d6df Make sure GC checks new Blob table as well before deleting CAS storage 2017-03-22 23:53:21 -04:00
Jimmy Zelinskie
ddad957a56 data.model.repository: add app methods 2017-03-22 21:51:55 -04:00
Jimmy Zelinskie
650723430b data.interfaces.appr: init 2017-03-22 21:51:41 -04:00
Jimmy Zelinskie
9f684fa73f data.oci_model: init with app methods 2017-03-22 21:51:28 -04:00
Jimmy Zelinskie
3ccf3c5f33 Merge pull request #2447 from jzelinskie/cnr-step2 
CNR Step 2
 2017-03-22 18:45:51 -04:00
Joseph Schorr
df1e7f90e0 Add verb security tests and fix small issues 2017-03-22 18:29:53 -04:00
Jimmy Zelinskie
d5fa2ad0c0 endpoints.verbs: abort 405 for non-container repos 2017-03-22 17:50:58 -04:00
Jimmy Zelinskie
40b638a981 data.migrations: rebase to HEAD of migration tree 2017-03-22 17:26:59 -04:00
Jimmy Zelinskie
48ba59d615 endpoints.v2: only work on docker repositories 2017-03-22 17:26:59 -04:00
Jimmy Zelinskie
72751592a3 data.interfaces.v1: document types 2017-03-22 15:51:19 -04:00
Jimmy Zelinskie
45f14f220d data.model.repository: optimize by using kind_id 2017-03-22 15:51:19 -04:00
Joseph Schorr
30b532254c Disallow non-apps-supported APIs for application repositories 2017-03-22 15:51:19 -04:00
Joseph Schorr
c3402fff5a Add test to ensure we cannot create repos with the same name but different kinds 2017-03-22 14:34:32 -04:00
Jimmy Zelinskie
a2bac7dabd endpoints.v1: only work on docker repositories 2017-03-22 14:31:22 -04:00
Jimmy Zelinskie
f086eea754 data.interfaces.v1: explicitly use kwargs 
This fixes the function from accidentally using the wrong arguments
positionally for the new `repo_kind` kwarg.
 2017-03-22 13:58:50 -04:00
Jimmy Zelinskie
d2a4c9d05a data.model.repository: audited for repo_kind usage 2017-03-22 13:58:50 -04:00
Jimmy Zelinskie
074c1bc4a8 data.model._basequery: audited for repo_kind usage 2017-03-22 13:58:49 -04:00
Jimmy Zelinskie
f842bc3a82 data.migrations.migration.sh: wait 25s for mysql 
Without this, there are frequent race conditions wheres the client fails
to connect to the server when using Docker For Mac.
 2017-03-21 15:38:39 -04:00
Jimmy Zelinskie
4492d2f210 data.migrations: add repository kind 2017-03-21 15:38:38 -04:00
Jimmy Zelinskie
5b362da8ac data.database: add RepositoryKind 2017-03-21 15:38:38 -04:00
Joseph Schorr
ff7f78e990 Have blob uploads be checked against configurable max layer size 2017-03-21 13:16:55 -04:00
Joseph Schorr
76de324ca8 Change blob upload ints into bigints 2017-03-21 13:14:11 -04:00
Charlton Austin
3502d9f61c Merge pull request #2438 from charltonaustin/phase_one_config 
refactor(data): add in new config for builder
 2017-03-21 10:16:51 -04:00
Jimmy Zelinskie
6a538647e4 data.database: beta classes skip transitive delete 2017-03-20 18:41:39 -04:00
josephschorr
27aa12de7a Merge pull request #2439 from coreos-inc/remove_redis_log_expiration 
Switch from expire to delete redis log_entries
 2017-03-20 13:41:57 -04:00
Charlton Austin
f701677a8e refactor(data): add in new config for builder 
we are doing phase one of the four phase migration on the builder config
 2017-03-20 13:03:41 -04:00
Jimmy Zelinskie
0ea600628b Merge pull request #2436 from jzelinskie/cnr-step1 
CNR - Step 1
 2017-03-17 15:37:29 -04:00
Jimmy Zelinskie
ad029fb331 data.migrations: don't use UTF-8 for unique fields 
Unique indexes must have less than 767 bytes and UTF-8 encoding with 255
chars is beyond this maximum. Since this is an internal identifier, we
can be confident that we will not require UTF-8 for it in the future.
 2017-03-17 15:21:24 -04:00
Jimmy Zelinskie
c915a40531 data.database: rm tag_kind from Tag indexes 
These shouldn't be necessary.
 2017-03-17 11:35:16 -04:00
Jimmy Zelinskie
0e32e77e99 data.database: document all CNR/OCI models 2017-03-17 11:35:16 -04:00
Jimmy Zelinskie
2a117f2d24 data.migrations: change CNR mimetypes to v0 
Our initial CNR support is of a pre-v1 implementation of the
specification.
 2017-03-17 11:33:16 -04:00
Jimmy Zelinskie
1e9ce85af6 data.database/migrations: remove repo_id from db 
This also manually organizes and removes broken parts of the migration.
 2017-03-17 11:33:16 -04:00
Antoine Legrand
8f323154ce data.migrations: add OCI/CNR models 2017-03-17 11:33:16 -04:00
Antoine Legrand
c61024586d data.database: add CNR/OCI models 2017-03-17 11:33:16 -04:00
Antoine Legrand
718aeeead8 Fix search group_by clause for PG 2017-03-17 16:30:24 +01:00
Antoine Legrand
ec847ce613 Switch from expire to delete redis log_entries 2017-03-17 15:35:47 +01:00
Joseph Schorr
e25c989fef Add a cleanup worker for blob uploads 2017-03-16 13:36:59 -04:00
Joseph Schorr
e90cab4d77 Change revert tag into restore tag and add manifest support 2017-03-14 11:34:42 -04:00
Joseph Schorr
af743b156b Show manifest digests in place of V1 ids in the tag view when possible 2017-03-14 11:34:41 -04:00
Jimmy Zelinskie
123d003d4e Merge pull request #2424 from jzelinskie/qss-image 
workers.securityworker: revert to image querying
 2017-03-10 17:38:02 -05:00
Jimmy Zelinskie
a780136337 workers.securityworker: revert to image querying 2017-03-10 17:37:40 -05:00
josephschorr
cbac673d58 Merge pull request #2404 from coreos-inc/cas-gc-fix 
Fix GC handling around CAS paths
 2017-03-10 17:34:21 -05:00
josephschorr
432b2d3fe8 Merge pull request #2392 from coreos-inc/search-optimization 
Optimize repository search by changing our lookup strategy
 2017-03-10 15:44:26 -05:00
Jimmy Zelinskie
53eb579459 data.model.tag: find min *alive* tag 2017-03-10 13:15:35 -05:00
Joseph Schorr
b5bb76cdea Optimize repository search by changing our lookup strategy 
Previous to this change, repositories were looked up unfiltered in six different queries, and then filtered using the permissions model, which issued a query per repository found, making search incredibly slow. Instead, we now lookup a chunk of repositories unfiltered and then filter them via a single query to the database. By layering the filtering on top of the lookup, each as queries, we can minimize the number of queries necessary, without (at the same time) using a super expensive join.

Other changes:
- Remove the 5 page pre-lookup on V1 search and simply return that there is one more page available, until there isn't. While technically not correct, it is much more efficient, and no one should be using pagination with V1 search anyway.
- Remove the lookup for repos without entries in the RAC table. Instead, we now add a new RAC entry when the repository is created for *the day before*, with count 0, so that it is immediately searchable
- Remove lookup of results with a matching namespace; these aren't very relevant anyway, and it overly complicates sorting
 2017-03-09 19:47:55 -05:00
Joseph Schorr
62312e6461 Add warning when CAS paths are skipped and ensure we are under a transaction 2017-03-08 17:01:07 -05:00
Joseph Schorr
69e550d125 Fix GC handling around CAS paths 
Adds code to ensure we never GC CAS paths that are shared amongst multiple ImageStorage rows, as well as an associated pair of tests to catch the positive and negative cases.
 2017-03-07 13:48:07 -05:00
Jimmy Zelinskie
40636d4103 find work based on tag IDs rather than image IDs 2017-03-06 17:09:57 -05:00
Jimmy Zelinskie
2cead05f53 data.model.tag: filter hidden for scan eligibility 2017-03-06 15:44:01 -05:00
Jimmy Zelinskie
904b902295 workers.securityworker: find eligible tag images 2017-03-06 14:37:34 -05:00
Jimmy Zelinskie
b9ac2b7b3b workers.securityworker: simplify min id 2017-03-03 14:51:18 -05:00
Jimmy Zelinskie
4ed0cdda14 securityscanner: add a min image id option 
This will enable us to force some instances of the securityworker to
scan only new images.
 2017-03-03 13:55:25 -05:00
Joseph Schorr
8e863b8cf5 Implement new create and manager trigger UI 
Implements the new trigger setup user interface, which is now a linear workflow found on its own page, rather than a tiny modal dialog

Fixes #1187
 2017-02-28 16:51:42 -05:00
Joseph Schorr
8ec6221ca2 Fix health check 2017-02-24 12:23:18 -05:00
Joseph Schorr
c0f7530b29 Pull out JWT auth validation into validator class 
Also fixes a small bug in validation (yay tests!)
 2017-02-24 12:23:16 -05:00
josephschorr
f7a7d30ec2 Merge pull request #2366 from coreos-inc/alert-spam-fixes 
Small fixes for alert spam
 2017-02-22 14:18:18 -05:00
Joseph Schorr
478b1642b2 Eat AttributeError in peewee close database call 
Fixes https://sentry.io/coreos/backend-production/issues/104257892/
 2017-02-22 13:21:12 -05:00
Joseph Schorr
d29d2da1ca Handle IntegrityError in tag update code 
Fixes https://sentry.io/coreos/backend-production/issues/173470565/events/4938537230/
 2017-02-22 13:20:04 -05:00
Joseph Schorr
ef9cb3757d Check for missing repository on GC call 
Fixes https://sentry.io/coreos/backend-production/issues/192273882/
 2017-02-22 13:18:23 -05:00
Joseph Schorr
89b7c13da5 Catch team member invite missing exception 
Fixes https://sentry.io/coreos/backend-production/issues/195926082/
 2017-02-22 13:18:22 -05:00
Jake Moshenko
27f5f14f90 Linter fixes 2017-02-22 11:45:38 -05:00
Jake Moshenko
add6b654ae Move the total image count stat back to the prom stat worker 2017-02-22 11:45:38 -05:00
Jimmy Zelinskie
3d21af59fd data.model.image: fake QSS progress metric 2017-02-21 17:48:40 -05:00
Joseph Schorr
eece782038 Prevent peewee from loading the visibility every time 
By calling `visibility` instead of `visibility_id`, peewee was issuing a SQL Select statement for the repository, which removes the benefit of the optimization
 2017-02-17 12:09:48 -05:00
Joseph Schorr
421c5d6012 Fix bug where the login service ID doesn't exist 2017-02-16 16:27:53 -05:00
josephschorr
2a7d1fbe57 Merge pull request #2358 from coreos-inc/better-logging 
Log more information to the action logs and display the namespaces for superusers
 2017-02-14 16:38:35 -05:00
Charlton Austin
3fd8c8a60d feature(app.py): adding queue_metrics to queues 
publishing queue metrics for SRE

[none]
 2017-02-14 16:01:28 -05:00
Joseph Schorr
11c931f781 Log more information to the action logs and display the namespaces for superusers 
This helps superusers understand better what, exactly, is going on in the registry
 2017-02-14 14:55:24 -05:00
Charlton Austin
85bcb63439 update(security_test.py): moving tests to new framework 
We should be moving tests over to pytest

[none]
 2017-02-02 13:40:00 -05:00
Joseph Schorr
b407f88a26 Remove unnecessary CloudWatch metrics 
They are spamming the API and costing us a lot of money
 2017-02-01 13:08:21 -05:00
josephschorr
01ec22b362 Merge pull request #2300 from coreos-inc/openid-connect 
OpenID Connect support and OAuth login refactoring
 2017-01-31 18:14:44 -05:00
Joseph Schorr
3324743bff Fix db migration revision 2017-01-31 11:38:31 -05:00
Joseph Schorr
973a110ac7 Full text search for repository name and description 
Adds support for searching full text against the name and description of a repository

[Delivers #134867401]
 2017-01-31 11:38:31 -05:00
Joseph Schorr
d65d32b284 Convert model to use moved prefix_search method 2017-01-31 11:38:31 -05:00
Joseph Schorr
d89c79b92d Full text support in peewee 
Adds support for full text search in peewee with the creation of two new field types: `FullIndexedCharField` and `FullIndexedTextField`.

Note that this change depends upon https://github.com/zzzeek/sqlalchemy/pull/339

[Delivers #137453279]
[Delivers #137453317]
 2017-01-31 11:38:31 -05:00
Joseph Schorr
fda203e4d7 Add proper and tested OIDC support on the server 
Note that this will still not work on the client side; the followup CL for the client side is right after this one.
 2017-01-23 17:53:34 -05:00
Evan Cordell
28813159e5 fix(userevent): ignore subscribe notifications in userevents 
[Fixes #138007389]
 2017-01-20 13:38:02 -05:00
Joseph Schorr
71ec23b550 Switch QueueItem state_id to be unique after a backfill 2017-01-18 17:43:41 -05:00
josephschorr
e2748fccd9 Merge pull request #2282 from coreos-inc/motd-updates 
Severity and Markdown support in MOTD
 2017-01-18 17:41:27 -05:00
Joseph Schorr
3106504f39 Severity and Markdown support in MOTD 
[Delivers #133555165]
 2017-01-18 16:55:32 -05:00
Joseph Schorr
af23d2bedd Remove unique from queue item state_id 2017-01-18 15:04:26 -05:00
Joseph Schorr
3cf8f6c28a Cleanup user event reporting and lower its timeout 2017-01-18 11:27:00 -05:00
Joseph Schorr
462f47924e More detailed namespace validation 
Fixes namespace validation to use the proper regex for checking length, as well as showing the proper messaging if the entered namespace is invalid

[Delivers #137830461]
 2017-01-17 17:31:59 -05:00
josephschorr
aafcb592a6 Merge pull request #2257 from coreos-inc/clair-gc-take2 
feat(gc): Garbage collection for security scanning
 2017-01-17 14:49:36 -05:00
Joseph Schorr
8c4e86f48b Change queue to use state-field for claiming items 
Before this change, the queue code would check that none of the fields on the item to be claimed had changed between the time when the item was selected and the item is claimed. While this is a safe approach, it also causes quite a bit of lock contention in MySQL, because InnoDB will take a lock on *any* rows examined by the `where` clause of the `update`, even if they will ultimately thrown out due to other clauses (See: http://dev.mysql.com/doc/refman/5.7/en/innodb-locks-set.html: "A ..., an UPDATE, ... generally set record locks on every index record that is scanned in the processing of the SQL statement. It does not matter whether there are WHERE conditions in the statement that would exclude the row. InnoDB does not remember the exact WHERE condition, but only knows which index ranges were scanned").

As a result, we want to minimize the number of fields accessed in the `where` clause on an update to the QueueItem row. To do so, we introduce a new `state_id` column, which is updated on *every change* to the QueueItem rows with a unique, random value. We can then have the queue item claiming code simply check that the `state_id` column has not changed between the retrieval and claiming steps. This minimizes the number of columns being checked to two (`id` and `state_id`), and thus, should significantly reduce lock contention. Note that we can not (yet) reduce to just a single `state_id` column (which should work in theory), because we need to maintain backwards compatibility with existing items in the QueueItem table, which will be given empty `state_id` values when the migration in this change runs.

Also adds a number of tests for other queue operations that we want to make sure operate correctly following this change.

[Delivers #133632501]
 2017-01-17 13:29:26 -05:00
Joseph Schorr
19cb64df5d Remove unused class 2017-01-17 13:26:09 -05:00
Joseph Schorr
7f63cbd14f Remove FOR UPDATE in Queue cancel and complete 
We have no need for them anymore and it should reduce lock contention a bit

Fixes #776
 2017-01-17 13:26:09 -05:00
Charlton Austin
ca832df975 Adding in new indices for queueitem table. 2017-01-17 10:04:31 -05:00
Joseph Schorr
1cbacbbb63 Add tool for handling abusing users 2017-01-13 14:42:03 -05:00
Joseph Schorr
5225642850 Garbage collection image+storage callback support 
Add support to GC to invoke a callback with the image+storages removed. Only images whose storage was also removed will be sent to the callback. This will be used by security scanning for its own GC in the followup change.
 2016-12-22 14:27:42 -05:00
Joseph Schorr
e2efb6c458 Add default and configurable LDAP timeouts 
Fixes https://www.pivotaltracker.com/story/show/135885019
 2016-12-19 11:53:06 -05:00
Joseph Schorr
58b7481a63 Make sure robot accounts always show up first in entity search 
Fixes https://www.pivotaltracker.com/story/show/136277321
Fixes #2241
 2016-12-16 15:04:30 -05:00
Joseph Schorr
785c74de52 Fix attempts to confirm team invite for mismatched email address 
Currently, if a user tries to confirm an invite sent to them on an account with a mismatching email address, we simply redirect to the org (where they get a 403). This change ensures they get the proper error response message, and restyles the error page to be nicer.

Fixes #2227
Fixes https://www.pivotaltracker.com/story/show/136088507
 2016-12-15 17:15:11 -05:00
Joseph Schorr
624b2a8385 Have security scanner analyze only send notifications for *new* layers 
Following this change, anytime a layer is indexed by the security scanner, we only send notifications out if the layer previously had a security_indexed_engine value of `-1`, thus ensuring it has *never* been indexed previously. This will allow us to change to version of the security scanner upwards, and have all the images be re-indexed, without firing off notifications in a spammy manner.
 2016-12-13 23:17:11 -05:00
Evan Cordell
5686c80af1 Revert "Add GC of layers in Clair" 
This reverts 49872838ab
 2016-12-13 18:40:58 -05:00
Charlton Austin
0b8c2ef92f Removing an unused import. 2016-12-08 13:53:52 -05:00
josephschorr
410b9d74fc Merge pull request #2214 from coreos-inc/clair-gc 
Add GC of layers in Clair
 2016-12-07 17:58:21 -05:00
josephschorr
543d86ae10 Merge pull request #2221 from coreos-inc/fix-error-pages 
Have all error pages be rendered by Angular
 2016-12-07 17:53:14 -05:00
josephschorr
111b7b0788 Merge pull request #2206 from coreos-inc/ldap-user-search-fix 
Fix external auth returns for query_user calls
 2016-12-07 17:53:04 -05:00
josephschorr
c5111d7930 Merge pull request #2144 from coreos-inc/buildlogs-improvements 
Change the append build log method to execute the two calls via one pipelined connection
 2016-12-07 17:52:22 -05:00
Joseph Schorr
c06bba38de Have all error pages be rendered by Angular 
Fixes #2198

Fixes https://www.pivotaltracker.com/story/show/135724483
 2016-12-07 17:49:02 -05:00
Jimmy Zelinskie
00eafff747 Merge pull request #2204 from jzelinskie/429builds 
add rate limiting to build queues
 2016-12-07 15:03:31 -05:00
Joseph Schorr
3203fd6de1 Fix external auth returns for query_user calls 
Adds the missing field on the query_user calls, updates the external auth tests to ensure it is returned properly, and adds new end-to-end tests which call the external auth engines via the *API*, to ensure this doesn't break again
 2016-12-07 14:28:42 -05:00
Jimmy Zelinskie
ebbe58d311 replace prefix w/ canonical name list 2016-12-07 12:56:56 -05:00
Jimmy Zelinskie
c41de8ded6 build queue rate limiting: address PR comments 2016-12-06 20:40:54 -05:00
Joseph Schorr
49872838ab Add GC of layers in Clair 
Fixes https://www.pivotaltracker.com/story/show/135583207
 2016-12-06 19:52:56 -05:00
Jimmy Zelinskie
eb69abff8b build rate limiting: tests 2016-12-06 16:30:12 -05:00
Jimmy Zelinskie
57770493fa build rate limiting: use a rate 2016-12-06 16:30:12 -05:00
Jimmy Zelinskie
7877c6ab94 add rate limiting to build queues 2016-12-06 16:30:12 -05:00
Charlton Austin
0aa6e6cd58 Merge pull request #2203 from charltonaustin/fix_build_component_cleanup 
Adding in a cancel method to the build component so we can properly c…
 2016-12-06 14:13:10 -05:00
Jake Moshenko
21e3001446 Add a bulk insert for queue and notifications. 
Use it for Clair spawned notifications.
 2016-12-06 14:00:16 -05:00
Charlton Austin
c6be12e31e Adding in a cancel method to the build component so we can properly clean up the job task. 2016-12-06 13:37:49 -05:00
Jimmy Zelinskie
3a7119d499 Merge pull request #2209 from coreos-inc/clair-notification-read 
Clair notification read and queue fixes
 2016-12-05 19:36:59 -05:00
Joseph Schorr
97d150e281 Have QSS only add security scanner notifications once 2016-12-05 19:08:20 -05:00
Jake Moshenko
7c490b46c8 Only save dirty fields on Queue queries. 2016-12-05 18:12:14 -05:00
Charlton Austin
0a6322015c Fix the queue item delete. 2016-12-02 15:30:35 -05:00
Charlton Austin
7b3d8e3977 Merge pull request #2183 from charltonaustin/metrics_for_unscanned_images 
Adding in some metrics around clair sec scan.
 2016-12-02 11:50:29 -05:00
Charlton Austin
edd9dcd7f6 Adding in some metrics around clair sec scan. 2016-12-01 16:50:02 -05:00
Charlton Austin
1f03fcb146 Adding in notification type for notification kind. 2016-12-01 12:26:18 -05:00
Charlton Austin
2c637fe5ce Merge pull request #2173 from charltonaustin/adding_in_build_cancel_notifications 
Adding in cancel notifications
 2016-11-30 15:03:17 -05:00
Charlton Austin
4103a0b75f Adding in cancel notifications 2016-11-30 14:38:34 -05:00
Joseph Schorr
730a220eb0 Fix user lookup query under Postgres 
Adds a missing group_by clause
 2016-11-29 11:36:53 -05:00
Joseph Schorr
402ad25690 Change team invitation acceptance to join all invited teams under the org 
Fixes #1989
 2016-11-28 18:39:28 -05:00
Joseph Schorr
e29cb34336 Fix Set calls to gauges 
Fixes #2150

The proper function is `Set` (not `set`), which was causing these gauges to not report to Prometheus
 2016-11-21 15:27:17 -05:00
Charlton Austin
2fe74e4057 Adding in UI for cancel anytime. 2016-11-21 10:58:32 -05:00
Joseph Schorr
1b8820f2e7 Change the append build log method to execute the two calls via one pipelined connection 
Should reduce the amount of packets used by the build manager

Reference: https://github.com/andymccurdy/redis-py#pipelines
 2016-11-18 11:47:16 -05:00
Charlton Austin
fd7c566d31 Adding in cancel for a build that is building. 2016-11-16 17:40:24 -05:00
Joseph Schorr
1a61ef4e04 Report the user's name and company to Marketo 
Also fixes the API to report the other changes (username and email) as well
 2016-11-14 17:34:50 -05:00
Jake Moshenko
10255d4052 Merge pull request #2109 from jakedt/collapsemigrations 
Collapse all migrations prior to 2.0.0 into one.
 2016-11-10 17:35:07 -05:00
Jake Moshenko
b5834a8a66 Collapse all migrations prior to 2.0.0 into one. 2016-11-10 17:31:00 -05:00
Joseph Schorr
536809a992 Change LDAP errors into debug statements to reduce log clutter 
Fixes #2083
 2016-11-10 16:39:26 -05:00
Joseph Schorr
0f2eb61f4a Add collection of user metadata: name and company 2016-11-08 16:15:02 -05:00
josephschorr
233b2be5c2 Merge pull request #2066 from coreos-inc/select-username 
Add support for temp usernames and an interstitial to confirm username
 2016-11-03 16:22:16 -04:00
Joseph Schorr
1e3b354201 Add support for temp usernames and an interstitial to confirm username 
When a user now logs in for the first time for any external auth (LDAP, JWT, Keystone, Github, Google, Dex), they will be presented with a confirmation screen that affords them the opportunity to change their Quay-assigned username.

Addresses most of the user issues around #74
 2016-11-03 15:59:14 -04:00
Joseph Schorr
3fd92aef35 Fix entity search API to not IndexError 2016-11-02 16:22:35 -04:00
Joseph Schorr
d7f56350a4 Make email addresses optional in external auth if email feature is turned off 
Before this change, external auth such as Keystone would fail if a user without an email address tried to login, even if the email feature was disabled.
 2016-10-31 13:50:24 -04:00
josephschorr
934cdecbd6 Merge pull request #1905 from coreos-inc/external-auth-search 
Add support for entity search against external auth users not yet linked
 2016-10-27 16:06:42 -04:00
Joseph Schorr
b3d1d7227c Add support to Keystone Auth for external user linking 
Also adds Keystone V3 support
 2016-10-27 15:42:03 -04:00
Joseph Schorr
fbb524e34e Add support to ExternalJWT Auth for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
f9ee8d2bef Add support to LDAP for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
d145222812 Add support for linking to external users in entity search 2016-10-27 15:42:03 -04:00
Charlton Austin
2147005d2c Adding a method of cancelling a build based on etcd message. 2016-10-25 12:50:58 -04:00
Charlton Austin
dc35769396 Merge pull request #2022 from charltonaustin/refactor_for_cancel_anytime 
Making some refactors to make it easier to cancel the build at any time.
 2016-10-24 16:17:55 -04:00
Charlton Austin
1cde22e76c Making some refactors to make it easier to cancel the build at any time. 2016-10-24 15:59:33 -04:00
josephschorr
edc2bc8b93 Merge pull request #1698 from coreos-inc/delete-namespace 
Add support for deleting namespaces (users, organizations)
 2016-10-21 16:54:52 -04:00
Joseph Schorr
73eb66eac5 Add support for deleting namespaces (users, organizations) 
Fixes #102
Fixes #105
 2016-10-21 15:41:09 -04:00
josephschorr
4d89052bbf Merge pull request #1764 from coreos-inc/db-timeout 
Add a default database connect timeout
 2016-10-20 15:16:53 -04:00
Joseph Schorr
b7fc7999c3 Delete old "license" checking code arounds user counts 
This is legacy code that doesn't actually do anything of value
 2016-10-20 14:58:35 -04:00
Jimmy Zelinskie
20ef43d5fb workers.queuecleanup: remove direct peewee usage 2016-10-20 13:46:00 -04:00
Joseph Schorr
715fc27474 Add a default database connect timeout 
Fixes #1760
 2016-10-17 13:33:30 -04:00
Charlton Austin
97d644d95d Adding in the delete api and the delete and create UI. 2016-10-13 10:40:52 -04:00
charltonaustin
5a4b702888 Adding in security tests and docs. 2016-10-11 09:30:37 -04:00
josephschorr
7fc33a9a57 Merge pull request #1965 from coreos-inc/condense-slack-notifications 
Less verbose notifications for QSS
 2016-10-10 15:38:12 -04:00
Joseph Schorr
ebf4120326 Less verbose notifications for QSS 
Fixes #1914
 2016-10-10 15:18:49 -04:00
charltonaustin
14eb3005b6 Some fixes for code review. 2016-10-10 12:55:00 -04:00
charltonaustin
4ae6e6efa9 Fixing some database integration errors 2016-10-10 10:51:30 -04:00
charltonaustin
1e733ddffb Adding in a new message data model and the corresponding methods to in the API. 2016-10-07 15:56:58 -04:00
Joseph Schorr
0b7bb6d6c6 Fix issue in V1 registry code with accessing locations under HEAD 
Fixes #1922
 2016-10-03 17:09:12 +03:00
josephschorr
b4dd5ea4dd Merge pull request #1867 from coreos-inc/keystone-timeout 
Add configurable timeout and debug flags to Keystone users
 2016-09-29 23:01:02 +02:00
Joseph Schorr
02b8afe127 Add labeling of built manifests with their build IDs 
Also sends the digests to the notification

Fixes #593
 2016-09-29 10:58:45 +02:00
Jimmy Zelinskie
44eca10c05 update interfaces to use ABC 2016-09-26 14:50:24 -04:00
Jimmy Zelinskie
a1a930b833 database: fix indices post-rebase 2016-09-26 14:49:58 -04:00
Jimmy Zelinskie
ca883e5662 port label support to refactored v2 registry 2016-09-26 14:49:58 -04:00
Joseph Schorr
3c8b87e086 Fix verbs in manifestlist 
All registry_tests now pass
 2016-09-26 14:49:58 -04:00
Jimmy Zelinskie
783c9e7a73 stop exporting experimental database models 2016-09-26 14:49:23 -04:00
Jimmy Zelinskie
c35413d4f6 add boilerplate for verbs data interface 2016-09-26 14:49:23 -04:00
Jimmy Zelinskie
2e5a94bc0b create key server data interface 2016-09-26 14:49:23 -04:00
Jimmy Zelinskie
c06d395f96 create interfaces for v1 and v2 data model 2016-09-26 14:49:23 -04:00
Joseph Schorr
b775458d4b lifetimes on Tags should now be in milliseconds 
Fixes #1779
 2016-09-26 14:49:04 -04:00
Joseph Schorr
db60df827d Implement V2 interfaces and remaining V1 interfaces 
Also adds some tests to registry tests for V1 stuff.
Note: All *registry* tests currently pass, but as verbs are not yet converted, the verb tests in registry_tests.py currently fail.
 2016-09-26 14:49:04 -04:00
Jimmy Zelinskie
d67991987b v1: refactor index 2016-09-26 14:48:42 -04:00
Jimmy Zelinskie
b68e1b5efc add "get_" prefix to all db read funcs 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
32a6c22b43 mv data/types image 
This change also merges formats into the new image module.
 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
a516c08deb v2: refactor auth to use data.types 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
3de6000428 v2: refactor blob.py to use data.types 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
e6c99bb471 re-ordered BlobUploading fields 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
3f722f880e v2: add pagination decorator 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
5b630ebdb0 v2/manifest: refactor to use types 2016-09-26 14:48:05 -04:00
Joseph Schorr
ea18790dfe Get V1 registry code working with new model methods 2016-09-26 14:47:06 -04:00
Joseph Schorr
94d71f2166 Fix model to actually initialize 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
8435c254c3 finish v1 registry refactor 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
c14437e54a initial v1 refactor to use model methods 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
9cfd6ec452 database: initial manifestlist schema changes 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
e3a39d7bd6 fix indentation 2016-09-26 14:47:06 -04:00
josephschorr
ad4efba802 Merge pull request #1830 from coreos-inc/superuser-dashboard 
Add prometheus stats to enable better dashboarding
 2016-09-26 17:19:22 +02:00
Joseph Schorr
fd770422bb Add configurable timeout and debug flags to Keystone users 
Fixes #1855
 2016-09-22 18:25:02 -04:00
Joseph Schorr
30af8aef1a Add a worker for reporting global stats to Prometheus 
Fixes #1789
 2016-09-12 16:19:19 -04:00
Jake Moshenko
91963c17a0 Remove a join to slightly optimize the gc query. 2016-09-09 15:40:40 -04:00
Joseph Schorr
3d542b5e93 Handle KeyError nicer in _get_parent_image 
Fixes #1810
 2016-09-09 13:34:56 -04:00
Jake Moshenko
cf83c9a16a Improve the garbage collection tests. 2016-09-07 13:25:19 -04:00
Jake Moshenko
584a5a7ddd Reduce database bandwidth by tracking gc candidate images. 2016-09-07 13:25:19 -04:00
Jake Moshenko
0815f6b6c4 Fix indentation for DB queries. 2016-09-07 10:48:58 -04:00
Jake Moshenko
1d8b72235a Add a helper method to Image to parse ancestor string. 2016-09-07 10:48:58 -04:00
josephschorr
cd8b45e25b Merge pull request #1754 from coreos-inc/team-add-perms 
Better UI and permissions handling for robots and teams
 2016-09-06 17:21:19 -04:00
Joseph Schorr
b4939a3cd0 Fix filtering of repos only visible to org admins 2016-08-31 13:51:53 -04:00
Joseph Schorr
357005e33f Raise a 409 if we try to insert a tag twice at the same time 
Also fixes handling of labels for existing manifests

Fixes #1775
 2016-08-29 16:03:35 -04:00
Joseph Schorr
1a2666be07 Fix deletion of labels and add tests 2016-08-26 16:07:49 -04:00
Joseph Schorr
608ffd9663 Basic labels support 
Adds basic labels support to the registry code (V2), and the API. Note that this does not yet add any UI related support.
 2016-08-26 15:24:26 -04:00
Joseph Schorr
391d70d9ec Add repo permissions dialog for existing teams and robots 
Fixes #1686
 2016-08-22 14:43:12 -04:00
Joseph Schorr
6ebb417923 Redesign the teams page to use a table 
Allows for faster loading and easier viewing of important information about teams
 2016-08-22 14:42:54 -04:00
Jake Moshenko
d6a396be34 Fix all foreign key constraints to use naming convention. 2016-08-18 14:29:53 -04:00
Joseph Schorr
aeddc6af06 Handle GC constraint failures in a nicer way 
Fixes #1739
 2016-08-17 16:13:27 -04:00
josephschorr
2caa82d091 Merge pull request #1713 from coreos-inc/enable-iam 
Enable IAM support for S3 storage
 2016-08-16 16:13:29 -04:00
Joseph Schorr
7f5b536ddb Fix pagination of repositories 
Fixes #1725
 2016-08-15 16:48:04 -04:00
Joseph Schorr
0f46230493 Add an index for lookup by account to log entries 
Also fixes the query to require one less join
 2016-08-12 17:39:31 -04:00
Joseph Schorr
855cc36057 Remove unneeded imports 2016-08-11 17:16:31 -04:00
Joseph Schorr
34d49e2d44 Fix duplicate derived storage cache creation issue 
Fixes #1699
 2016-08-10 16:18:52 -04:00
Joseph Schorr
4a2acac5dc Fix pagination of public repos, make more efficient and add test 2016-08-10 15:08:06 -04:00
Joseph Schorr
bf8f621278 Temporarily remove the migration which drops the foreign keys on LogEntry, as it is invalid 2016-08-08 17:47:04 -04:00
josephschorr
1a137ee7b3 Merge pull request #1643 from coreos-inc/db-retry 
Enable automatic retry for the database
 2016-08-08 15:04:25 -04:00
Joseph Schorr
700e7b74e4 Enable automatic retry for the database 2016-08-08 15:02:42 -04:00
Jimmy Zelinskie
22a25ac2d3 Revert "Merge pull request #1678 from coreos-inc/delete-repo-fix" 
This reverts commit df64caf133, reversing
changes made to 0d1e453566.
 2016-08-08 12:38:15 -04:00
Jimmy Zelinskie
ce14b9dddf modify log_action to internally resolve IDs 2016-08-08 12:38:15 -04:00
Jimmy Zelinskie
052c31752b MIGRATION: drop foreign keys on logentry table 
This migration generates the following for MySQL:

BEGIN;

-- Running upgrade 1093d8b212bb -> 6243159408b5

ALTER TABLE logentry DROP FOREIGN KEY fk_logentry_account_id_user;

ALTER TABLE logentry DROP FOREIGN KEY
fk_logentry_repository_id_repository;

ALTER TABLE logentry DROP FOREIGN KEY fk_logentry_performer_id_user;

UPDATE alembic_version SET version_num='6243159408b5' WHERE
alembic_version.version_num = '1093d8b212bb';

COMMIT;
 2016-08-08 12:38:15 -04:00
Jimmy Zelinskie
e05bc8bf7d migration.sh: default DOCKER_IP to localhost 2016-08-08 12:36:01 -04:00
josephschorr
6716a2562b Merge pull request #1680 from coreos-inc/add-missing-index 
Add various missing indexes
 2016-08-08 12:34:58 -04:00
Joseph Schorr
80a37fd295 Add various missing indexes 
Indexes added:

Image::repository - Needed for model.image.get_repository_images_without_placements
RepositoryTag::image - Needed for model.tag.get_tags_for_image
RepositoryTag::repository - Needed for repository deletion
RepositoryBuild::phase - Needed for model.build.list_repository_builds sorting
RepositoryBuild::started - Needed for model.build.list_repository_builds sorting
RepositoryBuild::repository+started+phase - Needed for model.build.list_repository_builds
RepositoryBuild::started+logs_archived+phase - Needed for model.build.get_archivable_build lookup
 2016-08-08 12:34:45 -04:00
josephschorr
df64caf133 Merge pull request #1678 from coreos-inc/delete-repo-fix 
Have repo deletion not lock all the things
 2016-08-04 16:48:03 -04:00
Joseph Schorr
0b5cd95693 Have repo deletion not lock all the things 2016-08-04 16:45:59 -04:00
Joseph Schorr
b1b0da7afd Fix off-by-one error in repo tags pagination 
Fixes #1665
 2016-08-02 14:17:33 -04:00
Jake Moshenko
05e2773fa7 Get rid of remaining slow query for garbage collection. 2016-08-01 18:22:38 -04:00
josephschorr
46a28617e8 Merge pull request #1651 from coreos-inc/fix-branches 
Fix handling of multi-part branches in the build triggers
 2016-07-26 16:00:21 -07:00
Joseph Schorr
9e4f8cac03 Optimize GC query for looking up deletable storages 2016-07-26 13:47:15 -07:00
Joseph Schorr
06d52f2c83 Fix handling of multi-part branches in the build triggers 
Fixes #1360
 2016-07-26 13:41:13 -07:00
Joseph Schorr
5de1e98d3c Fix LDAP DN building for empty RDN list 2016-07-22 14:40:53 -04:00
Joseph Schorr
4d6f96cd6c Add missing pass keyword 2016-07-19 22:24:27 -04:00
Joseph Schorr
b8d2570725 Don't raise an error on duplicate placements 
This can happen if two pushes are racing on the same storage.
 2016-07-19 16:44:05 -04:00
Joseph Schorr
b0b7b63be9 Fix queue tests for MySQL 
MySQL's date time's appear to have a 1 second threshold, so we need to make sure the queue items added for the tests are available as soon as they are added. Before this change, the available_after was set to `datetime.utcnow()`, and, if the `get` was called within 1 second, then its check would fail.
 2016-07-15 13:27:50 -04:00
Joseph Schorr
4e1259b58a Fix the Repository ID in pagination problem once and for all 
But.... ONCE AND FOR ALL!

Note: Tested on SQLite, Postgres and MySQL
 2016-07-14 17:09:52 -04:00
Jimmy Zelinskie
64d0c5b675 data.queue: fix race condition 
It's possible that multiple consumers will acquire a queue item if they
race on an expired item. To mitigate this, we check that the
processing_expires time hasn't been changed since we last read.
 2016-07-14 15:34:22 -04:00
Jimmy Zelinskie
609f4fccd8 data.queue: simplify put method 2016-07-14 15:34:22 -04:00
Joseph Schorr
c1e4bf79b7 Fix delete team error message for admin teams 2016-07-11 15:47:05 -04:00
Joseph Schorr
241ebaa084 Fix typo 2016-07-07 15:06:29 -04:00
Joseph Schorr
adaeeba5d0 Allow for multiple user RDNs in LDAP 
Fixes #1600
 2016-07-07 14:46:38 -04:00
Joseph Schorr
e252ee07cb Fix popularity metrics on list repos API 2016-07-06 16:15:54 -04:00
Joseph Schorr
713ba3abaf Further updates to the Prometheus client code 2016-07-01 14:16:51 -04:00
Jake Moshenko
668a8edc50 Refactor prometheus integration 
Move prometheus to SaaS and make it a plugin
Move static callers to use metrics_queue plugin
Change local-docker to support different quay clone dirnames
Change prom_aggregator to use logrus
 2016-07-01 14:16:50 -04:00
Matt Jibson
3d9acf2fff Use prometheus as a metric backend 
This entails writing a metric aggregation program since each worker has its
own memory, and thus own metrics because of python gunicorn. The python
client is a simple wrapper that makes web requests to it.
 2016-07-01 14:16:50 -04:00
Joseph Schorr
117ccda1cf Fix postgres error in SQL query 2016-07-01 13:04:20 -04:00
Joseph Schorr
1eec6f53b2 Fix SQL error with pagination around Repositories 
Fixes #1591
 2016-06-30 17:31:35 -04:00
Joseph Schorr
310ecd11cc Handle user events Redis not working in tutorial 
Also does some basic restyling

Fixes #1586
 2016-06-28 17:04:31 -04:00
Joseph Schorr
853cca35f3 Change repo stats to use the RAC table and a nice UI 2016-06-22 15:06:53 -04:00
josephschorr
7173d53030 Merge pull request #1549 from coreos-inc/certs 
Switch to install custom LDAP cert by name
 2016-06-21 15:13:44 -04:00
Joseph Schorr
66ec1d81ce Switch to install custom LDAP cert by name 2016-06-21 15:10:26 -04:00
josephschorr
9e6a264f5f Merge pull request #1523 from coreos-inc/verb-tag-cache-fix 
Add a uniqueness hash to derived image storage to break caching over …
 2016-06-20 16:38:25 -04:00
Joseph Schorr
a43b741f1b Add a uniqueness hash to derived image storage to break caching over tags 
This allows converted ACIs and squashed images to be unique based on the specified tag.

Fixes #92
 2016-06-20 16:34:52 -04:00
Joseph Schorr
3b994431eb Auto expire the build status and logs in redis 2016-06-20 13:53:13 -04:00
Joseph Schorr
986d20bcad Switch to generic RedisError 
Fixes #1558
 2016-06-20 11:20:17 -04:00
Jake Moshenko
a1cf12e460 Add a sitemap.txt for popular public repos 
and reference it from the robots.txt
 2016-06-17 14:34:20 -04:00
josephschorr
614b9124ae Merge pull request #1512 from coreos-inc/optimize-queries 
Optimize various queries
 2016-06-16 14:22:59 -04:00
josephschorr
58bef472d9 Merge pull request #1526 from coreos-inc/superuser-grant 
Add ability for super users to take ownership of namespaces
 2016-06-13 16:23:10 -04:00
Joseph Schorr
20816804e5 Add ability for super users to take ownership of namespaces 
Fixes #1395
 2016-06-13 16:22:52 -04:00
josephschorr
bda5d7ae29 Merge pull request #1511 from coreos-inc/location-cache 
Use a cache for ImageStorageLocation
 2016-06-09 14:03:07 -04:00
Joseph Schorr
7aa6b812e2 Use a cache for ImageStorageLocation 
No need to reload it from the DB or join as it is a static set only changed during migration
 2016-06-09 14:02:42 -04:00
Joseph Schorr
8887f09ba8 Use the instance service key for registry JWT signing 2016-06-07 11:58:10 -04:00
Joseph Schorr
894b5fed6f Remove TODO since we always need storage 2016-06-03 13:45:13 -04:00
Joseph Schorr
03fd2ea15a Remove Image from _load_tag_manifests query 
Doesn't appear used or necessary
 2016-06-03 13:44:01 -04:00
Joseph Schorr
9a747ca6a0 Have get_parent_images not join on placements 
The only case that needs the placements is in verbs, for which we use a new method
 2016-06-03 13:33:15 -04:00
Joseph Schorr
8064419715 Remove Image join from get_active_tag 
It isn't used anywhere in the query and appears to be completely unnecessary
 2016-06-03 13:06:57 -04:00
Joseph Schorr
53538f9001 Optimize get_tag_image query 
No caller uses the image placements or locations, so no need to load them.
 2016-06-02 16:36:38 -04:00
josephschorr
cad8746f9d Merge pull request #1502 from coreos-inc/image-replication 
Enable storage replication for V2 and add backfill tool
 2016-06-02 15:02:53 -04:00
Joseph Schorr
12924784ce Enable storage replication for V2 and add backfill tool 
Fixes #1501
 2016-06-02 14:36:08 -04:00
josephschorr
a85c3ebff7 Merge pull request #1457 from coreos-inc/xauth 
Add support for direct granting of OAuth tokens and add tests
 2016-06-01 12:07:12 -04:00
josephschorr
1ddc73416c Merge pull request #1500 from coreos-inc/better-errors 
Better errors
 2016-05-31 15:54:41 -04:00
Jimmy Zelinskie
1f488acf12 data.queue: move name matching clause 2016-05-31 15:44:11 -04:00
Joseph Schorr
04df2410ec Add better errors if Redis is down 
Fixes #1497
 2016-05-31 15:24:36 -04:00
Jimmy Zelinskie
26300d3c8e data.queue: lint 2016-05-27 14:51:19 -04:00
Jimmy Zelinskie
8a5aa65d74 data.queue: limiting before order by rand 2016-05-27 14:44:30 -04:00
Jimmy Zelinskie
44b56ae2cf queue: explicitly declare ordering requirement 
This change defaults the ordering requirement of queue items to be off
and only enables it for the build manager. This should make the queries
for getting queueitems significantly faster for every other use case.
 2016-05-27 14:44:30 -04:00
Joseph Schorr
7933aecf25 Add support for direct granting of OAuth tokens and add tests 
This allows a client (when authorized in a whitelist) to send direct credentials via a Basic auth header and therefore bypass the OAuth approval UI for that user.
 2016-05-23 17:17:06 -04:00
Joseph Schorr
60bbca2185 Fix setup tool when binding to external auth 
We now query the external auth provider for the external service's identifier before adding the linking row into the database. This fixes the case where the external service resolves a different identifier for the same username.

Fixes #1477
 2016-05-23 17:11:36 -04:00
Joseph Schorr
043699cfb3 Always use log entry kind cache 
Fixes #1445
 2016-05-13 15:20:55 -04:00
Jimmy Zelinskie
972e4be811 log: cutoff at the max id past the cutoff_date 
Previously we were using the min, which is always going to be equivalant
to the min id in the table.
 2016-05-10 20:13:10 -07:00
Joseph Schorr
a736407611 Fix user:admin scope handling and add test 2016-05-09 11:16:01 +02:00
josephschorr
f55fd2049f Merge pull request #1433 from coreos-inc/ldapoptions 
Add additional options for LDAP
 2016-05-04 14:06:29 -04:00
Joseph Schorr
42515ed9ec Add additional options for LDAP 
Fixes #1420
 2016-05-04 13:59:20 -04:00
Joseph Schorr
6e2df3b339 Fix key server to not list expired keys 
Fixes the key server to not list expire keys and by default not return expired or unapproved keys unless explicitly requested.

Fixes #1430
 2016-05-03 17:58:47 -04:00
Jimmy Zelinskie
2aa88dcb80 only send notifications when superusers enabled 2016-04-29 15:42:25 -04:00
Jimmy Zelinskie
29e2d7c9d4 data.model.log: remove unused method 2016-04-29 14:22:53 -04:00
Jimmy Zelinskie
e47b29a974 migration: add missing delete from down migration 
This also reorganizes the file a bit.
 2016-04-29 14:10:33 -04:00
Jimmy Zelinskie
4a521f5844 database: revert logentry foreign key proxy 2016-04-29 14:10:33 -04:00
Evan Cordell
489752a0b7 Only refresh current instance service key 2016-04-29 14:10:33 -04:00
Evan Cordell
a6f6a114c2 service key worker to refresh automatic keys 2016-04-29 14:10:33 -04:00
Evan Cordell
2242c6773d Add 'Automatic' ServiceKeyApprovalType 2016-04-29 14:10:33 -04:00
Joseph Schorr
6091db983b Hide expired keys outside of their staleness window 2016-04-29 14:10:33 -04:00
Joseph Schorr
4f63a50a17 Change account-less logs to use a user and not null 
This allows us to skip the migration
 2016-04-29 14:09:37 -04:00
Jimmy Zelinskie
5cb6ba4d12 keyserver migration: fix constraint name 2016-04-29 14:09:37 -04:00
Joseph Schorr
28a80ef6a9 Make sure to verify service names on key creation 2016-04-29 14:09:37 -04:00
Joseph Schorr
522cf68c5d Lots of smaller fixes: 
- Add the rotation_duration to the keys API
- Have the key service UI use the new rotation_duration field
- Fix notification deletion lookup path
- Add proper support for the new notification in the UI
- Only delete expired keys after 7 days (configurable)
- Fix angular digest loop
- Fix unit tests
- Regenerate initdb
 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
370ac3ecd0 service keys: add rotation_duration field 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
6577ac3e62 mv JWK-canonicalization util.security.fingerprint 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
4020ab9f55 service keys: delete notifications by prefix 2016-04-29 14:05:16 -04:00
Joseph Schorr
08017c5111 Further UI updates 2016-04-29 14:05:16 -04:00
Joseph Schorr
a4a01e76c0 Fix up the migration to include the additional changes needed 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
d19eb16b45 keyserver: add generate key function 
The superuser API, initdb, and tests will all need this functionality.
 2016-04-29 14:05:16 -04:00
Joseph Schorr
fb1dca4e94 Add API usage tests 2016-04-29 14:05:16 -04:00
Joseph Schorr
11ff3e9b59 keys ui WIP 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
885a41e6f5 key server: misc fixes to make jwtproxy work 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
5cdc7812dc migration.sh: update to reflect timing 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
50ad1bb6b1 key server: misc cleanup to get it working 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
c79bb14049 service keys: fix stale query 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
86df0124c1 service keys: join with approvals 
Also fixes a bug where we weren't reassigning the query after adding a
WHERE.
 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
dff59b4a39 service key migration 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
c6b8b3ce8c service_keys: s/get_keys/list_keys 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
0ec54fc70e clear notifications on delete/replace service_key 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
42b5196b21 add notification path and use for service keys 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
f406942984 converging on proper rotation 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
aaf9e83278 basically finish superuser key api 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
35ed73e195 rework superuser api 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
4079dba167 service keys: do all the right stuff 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
6ecff950ab service keys: add txs and select4update 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
499bb16306 service key server wip 2016-04-29 13:38:25 -04:00
josephschorr
9e88b1413d Merge pull request #1325 from coreos-inc/blobuncompressedsize 
Fix uncompressed size for blob store and add test
 2016-04-28 13:15:33 -04:00
Jimmy Zelinskie
7239c465bf improve stale cutoff id perf (#1392) 2016-04-20 15:03:06 -04:00
josephschorr
b0cc55276f Merge pull request #1373 from coreos-inc/orgconvert 
Org conversion improvements
 2016-04-19 16:16:35 -04:00
Jimmy Zelinskie
5585e16c90 Merge pull request #1356 from jzelinskie/actionlogarchive 
logrotateworker: save to storage via userfiles
 2016-04-15 13:57:11 -04:00
Jimmy Zelinskie
3d190b786f userfiles: make handler optional 2016-04-15 13:56:07 -04:00
Joseph Schorr
c604dbd0f6 Fix permissions when converting a user to an org 
Fixes #1366
 2016-04-14 17:39:45 -04:00
Joseph Schorr
1009362d26 Have recovery auto-verify the user 
Fixes #1355
 2016-04-08 13:41:16 -04:00
Jake Moshenko
bd5b44cbd2 Move the sequence fixer to a separate tool which can be run 2016-04-01 13:46:13 -04:00
josephschorr
edb157c5cb Merge pull request #1294 from coreos-inc/partialperms 
Change permissions to only load required by default
 2016-03-30 16:40:40 -04:00
Joseph Schorr
42e934d84f Make notification lookup faster and fix repo pagination on Postgres 2016-03-30 14:46:31 -04:00
Joseph Schorr
0dffdb87c9 Fix uncompressed size for blob store and add test 2016-03-29 14:16:56 -04:00
Joseph Schorr
a3aa4592cf Change permissions to only load required by default 
Permissions now load just the namespace and/or repository permissions requested, with a fallback to a full permissions load if necessary.
 2016-03-28 16:33:32 -04:00
Jake Moshenko
eed07722cb Add even larger plans for enterprises on SaaS 2016-03-21 16:38:34 -04:00
Jake Moshenko
fe2cd240bc Revert "Remove old search API which is no longer in use" 2016-03-07 10:07:41 -05:00
Jimmy Zelinskie
b5d904f373 Merge pull request #1218 from jzelinskie/logrotate5ever 
vastly simplify log rotation
 2016-03-04 13:48:21 -05:00
josephschorr
57430a18b4 Merge pull request #1224 from coreos-inc/removeoldsearch 
Remove old search API which is no longer in use
 2016-03-04 12:05:07 -05:00
Joseph Schorr
f498e92d58 Implement against new Clair paginated notification system 2016-02-25 15:58:42 -05:00
Joseph Schorr
c0374d71c9 Refactor the security worker and API calls and add a bunch of tests 2016-02-25 12:29:41 -05:00
Quentin Machu
0183c519f7 Merge pull request #1253 from Quentin-M/clair2 
Adapt securityworker, secscan API and Quay UI for Clair 1.0
 2016-02-19 18:21:25 -05:00
Quentin Machu
e5da33578c Adapt security worker for Clair v1.0 (except notifications) 2016-02-19 17:44:14 -05:00
Joseph Schorr
abd2e3c234 V1 Docker ID <-> V2 layer SHA mismatch fix 
Fix handling of V1 Docker ID <-> V2 layer SHA mismatch by dynamically rewriting the manifest to use new synthesized IDs for all layers above the mismatch. Also adds a bunch of tests for this and other use cases, fixes a bug around manifest digest uniqueness and fixes the 5.5 migration for MySQL.
 2016-02-12 17:39:27 +02:00
josephschorr
a9c64545fa Merge pull request #1228 from coreos-inc/v2storagevalidation 
Add a check that will fail if we try to mislink V1 layers
 2016-02-11 22:49:33 +02:00
Joseph Schorr
27f1cc0a13 Add a check that will fail if we try to mislink V1 layers 
Also logs some useful information
 2016-02-11 22:40:00 +02:00
Jake Moshenko
59a6f5bc77 Replace incompatible MySQL 5.5 server_default 2016-02-11 15:07:16 -05:00
Joseph Schorr
1887dc879c Remove old search API which is no longer in use 2016-02-10 15:02:27 +02:00
Jimmy Zelinskie
ee705fe7a9 vastly simplify log rotation 2016-02-09 18:20:14 -05:00
Joseph Schorr
4e771e667f Change sec scan candidate query to match parents to the expected version only 2016-02-09 22:23:48 +02:00
Joseph Schorr
534ec9cb2b Add pagination to the repository list API to make it better for public 
Fixes #1166
 2016-02-01 22:42:44 +02:00
Joseph Schorr
1536709c02 Small fixes 2016-01-29 20:01:17 +02:00
Joseph Schorr
a80ac8eabb Fix import for alembic 2016-01-29 17:59:23 +02:00
Joseph Schorr
bd0a098282 Add ID-based pagination to logs using new decorators and an encrypted token 
Fixes #599
 2016-01-26 12:50:48 -05:00
Jimmy Zelinskie
e54b86c6eb s/TORRENT/BITTORRENT 2016-01-22 15:52:28 -05:00
Jake Moshenko
fe2bdeb6cb Require some data from all models in initdb 2016-01-19 15:30:27 -05:00
Jake Moshenko
1b392dcb9a Remove dependent signatures before removing image storages 2016-01-19 14:56:02 -05:00
Jake Moshenko
7d0be20842 Formatting and syntax improvements 2016-01-19 14:56:02 -05:00
Joseph Schorr
e4da61a05d Fix piece hash calculation 2016-01-12 17:44:19 -05:00
josephschorr
047c2c2c0f Merge pull request #1129 from coreos-inc/backfill 
Add checksum and torrent info backfill
 2016-01-12 14:20:58 -05:00
Jake Moshenko
96c72e73df Clean up torrents before removing referenced storages 2016-01-12 11:43:07 -05:00
Jake Moshenko
8ab6c8a22d Fix torrent hash generation to work in mixed stacks 2016-01-11 16:43:46 -05:00
Joseph Schorr
c36a7c21c8 Order sadly matters with this check in peewee 2016-01-11 15:10:46 -05:00
Joseph Schorr
bd715c0c71 Add checksum and torrent info backfill 2016-01-08 17:32:30 -05:00
Jake Moshenko
1ae101c917 Address torrent feature review comments. 2016-01-08 16:38:21 -05:00
Jake Moshenko
073b68cf0d Fix torrent migration and update backfill to compute torrent pieces 2016-01-08 11:15:34 -05:00
Jake Moshenko
77aa58996a Fix the db definition for torrentinfo and add migration 2016-01-06 14:04:03 -05:00
Jake Moshenko
fd1e5f2407 Remove an unnecessary outer join 2016-01-05 14:43:40 -05:00
Jake Moshenko
8f80d7064b Hash v1 uploads for torrent chunks 2016-01-05 14:43:40 -05:00
Jake Moshenko
8d5f4466d6 Cleanup some indentation and imports 2016-01-05 12:12:57 -05:00
Jimmy Zelinskie
9b0a84c02f implement get_torrent_info 2016-01-04 16:17:51 -05:00
Jake Moshenko
a9b7ac6b48 Rotate robot user uuid when the credentials change 2016-01-04 16:17:51 -05:00
Jake Moshenko
ce8fcbeaae Update the pieces to use base64 encoded binary 2016-01-04 16:17:51 -05:00
Jake Moshenko
5c6e033d21 Fix indentation 2016-01-04 16:17:51 -05:00
Jake Moshenko
fe87d3c796 Hash and track layer file chunks for torrenting 2016-01-04 16:17:51 -05:00
josephschorr
f748d4348d Merge pull request #1106 from coreos-inc/billingemail 
Add support for custom billing invoice email address
 2016-01-04 14:34:30 -05:00
Joseph Schorr
31a8a0fba4 Better UX when recovering organization emails 
Fixes #291
 2015-12-28 15:25:31 -05:00
Joseph Schorr
10efa96009 Add support for custom billing invoice email address 
Fixes #782
 2015-12-28 13:59:50 -05:00
Joseph Schorr
01723d5546 Catch other cases where the queue item has been removed 
Fixes #1096
 2015-12-22 15:58:51 -05:00
Jake Moshenko
9c1a2e7e1b Improve performance by removing unnecessary group by fields 2015-12-22 11:35:49 -05:00
josephschorr
5ac7369bf5 Merge pull request #1068 from coreos-inc/slowqueryfix 
Remove check for derived image storages on image storage
 2015-12-18 16:32:22 -05:00
Joseph Schorr
94ece129d4 Remove remaining recursive queries on repo delete and add test 2015-12-18 16:04:03 -05:00
josephschorr
16f814b7d9 Merge pull request #1075 from coreos-inc/userdeletefix 
Fix user deletion under MySQL
 2015-12-17 15:09:18 -05:00
Joseph Schorr
2e7835c372 Fix user deletion under MySQL 
Fixes #973
 2015-12-17 15:05:15 -05:00
Jimmy Zelinskie
e1f955a3f6 add a log rotation worker 
Fixes #609.
 2015-12-16 17:22:28 -05:00
Joseph Schorr
f59f4e51e8 Remove check for derived image storages on image storage 
Derived image storages are now 1-to-1 with image storages, so we know they have already been removed at this point

Fixes #1067
 2015-12-16 13:41:25 -05:00
Joseph Schorr
73531d08b5 Add server default for the chunk_count column 2015-12-15 15:44:33 -05:00
Joseph Schorr
141f664bf7 Fix subquery delete which messes up MySQL 
Fixes #1061
 2015-12-15 13:15:10 -05:00
Joseph Schorr
9698d6f6a0 Add created column to blob upload 
Fixes first half of #1054
 2015-12-14 15:27:48 -05:00
Joseph Schorr
54095eb5cb Handle the common case of one chunk when calculating the uncompressed size 
Reference #992
 2015-12-14 15:27:48 -05:00
josephschorr
94effb5aaa Merge pull request #1023 from coreos-inc/getblobopt 
Optimize blob lookup
 2015-12-04 16:11:28 -05:00
Jake Moshenko
38cb63d195 Fix indentation on build model operations 2015-12-04 15:46:07 -05:00
Joseph Schorr
f07b940bc5 Optimize blob lookup 
Fixes #1013
 2015-12-04 14:47:09 -05:00
Joseph Schorr
c324ebd7f6 Only write exceptions for manifest gen when a tag exists 
Fixes #1019

Currently, we just raise an exception to the logs regardless, which can make it appear as if there is an issue (when there isn't).
 2015-12-03 16:04:17 -05:00
Silas Sewell
502e4c04d0 Fix seq migration down_revision 2015-11-30 17:59:04 -05:00