vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
8105 commits 2 branches 62 tags 205 MiB
Commit graph

1424 commits

Author SHA1 Message Date
Joseph Schorr
74dd0ef8e8 Add a batch get_matching_tags_for_images method 
This will be used in the security notification worker to retrieving the tags needed in a set of batch calls, rather than multiple calls per image
 2017-05-02 15:38:25 -04:00
Joseph Schorr
b67113e848 Move LDAP controls init into the inner loop 
We cannot use it across different DNs, so we need to move it down
 2017-05-01 16:04:33 -04:00
Joseph Schorr
30a681343f Make sure to escape LDAP queries 
Fixes an issue in team sync around group names that contain *s

Fixes https://www.pivotaltracker.com/story/show/144628235
 2017-05-01 14:00:54 -04:00
Joseph Schorr
e583be3914 Remove inner query for ancestors lookup on get_matching_tags 2017-04-28 20:10:54 -04:00
Joseph Schorr
8b2e4d3bcf Add a test for get_matching_tags 2017-04-28 19:57:24 -04:00
Joseph Schorr
e9ffe0e27b Implement new search UI 
We now have both autocomplete-based searching for quick results, as well as a full search page for a full listing of results
 2017-04-28 13:57:28 -04:00
Charlton Austin
f1d6a7284d Merge pull request #2483 from charltonaustin/phase_four_config 
feat(data): remove subdir
 2017-04-28 13:24:22 -04:00
Charlton Austin
c79711b6dc feat(data): remove subdir 
### Description of Changes
  This is the last step in the four phase migration of the config
 2017-04-28 13:23:51 -04:00
josephschorr
8b148bf1d4 Merge pull request #2576 from coreos-inc/full-db-tests-tox 
Reenable full database testing locally and in concourse
 2017-04-27 18:09:15 -04:00
Joseph Schorr
dd1addee29 LDAP Team sync improvements 
- Add a large amount of additional logging
- Handle NO_SUCH_OBJECT in AD searches
- Only check if *a* record exists when adding syncing, as opposed to loading the entire search set
 2017-04-26 20:26:12 -04:00
EvB
fddcd0a395 fix(data.archivedlogs): update endpoint name 2017-04-26 17:31:44 -04:00
josephschorr
5c4f7d50c6 Merge pull request #2580 from coreos-inc/team-sync-email-fix 
Fix handling of team sync when a user already exists with the email address
 2017-04-26 14:19:23 -04:00
EvB
5e995fae20 refactor(archivedlogs): move archivelog handler to endpoints 2017-04-26 11:41:55 -04:00
Joseph Schorr
36f2272fe2 Fix handling of team sync when a user already exists with the email address 2017-04-25 17:42:35 -04:00
Joseph Schorr
d7f3ef96ce Small fixes found by running full db tests 2017-04-24 16:45:15 -04:00
Joseph Schorr
7debd44b54 Switch fixture imports to wildcard in prep for full db test fixes 2017-04-24 16:45:14 -04:00
Jake Moshenko
a159bd3e77 Resolve race condition between multiple log archivers 2017-04-24 13:41:08 -04:00
Antoine Legrand
8499612c4c Merge pull request #2538 from coreos-inc/enable-robot-cnr 
Enable robot cnr
 2017-04-24 17:32:46 +02:00
josephschorr
cc88cfb6ad Merge pull request #2568 from coreos-inc/fix-postgres-search 
Fix search in postgres
 2017-04-21 17:22:59 -04:00
Joseph Schorr
e46e668cc5 Fix search in postgres 
Stupid missing group_by again
 2017-04-21 17:21:17 -04:00
Erica
fc5c255965 Merge pull request #2565 from coreos-inc/HOTFIX-2.3.1 
fix(migration): bring hotfix from v2.3.0 into master
 2017-04-21 17:12:51 -04:00
Joseph Schorr
5bba018009 Add tests for user files endpoint and add regex filter 2017-04-21 16:13:18 -04:00
Joseph Schorr
4bb725dce0 Add fix for relative paths in user files lookup and add test 2017-04-21 15:54:15 -04:00
Joseph Schorr
3413fc3d8a Add missing logging for IOError issue in archived logs 2017-04-21 14:40:09 -04:00
EvB
c5b411b704 fix(migrations): fix column additional w/ boolean default 2017-04-21 11:56:13 -04:00
Joseph Schorr
3dcbe3c631 If enabled, allow users and orgs to set their time machine expiration 
Fixes https://www.pivotaltracker.com/story/show/142881203
 2017-04-21 11:32:45 -04:00
Joseph Schorr
c5bb9abf11 Fix deleting repos when sec scan or signing is disabled 
Make sure we don't invoke the APIs to non-existent endpoints
 2017-04-19 16:57:36 -04:00
Charlton Austin
38d4af0d8b Merge pull request #2479 from charltonaustin/phase_three_config 
feat(data): remove writing of old config
 2017-04-18 10:25:50 -04:00
Antoine Legrand
599ce0de54 code-stye Yapf: 5 files updated 
data/interfaces/appr.py endpoints/appr/cnr_backend.py endpoints/appr/registry.py endpoints/appr/test/test_api.py endpoints/appr/test/test_registry.py
 2017-04-18 14:02:48 +02:00
Antoine Legrand
578f87f94c Fix login with robot to quay-appr 2017-04-18 13:59:21 +02:00
Evan Cordell
2661db7485 Add flag to enable trust per repo (#2541) 
* Add flag to enable trust per repo

* Add api for enabling/disabling trust

* Add new LogEntryKind for changing repo trust settings
Also add tests for repo trust api

* Add `set_trust` method to repository

* Expose new logkind to UI

* Fix registry tests

* Rebase migrations and regen test.db

* Raise downstreamissue if trust metadata can't be removed

* Refactor change_repo_trust

* Add show_if to change_repo_trust endpoint
 2017-04-15 08:26:33 -04:00
Joseph Schorr
626f306283 Add tests for list_active_tags 2017-04-14 17:53:31 -04:00
Joseph Schorr
4a1fad7520 Make sure to filter hidden tags from the active tags query 2017-04-13 19:29:38 -04:00
Joseph Schorr
ab2f044331 Switch get repo API to use a single list tags query 
Should make things faster since the join occurs on the database side
 2017-04-13 18:06:58 -04:00
Charlton Austin
40906afdd8 feat(data): remove writing of old config 
### Description of Changes
  Phase three of config data model change.
 2017-04-13 13:25:36 -04:00
Joseph Schorr
68331859b0 Add backfill for repository search score table 2017-04-13 12:30:44 -04:00
josephschorr
c78ec89305 Merge pull request #2531 from coreos-inc/repo-score 
Switch to using RepositorySearchScore table for search ranking
 2017-04-13 11:51:59 -04:00
Evan Cordell
ec63e495fc Add repo purge callbacks and register TUF metadata deletion as one 2017-04-12 17:33:51 -04:00
Joseph Schorr
80693d6b8c Fix NPE bug in RAC worker 
We need to return `None`, not `0` if there are no additional repositories to measure
 2017-04-11 15:42:11 -04:00
Joseph Schorr
e5a009a777 Switch to using RepositorySearchScore table for search ranking 
Should make search queries much, much faster as it contains the denormalized RAC data
 2017-04-11 14:55:20 -04:00
josephschorr
928b9915ed Merge pull request #2441 from coreos-inc/repo-score-denormalization 
Add a RepositorySearchScore table and calculation to the RAC worker
 2017-04-10 16:31:09 -04:00
Joseph Schorr
df3f47c79a Add a RepositorySearchScore table and calculation to the RAC worker 
This will be used in a followup PR to order search results instead of the RAC join. Currently, the join with the RAC table in search results in a lookup of ~600K rows, which causes searching to take ~6s. This PR denormalizes the data we need, as well as allowing us to score based on a wider band (6 months vs the current 1 week).
 2017-04-10 14:29:02 -04:00
Erica
3f79422a52 Merge pull request #2306 from coreos-inc/QUAY-2842-audit-log-strict-config-option 
feat(config.py): add setting for audit log strictness
 2017-04-07 13:43:11 -04:00
EvB
20c4d971c4 refactor(model/log): pull allowed action types into constant 2017-04-07 11:39:54 -04:00
Jake Moshenko
a8ec7865a7 Merge pull request #2511 from jakedt/fixwarnings 
Fixwarnings
 2017-04-06 16:12:19 -04:00
Jake Moshenko
c7241911a5 Fix old-style flask imports to silence deprecation warnings. 2017-04-06 13:15:48 -04:00
Joseph Schorr
f9e6110f73 Add basic user interface for application repos 
Adds support for creating app repos, viewing app repos and seeing the list of app repos in the Quay UI.
 2017-04-05 11:30:09 -04:00
EvB
d0aaaaa1ef test(data/model/log): add more log_action tests 
temp
 2017-04-05 11:26:10 -04:00
EvB
ea740d15ba fix(model/log): log exception on swallowed db errors 2017-04-05 11:26:10 -04:00
EvB
625bd66b42 test(data/model): test action logging 2017-04-05 11:26:10 -04:00
EvB
6916d82e0d feat(endpoints/trackhelper): wrap log op for silent fails 2017-04-05 11:26:10 -04:00
Joseph Schorr
b10608e277 Change teamsync config to be a UTF8 field 2017-04-03 14:16:34 -04:00
Joseph Schorr
bdd07d4f39 Fix flakiness in team sync tests 2017-04-03 11:36:42 -04:00
Joseph Schorr
bd22fb255e Rename get_federated_user to get_and_link_federated_user_info 
Better to be explicit wherever possible
 2017-04-03 11:36:42 -04:00
Joseph Schorr
1a31d98c44 Clarify variable name in Keystone auth 2017-04-03 11:36:41 -04:00
Joseph Schorr
8c07f733eb Add pagination tests for LDAP 2017-04-03 11:36:41 -04:00
Joseph Schorr
541aa722c2 Add sleeps to make test non-flaky 
Sucks, but MySQL only has second-level timing, so we need this to be sure
 2017-04-03 11:36:41 -04:00
Joseph Schorr
103186f5e8 Small renames to make team syncing code more clear 2017-04-03 11:36:41 -04:00
Joseph Schorr
7f0aa19292 Code cleanup and style improvements in team sync 2017-04-03 11:36:41 -04:00
Joseph Schorr
84e37b68ee Change if statement to be more readable 2017-04-03 11:31:30 -04:00
Joseph Schorr
71d52d45ba Add a test for same user returned twice in team sync 2017-04-03 11:31:30 -04:00
Joseph Schorr
d7825c6720 Add group iteration and syncing support to Keystone auth 2017-04-03 11:31:30 -04:00
Joseph Schorr
47278cc559 Cleanup test fixtures 2017-04-03 11:31:30 -04:00
Joseph Schorr
df603462b8 Add database migration for TeamSync 2017-04-03 11:31:29 -04:00
Joseph Schorr
96b9d6b0cd Add end-to-end test for team sync 2017-04-03 11:31:29 -04:00
Joseph Schorr
4055158fc4 Fix indentation 2017-04-03 11:31:29 -04:00
Joseph Schorr
938730c076 Move sync team into its own module and add tests 2017-04-03 11:31:29 -04:00
Joseph Schorr
eeadeb9383 Initial interfaces and support for team syncing worker 2017-04-03 11:31:29 -04:00
Joseph Schorr
94b07e6de9 Allow nulls in last_updated field to accurately report the last updated time to users for newly sync teams 2017-04-03 11:31:29 -04:00
Joseph Schorr
8ea3977140 Add ability to enable, disable and view team syncing in UI and API 
Also extracts out some common testing infrastructure to make testing APIs easier now using pytest
 2017-04-03 11:31:29 -04:00
Joseph Schorr
bb20422260 Fix pagination disabling in LDAP with mockldap 
Since mockldap doesn't support pagination, just disable it globally
 2017-04-03 11:31:28 -04:00
Joseph Schorr
ecfac81721 Add check_group_lookup_args and service_metadata to auth providers 2017-04-03 11:31:28 -04:00
Joseph Schorr
1cfc4a8341 Change max size of LDAP pages and add filtering to reduce attributes returned 2017-04-03 11:31:28 -04:00
Joseph Schorr
f5a854c189 Add TeamSync database and API support 
Teams can now have a TeamSync entry in the database, indicating how they are synced via an external group. If found, then the user membership of the team cannot be changed via the API.
 2017-04-03 11:31:28 -04:00
Joseph Schorr
d718829f5d Initial LDAP group member iteration support 
Add interface for group member iteration on internal auth providers and implement support in the LDAP interface.
 2017-04-03 11:31:28 -04:00
Charlton Austin
9ff189b16e fix(migration merge issue): missing .save() on migration 2017-03-28 15:17:51 -04:00
Charlton Austin
d559dc7b3e Fixing the migration path so we don't have incorrect branches. 2017-03-28 14:54:21 -04:00
Charlton Austin
ca99535774 Merge pull request #2449 from charltonaustin/phase_two_config 
feat(build runner): added in context, dockerfile_location
 2017-03-28 14:14:36 -04:00
Charlton Austin
e6d201e0b0 feat(build runner): added in context, dockerfile_location 
this is a new feature meant to allow people to use any file as
  a dockerfile and any folder as a context directory
 2017-03-28 13:55:31 -04:00
Antoine Legrand
d2ed37e158 Fix force push causing duplicated entries 2017-03-27 15:39:57 +02:00
Antoine Legrand
22c1a29892 fix strip_sha256 2017-03-24 19:49:52 +01:00
Antoine Legrand
bbd74eabd1 Allow force push for app 2017-03-23 22:50:07 +01:00
Joseph Schorr
ac4a79ae01 Update PR for rebase 2017-03-23 15:57:49 -04:00
Joseph Schorr
651666b60b Refactor our auth handling code to be cleaner 
Breaks out the validation code from the auth context modification calls, makes decorators easier to define and adds testing for each individual piece. Will be the basis of better error messaging in the following change.
 2017-03-23 15:42:45 -04:00
Joseph Schorr
dd9e4bf3e7 Remove transaction around OCI blobs 
Fixes https://www.pivotaltracker.com/story/show/142341399
 2017-03-23 14:51:37 -04:00
Jimmy Zelinskie
9c0cbbf57c data.oci_model: sloppily rewrite digest format 
We expect digests to be in the form 'sha256:digest'
 2017-03-23 12:37:32 -04:00
Joseph Schorr
7d66f30d52 Fix filtering of repositories in search 2017-03-23 11:35:17 -04:00
Joseph Schorr
917d5e2550 Fix typos in data model 2017-03-23 11:14:08 -04:00
Joseph Schorr
05ce571e3e Add missing return statement 2017-03-23 11:11:21 -04:00
Jimmy Zelinskie
d20ff785e6 data.model.repository: add back search fields 2017-03-23 10:46:04 -04:00
Jimmy Zelinskie
2bdd3d4fa1 data.oci_model.tag: add missing import 2017-03-23 00:58:57 -04:00
Jimmy Zelinskie
e872c310d0 data.oci_model: fix imports 2017-03-23 00:21:21 -04:00
Joseph Schorr
bdda74d6df Make sure GC checks new Blob table as well before deleting CAS storage 2017-03-22 23:53:21 -04:00
Jimmy Zelinskie
ddad957a56 data.model.repository: add app methods 2017-03-22 21:51:55 -04:00
Jimmy Zelinskie
650723430b data.interfaces.appr: init 2017-03-22 21:51:41 -04:00
Jimmy Zelinskie
9f684fa73f data.oci_model: init with app methods 2017-03-22 21:51:28 -04:00
Jimmy Zelinskie
3ccf3c5f33 Merge pull request #2447 from jzelinskie/cnr-step2 
CNR Step 2
 2017-03-22 18:45:51 -04:00
Joseph Schorr
df1e7f90e0 Add verb security tests and fix small issues 2017-03-22 18:29:53 -04:00
Jimmy Zelinskie
d5fa2ad0c0 endpoints.verbs: abort 405 for non-container repos 2017-03-22 17:50:58 -04:00
Jimmy Zelinskie
40b638a981 data.migrations: rebase to HEAD of migration tree 2017-03-22 17:26:59 -04:00
Jimmy Zelinskie
48ba59d615 endpoints.v2: only work on docker repositories 2017-03-22 17:26:59 -04:00
Jimmy Zelinskie
72751592a3 data.interfaces.v1: document types 2017-03-22 15:51:19 -04:00
Jimmy Zelinskie
45f14f220d data.model.repository: optimize by using kind_id 2017-03-22 15:51:19 -04:00
Joseph Schorr
30b532254c Disallow non-apps-supported APIs for application repositories 2017-03-22 15:51:19 -04:00
Joseph Schorr
c3402fff5a Add test to ensure we cannot create repos with the same name but different kinds 2017-03-22 14:34:32 -04:00
Jimmy Zelinskie
a2bac7dabd endpoints.v1: only work on docker repositories 2017-03-22 14:31:22 -04:00
Jimmy Zelinskie
f086eea754 data.interfaces.v1: explicitly use kwargs 
This fixes the function from accidentally using the wrong arguments
positionally for the new `repo_kind` kwarg.
 2017-03-22 13:58:50 -04:00
Jimmy Zelinskie
d2a4c9d05a data.model.repository: audited for repo_kind usage 2017-03-22 13:58:50 -04:00
Jimmy Zelinskie
074c1bc4a8 data.model._basequery: audited for repo_kind usage 2017-03-22 13:58:49 -04:00
Jimmy Zelinskie
f842bc3a82 data.migrations.migration.sh: wait 25s for mysql 
Without this, there are frequent race conditions wheres the client fails
to connect to the server when using Docker For Mac.
 2017-03-21 15:38:39 -04:00
Jimmy Zelinskie
4492d2f210 data.migrations: add repository kind 2017-03-21 15:38:38 -04:00
Jimmy Zelinskie
5b362da8ac data.database: add RepositoryKind 2017-03-21 15:38:38 -04:00
Joseph Schorr
ff7f78e990 Have blob uploads be checked against configurable max layer size 2017-03-21 13:16:55 -04:00
Joseph Schorr
76de324ca8 Change blob upload ints into bigints 2017-03-21 13:14:11 -04:00
Charlton Austin
3502d9f61c Merge pull request #2438 from charltonaustin/phase_one_config 
refactor(data): add in new config for builder
 2017-03-21 10:16:51 -04:00
Jimmy Zelinskie
6a538647e4 data.database: beta classes skip transitive delete 2017-03-20 18:41:39 -04:00
josephschorr
27aa12de7a Merge pull request #2439 from coreos-inc/remove_redis_log_expiration 
Switch from expire to delete redis log_entries
 2017-03-20 13:41:57 -04:00
Charlton Austin
f701677a8e refactor(data): add in new config for builder 
we are doing phase one of the four phase migration on the builder config
 2017-03-20 13:03:41 -04:00
Jimmy Zelinskie
0ea600628b Merge pull request #2436 from jzelinskie/cnr-step1 
CNR - Step 1
 2017-03-17 15:37:29 -04:00
Jimmy Zelinskie
ad029fb331 data.migrations: don't use UTF-8 for unique fields 
Unique indexes must have less than 767 bytes and UTF-8 encoding with 255
chars is beyond this maximum. Since this is an internal identifier, we
can be confident that we will not require UTF-8 for it in the future.
 2017-03-17 15:21:24 -04:00
Jimmy Zelinskie
c915a40531 data.database: rm tag_kind from Tag indexes 
These shouldn't be necessary.
 2017-03-17 11:35:16 -04:00
Jimmy Zelinskie
0e32e77e99 data.database: document all CNR/OCI models 2017-03-17 11:35:16 -04:00
Jimmy Zelinskie
2a117f2d24 data.migrations: change CNR mimetypes to v0 
Our initial CNR support is of a pre-v1 implementation of the
specification.
 2017-03-17 11:33:16 -04:00
Jimmy Zelinskie
1e9ce85af6 data.database/migrations: remove repo_id from db 
This also manually organizes and removes broken parts of the migration.
 2017-03-17 11:33:16 -04:00
Antoine Legrand
8f323154ce data.migrations: add OCI/CNR models 2017-03-17 11:33:16 -04:00
Antoine Legrand
c61024586d data.database: add CNR/OCI models 2017-03-17 11:33:16 -04:00
Antoine Legrand
718aeeead8 Fix search group_by clause for PG 2017-03-17 16:30:24 +01:00
Antoine Legrand
ec847ce613 Switch from expire to delete redis log_entries 2017-03-17 15:35:47 +01:00
Joseph Schorr
e25c989fef Add a cleanup worker for blob uploads 2017-03-16 13:36:59 -04:00
Joseph Schorr
e90cab4d77 Change revert tag into restore tag and add manifest support 2017-03-14 11:34:42 -04:00
Joseph Schorr
af743b156b Show manifest digests in place of V1 ids in the tag view when possible 2017-03-14 11:34:41 -04:00
Jimmy Zelinskie
123d003d4e Merge pull request #2424 from jzelinskie/qss-image 
workers.securityworker: revert to image querying
 2017-03-10 17:38:02 -05:00
Jimmy Zelinskie
a780136337 workers.securityworker: revert to image querying 2017-03-10 17:37:40 -05:00
josephschorr
cbac673d58 Merge pull request #2404 from coreos-inc/cas-gc-fix 
Fix GC handling around CAS paths
 2017-03-10 17:34:21 -05:00
josephschorr
432b2d3fe8 Merge pull request #2392 from coreos-inc/search-optimization 
Optimize repository search by changing our lookup strategy
 2017-03-10 15:44:26 -05:00
Jimmy Zelinskie
53eb579459 data.model.tag: find min *alive* tag 2017-03-10 13:15:35 -05:00
Joseph Schorr
b5bb76cdea Optimize repository search by changing our lookup strategy 
Previous to this change, repositories were looked up unfiltered in six different queries, and then filtered using the permissions model, which issued a query per repository found, making search incredibly slow. Instead, we now lookup a chunk of repositories unfiltered and then filter them via a single query to the database. By layering the filtering on top of the lookup, each as queries, we can minimize the number of queries necessary, without (at the same time) using a super expensive join.

Other changes:
- Remove the 5 page pre-lookup on V1 search and simply return that there is one more page available, until there isn't. While technically not correct, it is much more efficient, and no one should be using pagination with V1 search anyway.
- Remove the lookup for repos without entries in the RAC table. Instead, we now add a new RAC entry when the repository is created for *the day before*, with count 0, so that it is immediately searchable
- Remove lookup of results with a matching namespace; these aren't very relevant anyway, and it overly complicates sorting
 2017-03-09 19:47:55 -05:00
Joseph Schorr
62312e6461 Add warning when CAS paths are skipped and ensure we are under a transaction 2017-03-08 17:01:07 -05:00
Joseph Schorr
69e550d125 Fix GC handling around CAS paths 
Adds code to ensure we never GC CAS paths that are shared amongst multiple ImageStorage rows, as well as an associated pair of tests to catch the positive and negative cases.
 2017-03-07 13:48:07 -05:00
Jimmy Zelinskie
40636d4103 find work based on tag IDs rather than image IDs 2017-03-06 17:09:57 -05:00
Jimmy Zelinskie
2cead05f53 data.model.tag: filter hidden for scan eligibility 2017-03-06 15:44:01 -05:00
Jimmy Zelinskie
904b902295 workers.securityworker: find eligible tag images 2017-03-06 14:37:34 -05:00
Jimmy Zelinskie
b9ac2b7b3b workers.securityworker: simplify min id 2017-03-03 14:51:18 -05:00
Jimmy Zelinskie
4ed0cdda14 securityscanner: add a min image id option 
This will enable us to force some instances of the securityworker to
scan only new images.
 2017-03-03 13:55:25 -05:00
Joseph Schorr
8e863b8cf5 Implement new create and manager trigger UI 
Implements the new trigger setup user interface, which is now a linear workflow found on its own page, rather than a tiny modal dialog

Fixes #1187
 2017-02-28 16:51:42 -05:00
Joseph Schorr
8ec6221ca2 Fix health check 2017-02-24 12:23:18 -05:00
Joseph Schorr
c0f7530b29 Pull out JWT auth validation into validator class 
Also fixes a small bug in validation (yay tests!)
 2017-02-24 12:23:16 -05:00
josephschorr
f7a7d30ec2 Merge pull request #2366 from coreos-inc/alert-spam-fixes 
Small fixes for alert spam
 2017-02-22 14:18:18 -05:00
Joseph Schorr
478b1642b2 Eat AttributeError in peewee close database call 
Fixes https://sentry.io/coreos/backend-production/issues/104257892/
 2017-02-22 13:21:12 -05:00
Joseph Schorr
d29d2da1ca Handle IntegrityError in tag update code 
Fixes https://sentry.io/coreos/backend-production/issues/173470565/events/4938537230/
 2017-02-22 13:20:04 -05:00