vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9200 commits 2 branches 62 tags 205 MiB
Commit graph

1690 commits

Author SHA1 Message Date
Joseph Schorr
785c74de52 Fix attempts to confirm team invite for mismatched email address 
Currently, if a user tries to confirm an invite sent to them on an account with a mismatching email address, we simply redirect to the org (where they get a 403). This change ensures they get the proper error response message, and restyles the error page to be nicer.

Fixes #2227
Fixes https://www.pivotaltracker.com/story/show/136088507
 2016-12-15 17:15:11 -05:00
Joseph Schorr
624b2a8385 Have security scanner analyze only send notifications for *new* layers 
Following this change, anytime a layer is indexed by the security scanner, we only send notifications out if the layer previously had a security_indexed_engine value of `-1`, thus ensuring it has *never* been indexed previously. This will allow us to change to version of the security scanner upwards, and have all the images be re-indexed, without firing off notifications in a spammy manner.
 2016-12-13 23:17:11 -05:00
Evan Cordell
5686c80af1 Revert "Add GC of layers in Clair" 
This reverts 49872838ab
 2016-12-13 18:40:58 -05:00
Charlton Austin
0b8c2ef92f Removing an unused import. 2016-12-08 13:53:52 -05:00
josephschorr
410b9d74fc Merge pull request #2214 from coreos-inc/clair-gc 
Add GC of layers in Clair
 2016-12-07 17:58:21 -05:00
josephschorr
543d86ae10 Merge pull request #2221 from coreos-inc/fix-error-pages 
Have all error pages be rendered by Angular
 2016-12-07 17:53:14 -05:00
josephschorr
111b7b0788 Merge pull request #2206 from coreos-inc/ldap-user-search-fix 
Fix external auth returns for query_user calls
 2016-12-07 17:53:04 -05:00
josephschorr
c5111d7930 Merge pull request #2144 from coreos-inc/buildlogs-improvements 
Change the append build log method to execute the two calls via one pipelined connection
 2016-12-07 17:52:22 -05:00
Joseph Schorr
c06bba38de Have all error pages be rendered by Angular 
Fixes #2198

Fixes https://www.pivotaltracker.com/story/show/135724483
 2016-12-07 17:49:02 -05:00
Jimmy Zelinskie
00eafff747 Merge pull request #2204 from jzelinskie/429builds 
add rate limiting to build queues
 2016-12-07 15:03:31 -05:00
Joseph Schorr
3203fd6de1 Fix external auth returns for query_user calls 
Adds the missing field on the query_user calls, updates the external auth tests to ensure it is returned properly, and adds new end-to-end tests which call the external auth engines via the *API*, to ensure this doesn't break again
 2016-12-07 14:28:42 -05:00
Jimmy Zelinskie
ebbe58d311 replace prefix w/ canonical name list 2016-12-07 12:56:56 -05:00
Jimmy Zelinskie
c41de8ded6 build queue rate limiting: address PR comments 2016-12-06 20:40:54 -05:00
Joseph Schorr
49872838ab Add GC of layers in Clair 
Fixes https://www.pivotaltracker.com/story/show/135583207
 2016-12-06 19:52:56 -05:00
Jimmy Zelinskie
eb69abff8b build rate limiting: tests 2016-12-06 16:30:12 -05:00
Jimmy Zelinskie
57770493fa build rate limiting: use a rate 2016-12-06 16:30:12 -05:00
Jimmy Zelinskie
7877c6ab94 add rate limiting to build queues 2016-12-06 16:30:12 -05:00
Charlton Austin
0aa6e6cd58 Merge pull request #2203 from charltonaustin/fix_build_component_cleanup 
Adding in a cancel method to the build component so we can properly c…
 2016-12-06 14:13:10 -05:00
Jake Moshenko
21e3001446 Add a bulk insert for queue and notifications. 
Use it for Clair spawned notifications.
 2016-12-06 14:00:16 -05:00
Charlton Austin
c6be12e31e Adding in a cancel method to the build component so we can properly clean up the job task. 2016-12-06 13:37:49 -05:00
Jimmy Zelinskie
3a7119d499 Merge pull request #2209 from coreos-inc/clair-notification-read 
Clair notification read and queue fixes
 2016-12-05 19:36:59 -05:00
Joseph Schorr
97d150e281 Have QSS only add security scanner notifications once 2016-12-05 19:08:20 -05:00
Jake Moshenko
7c490b46c8 Only save dirty fields on Queue queries. 2016-12-05 18:12:14 -05:00
Charlton Austin
0a6322015c Fix the queue item delete. 2016-12-02 15:30:35 -05:00
Charlton Austin
7b3d8e3977 Merge pull request #2183 from charltonaustin/metrics_for_unscanned_images 
Adding in some metrics around clair sec scan.
 2016-12-02 11:50:29 -05:00
Charlton Austin
edd9dcd7f6 Adding in some metrics around clair sec scan. 2016-12-01 16:50:02 -05:00
Charlton Austin
1f03fcb146 Adding in notification type for notification kind. 2016-12-01 12:26:18 -05:00
Charlton Austin
2c637fe5ce Merge pull request #2173 from charltonaustin/adding_in_build_cancel_notifications 
Adding in cancel notifications
 2016-11-30 15:03:17 -05:00
Charlton Austin
4103a0b75f Adding in cancel notifications 2016-11-30 14:38:34 -05:00
Joseph Schorr
730a220eb0 Fix user lookup query under Postgres 
Adds a missing group_by clause
 2016-11-29 11:36:53 -05:00
Joseph Schorr
402ad25690 Change team invitation acceptance to join all invited teams under the org 
Fixes #1989
 2016-11-28 18:39:28 -05:00
Joseph Schorr
e29cb34336 Fix Set calls to gauges 
Fixes #2150

The proper function is `Set` (not `set`), which was causing these gauges to not report to Prometheus
 2016-11-21 15:27:17 -05:00
Charlton Austin
2fe74e4057 Adding in UI for cancel anytime. 2016-11-21 10:58:32 -05:00
Joseph Schorr
1b8820f2e7 Change the append build log method to execute the two calls via one pipelined connection 
Should reduce the amount of packets used by the build manager

Reference: https://github.com/andymccurdy/redis-py#pipelines
 2016-11-18 11:47:16 -05:00
Charlton Austin
fd7c566d31 Adding in cancel for a build that is building. 2016-11-16 17:40:24 -05:00
Joseph Schorr
1a61ef4e04 Report the user's name and company to Marketo 
Also fixes the API to report the other changes (username and email) as well
 2016-11-14 17:34:50 -05:00
Jake Moshenko
10255d4052 Merge pull request #2109 from jakedt/collapsemigrations 
Collapse all migrations prior to 2.0.0 into one.
 2016-11-10 17:35:07 -05:00
Jake Moshenko
b5834a8a66 Collapse all migrations prior to 2.0.0 into one. 2016-11-10 17:31:00 -05:00
Joseph Schorr
536809a992 Change LDAP errors into debug statements to reduce log clutter 
Fixes #2083
 2016-11-10 16:39:26 -05:00
Joseph Schorr
0f2eb61f4a Add collection of user metadata: name and company 2016-11-08 16:15:02 -05:00
josephschorr
233b2be5c2 Merge pull request #2066 from coreos-inc/select-username 
Add support for temp usernames and an interstitial to confirm username
 2016-11-03 16:22:16 -04:00
Joseph Schorr
1e3b354201 Add support for temp usernames and an interstitial to confirm username 
When a user now logs in for the first time for any external auth (LDAP, JWT, Keystone, Github, Google, Dex), they will be presented with a confirmation screen that affords them the opportunity to change their Quay-assigned username.

Addresses most of the user issues around #74
 2016-11-03 15:59:14 -04:00
Joseph Schorr
3fd92aef35 Fix entity search API to not IndexError 2016-11-02 16:22:35 -04:00
Joseph Schorr
d7f56350a4 Make email addresses optional in external auth if email feature is turned off 
Before this change, external auth such as Keystone would fail if a user without an email address tried to login, even if the email feature was disabled.
 2016-10-31 13:50:24 -04:00
josephschorr
934cdecbd6 Merge pull request #1905 from coreos-inc/external-auth-search 
Add support for entity search against external auth users not yet linked
 2016-10-27 16:06:42 -04:00
Joseph Schorr
b3d1d7227c Add support to Keystone Auth for external user linking 
Also adds Keystone V3 support
 2016-10-27 15:42:03 -04:00
Joseph Schorr
fbb524e34e Add support to ExternalJWT Auth for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
f9ee8d2bef Add support to LDAP for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
d145222812 Add support for linking to external users in entity search 2016-10-27 15:42:03 -04:00
Charlton Austin
2147005d2c Adding a method of cancelling a build based on etcd message. 2016-10-25 12:50:58 -04:00
Charlton Austin
dc35769396 Merge pull request #2022 from charltonaustin/refactor_for_cancel_anytime 
Making some refactors to make it easier to cancel the build at any time.
 2016-10-24 16:17:55 -04:00
Charlton Austin
1cde22e76c Making some refactors to make it easier to cancel the build at any time. 2016-10-24 15:59:33 -04:00
josephschorr
edc2bc8b93 Merge pull request #1698 from coreos-inc/delete-namespace 
Add support for deleting namespaces (users, organizations)
 2016-10-21 16:54:52 -04:00
Joseph Schorr
73eb66eac5 Add support for deleting namespaces (users, organizations) 
Fixes #102
Fixes #105
 2016-10-21 15:41:09 -04:00
josephschorr
4d89052bbf Merge pull request #1764 from coreos-inc/db-timeout 
Add a default database connect timeout
 2016-10-20 15:16:53 -04:00
Joseph Schorr
b7fc7999c3 Delete old "license" checking code arounds user counts 
This is legacy code that doesn't actually do anything of value
 2016-10-20 14:58:35 -04:00
Jimmy Zelinskie
20ef43d5fb workers.queuecleanup: remove direct peewee usage 2016-10-20 13:46:00 -04:00
Joseph Schorr
715fc27474 Add a default database connect timeout 
Fixes #1760
 2016-10-17 13:33:30 -04:00
Charlton Austin
97d644d95d Adding in the delete api and the delete and create UI. 2016-10-13 10:40:52 -04:00
charltonaustin
5a4b702888 Adding in security tests and docs. 2016-10-11 09:30:37 -04:00
josephschorr
7fc33a9a57 Merge pull request #1965 from coreos-inc/condense-slack-notifications 
Less verbose notifications for QSS
 2016-10-10 15:38:12 -04:00
Joseph Schorr
ebf4120326 Less verbose notifications for QSS 
Fixes #1914
 2016-10-10 15:18:49 -04:00
charltonaustin
14eb3005b6 Some fixes for code review. 2016-10-10 12:55:00 -04:00
charltonaustin
4ae6e6efa9 Fixing some database integration errors 2016-10-10 10:51:30 -04:00
charltonaustin
1e733ddffb Adding in a new message data model and the corresponding methods to in the API. 2016-10-07 15:56:58 -04:00
Joseph Schorr
0b7bb6d6c6 Fix issue in V1 registry code with accessing locations under HEAD 
Fixes #1922
 2016-10-03 17:09:12 +03:00
josephschorr
b4dd5ea4dd Merge pull request #1867 from coreos-inc/keystone-timeout 
Add configurable timeout and debug flags to Keystone users
 2016-09-29 23:01:02 +02:00
Joseph Schorr
02b8afe127 Add labeling of built manifests with their build IDs 
Also sends the digests to the notification

Fixes #593
 2016-09-29 10:58:45 +02:00
Jimmy Zelinskie
44eca10c05 update interfaces to use ABC 2016-09-26 14:50:24 -04:00
Jimmy Zelinskie
a1a930b833 database: fix indices post-rebase 2016-09-26 14:49:58 -04:00
Jimmy Zelinskie
ca883e5662 port label support to refactored v2 registry 2016-09-26 14:49:58 -04:00
Joseph Schorr
3c8b87e086 Fix verbs in manifestlist 
All registry_tests now pass
 2016-09-26 14:49:58 -04:00
Jimmy Zelinskie
783c9e7a73 stop exporting experimental database models 2016-09-26 14:49:23 -04:00
Jimmy Zelinskie
c35413d4f6 add boilerplate for verbs data interface 2016-09-26 14:49:23 -04:00
Jimmy Zelinskie
2e5a94bc0b create key server data interface 2016-09-26 14:49:23 -04:00
Jimmy Zelinskie
c06d395f96 create interfaces for v1 and v2 data model 2016-09-26 14:49:23 -04:00
Joseph Schorr
b775458d4b lifetimes on Tags should now be in milliseconds 
Fixes #1779
 2016-09-26 14:49:04 -04:00
Joseph Schorr
db60df827d Implement V2 interfaces and remaining V1 interfaces 
Also adds some tests to registry tests for V1 stuff.
Note: All *registry* tests currently pass, but as verbs are not yet converted, the verb tests in registry_tests.py currently fail.
 2016-09-26 14:49:04 -04:00
Jimmy Zelinskie
d67991987b v1: refactor index 2016-09-26 14:48:42 -04:00
Jimmy Zelinskie
b68e1b5efc add "get_" prefix to all db read funcs 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
32a6c22b43 mv data/types image 
This change also merges formats into the new image module.
 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
a516c08deb v2: refactor auth to use data.types 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
3de6000428 v2: refactor blob.py to use data.types 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
e6c99bb471 re-ordered BlobUploading fields 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
3f722f880e v2: add pagination decorator 2016-09-26 14:48:05 -04:00
Jimmy Zelinskie
5b630ebdb0 v2/manifest: refactor to use types 2016-09-26 14:48:05 -04:00
Joseph Schorr
ea18790dfe Get V1 registry code working with new model methods 2016-09-26 14:47:06 -04:00
Joseph Schorr
94d71f2166 Fix model to actually initialize 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
8435c254c3 finish v1 registry refactor 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
c14437e54a initial v1 refactor to use model methods 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
9cfd6ec452 database: initial manifestlist schema changes 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
e3a39d7bd6 fix indentation 2016-09-26 14:47:06 -04:00
josephschorr
ad4efba802 Merge pull request #1830 from coreos-inc/superuser-dashboard 
Add prometheus stats to enable better dashboarding
 2016-09-26 17:19:22 +02:00
Joseph Schorr
fd770422bb Add configurable timeout and debug flags to Keystone users 
Fixes #1855
 2016-09-22 18:25:02 -04:00
Joseph Schorr
30af8aef1a Add a worker for reporting global stats to Prometheus 
Fixes #1789
 2016-09-12 16:19:19 -04:00
Jake Moshenko
91963c17a0 Remove a join to slightly optimize the gc query. 2016-09-09 15:40:40 -04:00
Joseph Schorr
3d542b5e93 Handle KeyError nicer in _get_parent_image 
Fixes #1810
 2016-09-09 13:34:56 -04:00
Jake Moshenko
cf83c9a16a Improve the garbage collection tests. 2016-09-07 13:25:19 -04:00
Jake Moshenko
584a5a7ddd Reduce database bandwidth by tracking gc candidate images. 2016-09-07 13:25:19 -04:00
Jake Moshenko
0815f6b6c4 Fix indentation for DB queries. 2016-09-07 10:48:58 -04:00
Jake Moshenko
1d8b72235a Add a helper method to Image to parse ancestor string. 2016-09-07 10:48:58 -04:00
josephschorr
cd8b45e25b Merge pull request #1754 from coreos-inc/team-add-perms 
Better UI and permissions handling for robots and teams
 2016-09-06 17:21:19 -04:00
Joseph Schorr
b4939a3cd0 Fix filtering of repos only visible to org admins 2016-08-31 13:51:53 -04:00
Joseph Schorr
357005e33f Raise a 409 if we try to insert a tag twice at the same time 
Also fixes handling of labels for existing manifests

Fixes #1775
 2016-08-29 16:03:35 -04:00
Joseph Schorr
1a2666be07 Fix deletion of labels and add tests 2016-08-26 16:07:49 -04:00
Joseph Schorr
608ffd9663 Basic labels support 
Adds basic labels support to the registry code (V2), and the API. Note that this does not yet add any UI related support.
 2016-08-26 15:24:26 -04:00
Joseph Schorr
391d70d9ec Add repo permissions dialog for existing teams and robots 
Fixes #1686
 2016-08-22 14:43:12 -04:00
Joseph Schorr
6ebb417923 Redesign the teams page to use a table 
Allows for faster loading and easier viewing of important information about teams
 2016-08-22 14:42:54 -04:00
Jake Moshenko
d6a396be34 Fix all foreign key constraints to use naming convention. 2016-08-18 14:29:53 -04:00
Joseph Schorr
aeddc6af06 Handle GC constraint failures in a nicer way 
Fixes #1739
 2016-08-17 16:13:27 -04:00
josephschorr
2caa82d091 Merge pull request #1713 from coreos-inc/enable-iam 
Enable IAM support for S3 storage
 2016-08-16 16:13:29 -04:00
Joseph Schorr
7f5b536ddb Fix pagination of repositories 
Fixes #1725
 2016-08-15 16:48:04 -04:00
Joseph Schorr
0f46230493 Add an index for lookup by account to log entries 
Also fixes the query to require one less join
 2016-08-12 17:39:31 -04:00
Joseph Schorr
855cc36057 Remove unneeded imports 2016-08-11 17:16:31 -04:00
Joseph Schorr
34d49e2d44 Fix duplicate derived storage cache creation issue 
Fixes #1699
 2016-08-10 16:18:52 -04:00
Joseph Schorr
4a2acac5dc Fix pagination of public repos, make more efficient and add test 2016-08-10 15:08:06 -04:00
Joseph Schorr
bf8f621278 Temporarily remove the migration which drops the foreign keys on LogEntry, as it is invalid 2016-08-08 17:47:04 -04:00
josephschorr
1a137ee7b3 Merge pull request #1643 from coreos-inc/db-retry 
Enable automatic retry for the database
 2016-08-08 15:04:25 -04:00
Joseph Schorr
700e7b74e4 Enable automatic retry for the database 2016-08-08 15:02:42 -04:00
Jimmy Zelinskie
22a25ac2d3 Revert "Merge pull request #1678 from coreos-inc/delete-repo-fix" 
This reverts commit df64caf133, reversing
changes made to 0d1e453566.
 2016-08-08 12:38:15 -04:00
Jimmy Zelinskie
ce14b9dddf modify log_action to internally resolve IDs 2016-08-08 12:38:15 -04:00
Jimmy Zelinskie
052c31752b MIGRATION: drop foreign keys on logentry table 
This migration generates the following for MySQL:

BEGIN;

-- Running upgrade 1093d8b212bb -> 6243159408b5

ALTER TABLE logentry DROP FOREIGN KEY fk_logentry_account_id_user;

ALTER TABLE logentry DROP FOREIGN KEY
fk_logentry_repository_id_repository;

ALTER TABLE logentry DROP FOREIGN KEY fk_logentry_performer_id_user;

UPDATE alembic_version SET version_num='6243159408b5' WHERE
alembic_version.version_num = '1093d8b212bb';

COMMIT;
 2016-08-08 12:38:15 -04:00
Jimmy Zelinskie
e05bc8bf7d migration.sh: default DOCKER_IP to localhost 2016-08-08 12:36:01 -04:00
josephschorr
6716a2562b Merge pull request #1680 from coreos-inc/add-missing-index 
Add various missing indexes
 2016-08-08 12:34:58 -04:00
Joseph Schorr
80a37fd295 Add various missing indexes 
Indexes added:

Image::repository - Needed for model.image.get_repository_images_without_placements
RepositoryTag::image - Needed for model.tag.get_tags_for_image
RepositoryTag::repository - Needed for repository deletion
RepositoryBuild::phase - Needed for model.build.list_repository_builds sorting
RepositoryBuild::started - Needed for model.build.list_repository_builds sorting
RepositoryBuild::repository+started+phase - Needed for model.build.list_repository_builds
RepositoryBuild::started+logs_archived+phase - Needed for model.build.get_archivable_build lookup
 2016-08-08 12:34:45 -04:00
josephschorr
df64caf133 Merge pull request #1678 from coreos-inc/delete-repo-fix 
Have repo deletion not lock all the things
 2016-08-04 16:48:03 -04:00
Joseph Schorr
0b5cd95693 Have repo deletion not lock all the things 2016-08-04 16:45:59 -04:00
Joseph Schorr
b1b0da7afd Fix off-by-one error in repo tags pagination 
Fixes #1665
 2016-08-02 14:17:33 -04:00
Jake Moshenko
05e2773fa7 Get rid of remaining slow query for garbage collection. 2016-08-01 18:22:38 -04:00
josephschorr
46a28617e8 Merge pull request #1651 from coreos-inc/fix-branches 
Fix handling of multi-part branches in the build triggers
 2016-07-26 16:00:21 -07:00
Joseph Schorr
9e4f8cac03 Optimize GC query for looking up deletable storages 2016-07-26 13:47:15 -07:00
Joseph Schorr
06d52f2c83 Fix handling of multi-part branches in the build triggers 
Fixes #1360
 2016-07-26 13:41:13 -07:00
Joseph Schorr
5de1e98d3c Fix LDAP DN building for empty RDN list 2016-07-22 14:40:53 -04:00
Joseph Schorr
4d6f96cd6c Add missing pass keyword 2016-07-19 22:24:27 -04:00
Joseph Schorr
b8d2570725 Don't raise an error on duplicate placements 
This can happen if two pushes are racing on the same storage.
 2016-07-19 16:44:05 -04:00
Joseph Schorr
b0b7b63be9 Fix queue tests for MySQL 
MySQL's date time's appear to have a 1 second threshold, so we need to make sure the queue items added for the tests are available as soon as they are added. Before this change, the available_after was set to `datetime.utcnow()`, and, if the `get` was called within 1 second, then its check would fail.
 2016-07-15 13:27:50 -04:00
Joseph Schorr
4e1259b58a Fix the Repository ID in pagination problem once and for all 
But.... ONCE AND FOR ALL!

Note: Tested on SQLite, Postgres and MySQL
 2016-07-14 17:09:52 -04:00
Jimmy Zelinskie
64d0c5b675 data.queue: fix race condition 
It's possible that multiple consumers will acquire a queue item if they
race on an expired item. To mitigate this, we check that the
processing_expires time hasn't been changed since we last read.
 2016-07-14 15:34:22 -04:00
Jimmy Zelinskie
609f4fccd8 data.queue: simplify put method 2016-07-14 15:34:22 -04:00
Joseph Schorr
c1e4bf79b7 Fix delete team error message for admin teams 2016-07-11 15:47:05 -04:00
Joseph Schorr
241ebaa084 Fix typo 2016-07-07 15:06:29 -04:00
Joseph Schorr
adaeeba5d0 Allow for multiple user RDNs in LDAP 
Fixes #1600
 2016-07-07 14:46:38 -04:00
Joseph Schorr
e252ee07cb Fix popularity metrics on list repos API 2016-07-06 16:15:54 -04:00
Joseph Schorr
713ba3abaf Further updates to the Prometheus client code 2016-07-01 14:16:51 -04:00
Jake Moshenko
668a8edc50 Refactor prometheus integration 
Move prometheus to SaaS and make it a plugin
Move static callers to use metrics_queue plugin
Change local-docker to support different quay clone dirnames
Change prom_aggregator to use logrus
 2016-07-01 14:16:50 -04:00
Matt Jibson
3d9acf2fff Use prometheus as a metric backend 
This entails writing a metric aggregation program since each worker has its
own memory, and thus own metrics because of python gunicorn. The python
client is a simple wrapper that makes web requests to it.
 2016-07-01 14:16:50 -04:00
Joseph Schorr
117ccda1cf Fix postgres error in SQL query 2016-07-01 13:04:20 -04:00
Joseph Schorr
1eec6f53b2 Fix SQL error with pagination around Repositories 
Fixes #1591
 2016-06-30 17:31:35 -04:00
Joseph Schorr
310ecd11cc Handle user events Redis not working in tutorial 
Also does some basic restyling

Fixes #1586
 2016-06-28 17:04:31 -04:00
Joseph Schorr
853cca35f3 Change repo stats to use the RAC table and a nice UI 2016-06-22 15:06:53 -04:00
josephschorr
7173d53030 Merge pull request #1549 from coreos-inc/certs 
Switch to install custom LDAP cert by name
 2016-06-21 15:13:44 -04:00
Joseph Schorr
66ec1d81ce Switch to install custom LDAP cert by name 2016-06-21 15:10:26 -04:00
josephschorr
9e6a264f5f Merge pull request #1523 from coreos-inc/verb-tag-cache-fix 
Add a uniqueness hash to derived image storage to break caching over …
 2016-06-20 16:38:25 -04:00
Joseph Schorr
a43b741f1b Add a uniqueness hash to derived image storage to break caching over tags 
This allows converted ACIs and squashed images to be unique based on the specified tag.

Fixes #92
 2016-06-20 16:34:52 -04:00
Joseph Schorr
3b994431eb Auto expire the build status and logs in redis 2016-06-20 13:53:13 -04:00
Joseph Schorr
986d20bcad Switch to generic RedisError 
Fixes #1558
 2016-06-20 11:20:17 -04:00
Jake Moshenko
a1cf12e460 Add a sitemap.txt for popular public repos 
and reference it from the robots.txt
 2016-06-17 14:34:20 -04:00
josephschorr
614b9124ae Merge pull request #1512 from coreos-inc/optimize-queries 
Optimize various queries
 2016-06-16 14:22:59 -04:00
josephschorr
58bef472d9 Merge pull request #1526 from coreos-inc/superuser-grant 
Add ability for super users to take ownership of namespaces
 2016-06-13 16:23:10 -04:00
Joseph Schorr
20816804e5 Add ability for super users to take ownership of namespaces 
Fixes #1395
 2016-06-13 16:22:52 -04:00
josephschorr
bda5d7ae29 Merge pull request #1511 from coreos-inc/location-cache 
Use a cache for ImageStorageLocation
 2016-06-09 14:03:07 -04:00
Joseph Schorr
7aa6b812e2 Use a cache for ImageStorageLocation 
No need to reload it from the DB or join as it is a static set only changed during migration
 2016-06-09 14:02:42 -04:00
Joseph Schorr
8887f09ba8 Use the instance service key for registry JWT signing 2016-06-07 11:58:10 -04:00
Joseph Schorr
894b5fed6f Remove TODO since we always need storage 2016-06-03 13:45:13 -04:00
Joseph Schorr
03fd2ea15a Remove Image from _load_tag_manifests query 
Doesn't appear used or necessary
 2016-06-03 13:44:01 -04:00
Joseph Schorr
9a747ca6a0 Have get_parent_images not join on placements 
The only case that needs the placements is in verbs, for which we use a new method
 2016-06-03 13:33:15 -04:00
Joseph Schorr
8064419715 Remove Image join from get_active_tag 
It isn't used anywhere in the query and appears to be completely unnecessary
 2016-06-03 13:06:57 -04:00
Joseph Schorr
53538f9001 Optimize get_tag_image query 
No caller uses the image placements or locations, so no need to load them.
 2016-06-02 16:36:38 -04:00
josephschorr
cad8746f9d Merge pull request #1502 from coreos-inc/image-replication 
Enable storage replication for V2 and add backfill tool
 2016-06-02 15:02:53 -04:00
Joseph Schorr
12924784ce Enable storage replication for V2 and add backfill tool 
Fixes #1501
 2016-06-02 14:36:08 -04:00
josephschorr
a85c3ebff7 Merge pull request #1457 from coreos-inc/xauth 
Add support for direct granting of OAuth tokens and add tests
 2016-06-01 12:07:12 -04:00
josephschorr
1ddc73416c Merge pull request #1500 from coreos-inc/better-errors 
Better errors
 2016-05-31 15:54:41 -04:00
Jimmy Zelinskie
1f488acf12 data.queue: move name matching clause 2016-05-31 15:44:11 -04:00
Joseph Schorr
04df2410ec Add better errors if Redis is down 
Fixes #1497
 2016-05-31 15:24:36 -04:00
Jimmy Zelinskie
26300d3c8e data.queue: lint 2016-05-27 14:51:19 -04:00
Jimmy Zelinskie
8a5aa65d74 data.queue: limiting before order by rand 2016-05-27 14:44:30 -04:00
Jimmy Zelinskie
44b56ae2cf queue: explicitly declare ordering requirement 
This change defaults the ordering requirement of queue items to be off
and only enables it for the build manager. This should make the queries
for getting queueitems significantly faster for every other use case.
 2016-05-27 14:44:30 -04:00
Joseph Schorr
7933aecf25 Add support for direct granting of OAuth tokens and add tests 
This allows a client (when authorized in a whitelist) to send direct credentials via a Basic auth header and therefore bypass the OAuth approval UI for that user.
 2016-05-23 17:17:06 -04:00
Joseph Schorr
60bbca2185 Fix setup tool when binding to external auth 
We now query the external auth provider for the external service's identifier before adding the linking row into the database. This fixes the case where the external service resolves a different identifier for the same username.

Fixes #1477
 2016-05-23 17:11:36 -04:00
Joseph Schorr
043699cfb3 Always use log entry kind cache 
Fixes #1445
 2016-05-13 15:20:55 -04:00
Jimmy Zelinskie
972e4be811 log: cutoff at the max id past the cutoff_date 
Previously we were using the min, which is always going to be equivalant
to the min id in the table.
 2016-05-10 20:13:10 -07:00
Joseph Schorr
a736407611 Fix user:admin scope handling and add test 2016-05-09 11:16:01 +02:00
josephschorr
f55fd2049f Merge pull request #1433 from coreos-inc/ldapoptions 
Add additional options for LDAP
 2016-05-04 14:06:29 -04:00
Joseph Schorr
42515ed9ec Add additional options for LDAP 
Fixes #1420
 2016-05-04 13:59:20 -04:00
Joseph Schorr
6e2df3b339 Fix key server to not list expired keys 
Fixes the key server to not list expire keys and by default not return expired or unapproved keys unless explicitly requested.

Fixes #1430
 2016-05-03 17:58:47 -04:00
Jimmy Zelinskie
2aa88dcb80 only send notifications when superusers enabled 2016-04-29 15:42:25 -04:00
Jimmy Zelinskie
29e2d7c9d4 data.model.log: remove unused method 2016-04-29 14:22:53 -04:00
Jimmy Zelinskie
e47b29a974 migration: add missing delete from down migration 
This also reorganizes the file a bit.
 2016-04-29 14:10:33 -04:00
Jimmy Zelinskie
4a521f5844 database: revert logentry foreign key proxy 2016-04-29 14:10:33 -04:00
Evan Cordell
489752a0b7 Only refresh current instance service key 2016-04-29 14:10:33 -04:00
Evan Cordell
a6f6a114c2 service key worker to refresh automatic keys 2016-04-29 14:10:33 -04:00
Evan Cordell
2242c6773d Add 'Automatic' ServiceKeyApprovalType 2016-04-29 14:10:33 -04:00
Joseph Schorr
6091db983b Hide expired keys outside of their staleness window 2016-04-29 14:10:33 -04:00
Joseph Schorr
4f63a50a17 Change account-less logs to use a user and not null 
This allows us to skip the migration
 2016-04-29 14:09:37 -04:00
Jimmy Zelinskie
5cb6ba4d12 keyserver migration: fix constraint name 2016-04-29 14:09:37 -04:00
Joseph Schorr
28a80ef6a9 Make sure to verify service names on key creation 2016-04-29 14:09:37 -04:00
Joseph Schorr
522cf68c5d Lots of smaller fixes: 
- Add the rotation_duration to the keys API
- Have the key service UI use the new rotation_duration field
- Fix notification deletion lookup path
- Add proper support for the new notification in the UI
- Only delete expired keys after 7 days (configurable)
- Fix angular digest loop
- Fix unit tests
- Regenerate initdb
 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
370ac3ecd0 service keys: add rotation_duration field 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
6577ac3e62 mv JWK-canonicalization util.security.fingerprint 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
4020ab9f55 service keys: delete notifications by prefix 2016-04-29 14:05:16 -04:00
Joseph Schorr
08017c5111 Further UI updates 2016-04-29 14:05:16 -04:00
Joseph Schorr
a4a01e76c0 Fix up the migration to include the additional changes needed 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
d19eb16b45 keyserver: add generate key function 
The superuser API, initdb, and tests will all need this functionality.
 2016-04-29 14:05:16 -04:00
Joseph Schorr
fb1dca4e94 Add API usage tests 2016-04-29 14:05:16 -04:00
Joseph Schorr
11ff3e9b59 keys ui WIP 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
885a41e6f5 key server: misc fixes to make jwtproxy work 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
5cdc7812dc migration.sh: update to reflect timing 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
50ad1bb6b1 key server: misc cleanup to get it working 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
c79bb14049 service keys: fix stale query 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
86df0124c1 service keys: join with approvals 
Also fixes a bug where we weren't reassigning the query after adding a
WHERE.
 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
dff59b4a39 service key migration 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
c6b8b3ce8c service_keys: s/get_keys/list_keys 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
0ec54fc70e clear notifications on delete/replace service_key 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
42b5196b21 add notification path and use for service keys 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
f406942984 converging on proper rotation 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
aaf9e83278 basically finish superuser key api 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
35ed73e195 rework superuser api 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
4079dba167 service keys: do all the right stuff 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
6ecff950ab service keys: add txs and select4update 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
499bb16306 service key server wip 2016-04-29 13:38:25 -04:00
josephschorr
9e88b1413d Merge pull request #1325 from coreos-inc/blobuncompressedsize 
Fix uncompressed size for blob store and add test
 2016-04-28 13:15:33 -04:00
Jimmy Zelinskie
7239c465bf improve stale cutoff id perf (#1392) 2016-04-20 15:03:06 -04:00
josephschorr
b0cc55276f Merge pull request #1373 from coreos-inc/orgconvert 
Org conversion improvements
 2016-04-19 16:16:35 -04:00
Jimmy Zelinskie
5585e16c90 Merge pull request #1356 from jzelinskie/actionlogarchive 
logrotateworker: save to storage via userfiles
 2016-04-15 13:57:11 -04:00
Jimmy Zelinskie
3d190b786f userfiles: make handler optional 2016-04-15 13:56:07 -04:00
Joseph Schorr
c604dbd0f6 Fix permissions when converting a user to an org 
Fixes #1366
 2016-04-14 17:39:45 -04:00
Joseph Schorr
1009362d26 Have recovery auto-verify the user 
Fixes #1355
 2016-04-08 13:41:16 -04:00
Jake Moshenko
bd5b44cbd2 Move the sequence fixer to a separate tool which can be run 2016-04-01 13:46:13 -04:00
josephschorr
edb157c5cb Merge pull request #1294 from coreos-inc/partialperms 
Change permissions to only load required by default
 2016-03-30 16:40:40 -04:00
Joseph Schorr
42e934d84f Make notification lookup faster and fix repo pagination on Postgres 2016-03-30 14:46:31 -04:00
Joseph Schorr
0dffdb87c9 Fix uncompressed size for blob store and add test 2016-03-29 14:16:56 -04:00
Joseph Schorr
a3aa4592cf Change permissions to only load required by default 
Permissions now load just the namespace and/or repository permissions requested, with a fallback to a full permissions load if necessary.
 2016-03-28 16:33:32 -04:00
Jake Moshenko
eed07722cb Add even larger plans for enterprises on SaaS 2016-03-21 16:38:34 -04:00
Jake Moshenko
fe2cd240bc Revert "Remove old search API which is no longer in use" 2016-03-07 10:07:41 -05:00
Jimmy Zelinskie
b5d904f373 Merge pull request #1218 from jzelinskie/logrotate5ever 
vastly simplify log rotation
 2016-03-04 13:48:21 -05:00
josephschorr
57430a18b4 Merge pull request #1224 from coreos-inc/removeoldsearch 
Remove old search API which is no longer in use
 2016-03-04 12:05:07 -05:00
Joseph Schorr
f498e92d58 Implement against new Clair paginated notification system 2016-02-25 15:58:42 -05:00
Joseph Schorr
c0374d71c9 Refactor the security worker and API calls and add a bunch of tests 2016-02-25 12:29:41 -05:00
Quentin Machu
0183c519f7 Merge pull request #1253 from Quentin-M/clair2 
Adapt securityworker, secscan API and Quay UI for Clair 1.0
 2016-02-19 18:21:25 -05:00
Quentin Machu
e5da33578c Adapt security worker for Clair v1.0 (except notifications) 2016-02-19 17:44:14 -05:00
Joseph Schorr
abd2e3c234 V1 Docker ID <-> V2 layer SHA mismatch fix 
Fix handling of V1 Docker ID <-> V2 layer SHA mismatch by dynamically rewriting the manifest to use new synthesized IDs for all layers above the mismatch. Also adds a bunch of tests for this and other use cases, fixes a bug around manifest digest uniqueness and fixes the 5.5 migration for MySQL.
 2016-02-12 17:39:27 +02:00
josephschorr
a9c64545fa Merge pull request #1228 from coreos-inc/v2storagevalidation 
Add a check that will fail if we try to mislink V1 layers
 2016-02-11 22:49:33 +02:00
Joseph Schorr
27f1cc0a13 Add a check that will fail if we try to mislink V1 layers 
Also logs some useful information
 2016-02-11 22:40:00 +02:00
Jake Moshenko
59a6f5bc77 Replace incompatible MySQL 5.5 server_default 2016-02-11 15:07:16 -05:00
Joseph Schorr
1887dc879c Remove old search API which is no longer in use 2016-02-10 15:02:27 +02:00
Jimmy Zelinskie
ee705fe7a9 vastly simplify log rotation 2016-02-09 18:20:14 -05:00
Joseph Schorr
4e771e667f Change sec scan candidate query to match parents to the expected version only 2016-02-09 22:23:48 +02:00
Joseph Schorr
534ec9cb2b Add pagination to the repository list API to make it better for public 
Fixes #1166
 2016-02-01 22:42:44 +02:00
Joseph Schorr
1536709c02 Small fixes 2016-01-29 20:01:17 +02:00
Joseph Schorr
a80ac8eabb Fix import for alembic 2016-01-29 17:59:23 +02:00
Joseph Schorr
bd0a098282 Add ID-based pagination to logs using new decorators and an encrypted token 
Fixes #599
 2016-01-26 12:50:48 -05:00
Jimmy Zelinskie
e54b86c6eb s/TORRENT/BITTORRENT 2016-01-22 15:52:28 -05:00
Jake Moshenko
fe2bdeb6cb Require some data from all models in initdb 2016-01-19 15:30:27 -05:00
Jake Moshenko
1b392dcb9a Remove dependent signatures before removing image storages 2016-01-19 14:56:02 -05:00
Jake Moshenko
7d0be20842 Formatting and syntax improvements 2016-01-19 14:56:02 -05:00
Joseph Schorr
e4da61a05d Fix piece hash calculation 2016-01-12 17:44:19 -05:00
josephschorr
047c2c2c0f Merge pull request #1129 from coreos-inc/backfill 
Add checksum and torrent info backfill
 2016-01-12 14:20:58 -05:00
Jake Moshenko
96c72e73df Clean up torrents before removing referenced storages 2016-01-12 11:43:07 -05:00
Jake Moshenko
8ab6c8a22d Fix torrent hash generation to work in mixed stacks 2016-01-11 16:43:46 -05:00
Joseph Schorr
c36a7c21c8 Order sadly matters with this check in peewee 2016-01-11 15:10:46 -05:00
Joseph Schorr
bd715c0c71 Add checksum and torrent info backfill 2016-01-08 17:32:30 -05:00
Jake Moshenko
1ae101c917 Address torrent feature review comments. 2016-01-08 16:38:21 -05:00
Jake Moshenko
073b68cf0d Fix torrent migration and update backfill to compute torrent pieces 2016-01-08 11:15:34 -05:00
Jake Moshenko
77aa58996a Fix the db definition for torrentinfo and add migration 2016-01-06 14:04:03 -05:00
Jake Moshenko
fd1e5f2407 Remove an unnecessary outer join 2016-01-05 14:43:40 -05:00
Jake Moshenko
8f80d7064b Hash v1 uploads for torrent chunks 2016-01-05 14:43:40 -05:00
Jake Moshenko
8d5f4466d6 Cleanup some indentation and imports 2016-01-05 12:12:57 -05:00
Jimmy Zelinskie
9b0a84c02f implement get_torrent_info 2016-01-04 16:17:51 -05:00
Jake Moshenko
a9b7ac6b48 Rotate robot user uuid when the credentials change 2016-01-04 16:17:51 -05:00
Jake Moshenko
ce8fcbeaae Update the pieces to use base64 encoded binary 2016-01-04 16:17:51 -05:00
Jake Moshenko
5c6e033d21 Fix indentation 2016-01-04 16:17:51 -05:00
Jake Moshenko
fe87d3c796 Hash and track layer file chunks for torrenting 2016-01-04 16:17:51 -05:00
josephschorr
f748d4348d Merge pull request #1106 from coreos-inc/billingemail 
Add support for custom billing invoice email address
 2016-01-04 14:34:30 -05:00
Joseph Schorr
31a8a0fba4 Better UX when recovering organization emails 
Fixes #291
 2015-12-28 15:25:31 -05:00
Joseph Schorr
10efa96009 Add support for custom billing invoice email address 
Fixes #782
 2015-12-28 13:59:50 -05:00
Joseph Schorr
01723d5546 Catch other cases where the queue item has been removed 
Fixes #1096
 2015-12-22 15:58:51 -05:00
Jake Moshenko
9c1a2e7e1b Improve performance by removing unnecessary group by fields 2015-12-22 11:35:49 -05:00
josephschorr
5ac7369bf5 Merge pull request #1068 from coreos-inc/slowqueryfix 
Remove check for derived image storages on image storage
 2015-12-18 16:32:22 -05:00
Joseph Schorr
94ece129d4 Remove remaining recursive queries on repo delete and add test 2015-12-18 16:04:03 -05:00
josephschorr
16f814b7d9 Merge pull request #1075 from coreos-inc/userdeletefix 
Fix user deletion under MySQL
 2015-12-17 15:09:18 -05:00
Joseph Schorr
2e7835c372 Fix user deletion under MySQL 
Fixes #973
 2015-12-17 15:05:15 -05:00
Jimmy Zelinskie
e1f955a3f6 add a log rotation worker 
Fixes #609.
 2015-12-16 17:22:28 -05:00
Joseph Schorr
f59f4e51e8 Remove check for derived image storages on image storage 
Derived image storages are now 1-to-1 with image storages, so we know they have already been removed at this point

Fixes #1067
 2015-12-16 13:41:25 -05:00
Joseph Schorr
73531d08b5 Add server default for the chunk_count column 2015-12-15 15:44:33 -05:00
Joseph Schorr
141f664bf7 Fix subquery delete which messes up MySQL 
Fixes #1061
 2015-12-15 13:15:10 -05:00
Joseph Schorr
9698d6f6a0 Add created column to blob upload 
Fixes first half of #1054
 2015-12-14 15:27:48 -05:00
Joseph Schorr
54095eb5cb Handle the common case of one chunk when calculating the uncompressed size 
Reference #992
 2015-12-14 15:27:48 -05:00
josephschorr
94effb5aaa Merge pull request #1023 from coreos-inc/getblobopt 
Optimize blob lookup
 2015-12-04 16:11:28 -05:00
Jake Moshenko
38cb63d195 Fix indentation on build model operations 2015-12-04 15:46:07 -05:00
Joseph Schorr
f07b940bc5 Optimize blob lookup 
Fixes #1013
 2015-12-04 14:47:09 -05:00
Joseph Schorr
c324ebd7f6 Only write exceptions for manifest gen when a tag exists 
Fixes #1019

Currently, we just raise an exception to the logs regardless, which can make it appear as if there is an issue (when there isn't).
 2015-12-03 16:04:17 -05:00
Silas Sewell
502e4c04d0 Fix seq migration down_revision 2015-11-30 17:59:04 -05:00
Silas Sewell
3833fb6530 Merge pull request #888 from coreos-inc/remove-hardcoded-ids 
Fix seq generators for enum tables in postgres
 2015-11-30 17:54:13 -05:00
Joseph Schorr
0f7c8105b0 Remove DerivedImageStorage table 2015-11-25 11:46:59 -05:00
Joseph Schorr
762cd56e64 Change derived storage to be based on image 
Fixes #971
 2015-11-24 12:44:07 -05:00
Jake Moshenko
3a29dfc535 Reducing in a tree to avoid recursion depth limits 2015-11-23 15:57:13 -05:00
Joseph Schorr
f4266d08d2 Fix handling of aggregate size in V2 
Fixes #931
 2015-11-20 11:44:03 -05:00
Joseph Schorr
4981ccbc4e Fix issue with query when manifest count is 0 2015-11-19 17:44:16 -05:00
Jake Moshenko
c352050b07 For the last time, you can't delete with a subquery on the same table! 2015-11-19 16:44:27 -05:00
Jake Moshenko
7b53797677 Fix garbage collection when manifests may reference tags 2015-11-19 16:01:36 -05:00
Jake Moshenko
7ae94f414c Alias our subqueries to appease the MySQL beast 2015-11-19 12:58:06 -05:00
Silas Sewell
1162814734 securityworker: mark children we can't analyze 
This allows us to differentiate between images that are queued and those we
can't analyze in constant time.
 2015-11-19 11:22:15 -05:00
Jake Moshenko
e6bd5488c9 Ensure that manifest tags are still alive 2015-11-19 11:01:47 -05:00
Jake Moshenko
b564492ea7 Improve the performance of fetching manifest blobs by checksum. 2015-11-19 11:01:47 -05:00
Quentin Machu
f2d874386b Fix security worker (ok last time before I give up on engineering) 2015-11-18 21:21:00 -05:00
Quentin Machu
04f2688944 Merge pull request #917 from Quentin-M/fix_secwor 
Fix security worker (again?)
 2015-11-18 19:45:36 -05:00
Quentin Machu
88e85cded0 Fix security worker (again?) 2015-11-18 19:45:09 -05:00
Quentin Machu
6d89f259f5 Merge pull request #894 from Quentin-M/fix_secwor 
Refactor security worker
 2015-11-18 14:40:34 -05:00
Quentin Machu
605ed1fc77 Refactor security worker 2015-11-18 14:38:32 -05:00
Jake Moshenko
18b14001b4 Add indices for the security worker fields on Image 
Fixes #906
 2015-11-18 13:29:51 -05:00
Jake Moshenko
206e18d160 Image parents do not have to be nulled transitively on repo delete 2015-11-17 16:48:26 -05:00
Jake Moshenko
e252397292 Switch parent back to a ForeignKeyField without a constraint 2015-11-17 16:09:33 -05:00
Jake Moshenko
3374e8c812 Do not constrain deferred fields in SQLAlchemy bridge 2015-11-17 15:55:18 -05:00
Jake Moshenko
ae61ebeac9 The translate placements query was renamed in v2 2015-11-17 12:24:05 -05:00
Jake Moshenko
7205bf5e7f Merge pull request #885 from jakedt/python-registry-v2 
Python registry v2 mega merge
 2015-11-16 16:15:40 -05:00
Silas Sewell
30b0101584 Fix seq generators for enum tables in postgres 
This attempts to insert a temporary entry into each enum table until it
succeeds. It re-synchronizes the postgres sequence generators with the max id
of the table.

Fixes #883 and #880
 2015-11-16 15:29:51 -05:00
Matt Jibson
13aa6cfcfc No PUT for logarchive 
fixes #862
 2015-11-16 15:01:12 -05:00
Jake Moshenko
0459c3bc54 Merge remote-tracking branch 'upstream/master' into python-registry-v2 2015-11-16 14:22:54 -05:00
Jake Moshenko
52125bbfed Fix gc by using the v1/v2 storage location helper everywhere 2015-11-16 14:13:37 -05:00
Joseph Schorr
819d461ed6 Remove migration re-added by merge accidentally 2015-11-12 22:02:26 -05:00
Joseph Schorr
030c69d7d2 Further merge fixes 2015-11-12 22:00:28 -05:00
Joseph Schorr
7816b0c657 Merge master into vulnerability-tool 2015-11-12 21:52:47 -05:00
Joseph Schorr
25b8b7590f Fix all the things! 2015-11-12 20:55:41 -05:00
Jake Moshenko
44d06b0c2e Fix v1 backward compatibility 2015-11-12 16:22:19 -05:00
Jake Moshenko
cf1ec68046 Correlate a specific blob storage with its placements 2015-11-12 16:20:59 -05:00
Jake Moshenko
ab340e20ea Merge remote-tracking branch 'upstream/master' into python-registry-v2 2015-11-11 16:41:40 -05:00
Jake Moshenko
1d7be74a1f Revert "Drop the v1 checksum column from imagestorage" 
This reverts commit d292a34343.
 2015-11-11 16:39:46 -05:00
Jake Moshenko
a1ccd860e7 Merge pull request #823 from coreos-inc/phase3-11-07-2015 
Phase3 11 07 2015
 2015-11-11 14:22:19 -05:00
Jake Moshenko
1c6919dd93 We must fill in the parent_id on linking 2015-11-10 14:31:46 -05:00
Silas Sewell
e826b14ca4 Merge pull request #725 from coreos-inc/setup-tool-georeplication 
superuser: add storage replication config
 2015-11-09 17:43:38 -05:00
Silas Sewell
5000b1621c superuser: add storage replication config 2015-11-09 17:34:22 -05:00
Joseph Schorr
a69c9e12fd Update quay sec code to fix problems identified in previous review 
- Change get_repository_images_recursive to operate over a single docker image and storage uuid
- Move endpoints/sec to endpoints/secscan
- Change notification system to work with new Quay-sec format

Fixes #768
 2015-11-09 17:14:35 -05:00
Silas Sewell
c739c453da Merge pull request #807 from coreos-inc/storage-preference 
Enable storage preference
 2015-11-09 16:30:47 -05:00
josephschorr
eb2e42dce9 Merge pull request #830 from coreos-inc/fix_parent_id 
Fix deleting repos and images under MySQL
 2015-11-09 14:43:01 -05:00
Joseph Schorr
2d2662f53f Fix deleting repos and images under MySQL 
MySQL doesn't handle constraints at the end of transactions, so deleting images currently fails. This removes the constraint and just leaves parent_id as an int
 2015-11-09 14:42:05 -05:00
Matt Jibson
e5282a216f Merge pull request #818 from mjibson/redis-socket-timeout 
Set timeout for redis commands
 2015-11-09 14:39:00 -05:00
Joseph Schorr
b408cfd2cc Ready for demo 2015-11-09 12:51:05 -05:00
Joseph Schorr
407eaae137 WIP: Towards sec demo 2015-11-09 12:50:39 -05:00
Joseph Schorr
d7ace69fe3 Add a vulnerability_found event for notice when we detect a vuln 
Fixes #637

Note: This PR does *not* actually raise the event; it merely adds support for it
 2015-11-09 12:49:19 -05:00
Quentin Machu
37118423a5 Add support for Quay's vulnerability tool 2015-11-09 12:49:19 -05:00
Quentin Machu
ade664036c Fix migration 57dad559ff2d's header 
Fixes #825
 2015-11-08 15:22:15 -05:00
Jake Moshenko
b526e2a3cd Merge pull request #822 from coreos-inc/phase2-11-07-2015 
Phase2 11 07 2015
 2015-11-08 13:21:59 -05:00
Joseph Schorr
8463514a09 Fix delete_user call to remove all user data 2015-11-08 13:10:01 -05:00
Matt Jibson
afa119d82e Set timeout for redis commands 
fixes #779
 2015-11-06 18:48:47 -05:00
Jake Moshenko
7efa6265bf Merge branch 'newchanges' into python-registry-v2 2015-11-06 18:24:32 -05:00
Jake Moshenko
c2fcf8bead Merge remote-tracking branch 'upstream/phase4-11-07-2015' into python-registry-v2 2015-11-06 18:18:29 -05:00
Jake Moshenko
4314882fa0 Reverse the order of get_parent_images 2015-11-06 17:47:08 -05:00
Jake Moshenko
d292a34343 Drop the v1 checksum column from imagestorage 2015-11-06 16:49:16 -05:00
Jake Moshenko
ad93425ead Stop writing to v1 checksum on ImageStorage 2015-11-06 16:40:04 -05:00
Jake Moshenko
fd3f88f489 Re-enable parent id backfill, use new backfill style 2015-11-06 16:17:12 -05:00
Jake Moshenko
9036ca2f2f Backfill the v1 checksums from imagestorage 2015-11-06 16:17:12 -05:00
Jimmy Zelinskie
d5e7f6bea7 resolve migration branches and run initdb 2015-11-06 16:10:31 -05:00
Jimmy Zelinskie
f3c3e684a1 prepare branch to be merged into phase1-11-07-2015 
This removes the checksum backfill, removes the migration that runs the
backfills, and defaults the security scan feature off.
 2015-11-06 15:22:18 -05:00
Joseph Schorr
0f3db709ea Add a vulnerability_found event for notice when we detect a vuln 
Fixes #637

Note: This PR does *not* actually raise the event; it merely adds support for it
 2015-11-06 15:22:18 -05:00
Quentin Machu
3677947521 Add support for Quay's vulnerability tool 2015-11-06 15:22:18 -05:00
Quentin Machu
a99b8fcfe4 Fix migration 2015-11-06 15:22:18 -05:00
Quentin Machu
f59e35cc81 Add support for Quay's vulnerability tool 2015-11-06 15:22:18 -05:00
Jake Moshenko
3d0bcbaaeb Move v1 checksums to image and track v2 separately 2015-11-06 15:17:55 -05:00
Joseph Schorr
2b3633b107 Remove the used_legacy_github column 2015-11-06 15:17:55 -05:00
Joseph Schorr
bbf4a1fac4 Remove the used_legacy_github column 2015-11-06 15:17:55 -05:00
Silas Sewell
a7fef8377c Enable storage preference 2015-11-06 13:34:49 -05:00
Matt Jibson
57ffb39651 Merge pull request #714 from mjibson/queue-locking 
Refactor queue locking to not use select for update
 2015-11-04 12:05:53 -05:00
Matt Jibson
a994b367da Refactor queue locking to not use select for update 
The test suggests this works.

fixes #622
 2015-11-03 11:32:28 -05:00
Jimmy Zelinskie
c78c450211 UTF-8 v1_json_metadata, comment, manifest 
This will allow us to store unicode JSON blobs in the column on MySQL.
 2015-11-02 15:40:19 -05:00
Jake Moshenko
2c10d28afc Merge remote-tracking branch 'upstream/master' into python-registry-v2 2015-10-26 14:44:16 -04:00
Jake Moshenko
3186311669 Test postgres before mysql variations in migration 2015-10-24 15:00:19 -04:00
Jake Moshenko
b920bf6436 Fix references to mysql in migrations 2015-10-24 15:00:19 -04:00
Jake Moshenko
ddbe33e2ce Switch Text to LongText for MySQL manifests 2015-10-24 15:00:19 -04:00
Jake Moshenko
cb7ec2f239 Backport remaining v2 changes to phase4 2015-10-24 15:00:13 -04:00
Jake Moshenko
e965ed9f3b Remove the unused imagestorage columns from the db. 2015-10-24 14:51:35 -04:00
Jake Moshenko
9da64f3aba Stop writing to deprecated columns for image data. 2015-10-24 14:45:15 -04:00
Jake Moshenko
fee95bc096 Merge remote-tracking branch 'upstream/v2-phase4' into python-registry-v2 2015-10-23 16:47:13 -04:00
Jake Moshenko
5dd377400e Test postgres before mysql variations in migration 2015-10-23 16:26:47 -04:00
Jake Moshenko
a1e92f7150 Fix references to mysql in migrations 2015-10-23 16:23:27 -04:00
Jake Moshenko
4191d69055 Switch Text to LongText for MySQL manifests 2015-10-23 15:55:25 -04:00
Jimmy Zelinskie
e973289397 Revert "Revert "Merge pull request #682 from jzelinskie/revertrevert"" 
This reverts commit 278bc736e3.
 2015-10-23 15:26:33 -04:00
Jake Moshenko
e9722c9468 Backport remaining v2 changes to phase4 2015-10-23 13:49:23 -04:00
Jake Moshenko
e7a6176594 Merge remote-tracking branch 'upstream/v2-phase4' into python-registry-v2 2015-10-22 16:59:28 -04:00
Jimmy Zelinskie
278bc736e3 Revert "Merge pull request #682 from jzelinskie/revertrevert" 
This reverts commit 627ad25c9c, reversing
changes made to 31c392fecc.
 2015-10-22 16:02:07 -04:00
Jimmy Zelinskie
d568697034 remove migration to get fixes into prod 2015-10-22 15:30:49 -04:00
Jimmy Zelinskie
a34ddc1f71 copy over v1 metadata when linking existing image 2015-10-22 15:21:15 -04:00
Joseph Schorr
c518874ded I hate Redis! 
- Remove redis check from our health endpoint in prod entirely
- Have the redis check have a maximum timeout of 1 second
 2015-10-22 14:24:42 -04:00
Jake Moshenko
67ad7ecc88 Remove the unused imagestorage columns from the db. 2015-10-22 12:14:54 -04:00
Jake Moshenko
ce94931540 Stop writing to deprecated columns for image data. 2015-10-22 12:14:39 -04:00
Joseph Schorr
803a983126 Fix deletion of repos and users with V2 stuff 
Fixes #674
 2015-10-22 11:58:52 -04:00
Jimmy Zelinskie
39cfe77d42 Revert "Merge pull request #557 from coreos-inc/revert-migration" 
This reverts commit c4f938898a, reversing
changes made to 7ad2522dbe.
 2015-10-21 15:29:57 -04:00
Joseph Schorr
147b7b26b4 Fix is_public in repo list 
Fixes #678
 2015-10-21 14:13:39 -04:00
Joseph Schorr
4e5c8a9281 Reduce GC work time and make sure to use distinct query 2015-10-20 18:13:29 -04:00
Jimmy Zelinskie
2dea9cf05e Merge pull request #666 from jzelinskie/fixbackfill 
Copy new fields over to the linked image
 2015-10-19 17:18:06 -04:00
Jimmy Zelinskie
109d69abfd Copy new fields over to the linked image 
This potentially fixes an issue with the v2 image field backfill. We
should be safe to copy these fields over at link time so that hopefully
it doesn't get skipped by the docker client. `_find_or_link_image`
should NEVER be used by the registry v2 protocol.
 2015-10-19 17:11:11 -04:00
Joseph Schorr
5941f3937c Enable async GC for all 
Fixes #569
 2015-10-19 14:22:41 -04:00
josephschorr
2f42a4d94d Merge pull request #641 from coreos-inc/wildcardfix 
Make sure to filter wildcard queries
 2015-10-15 14:26:51 -04:00
Joseph Schorr
6df7f60e4a Make sure to filter wildcard queries 
Fixes #640
 2015-10-15 14:26:33 -04:00
josephschorr
d3857e509f Merge pull request #643 from coreos-inc/nullimage 
Check and handle NULL image_size
 2015-10-15 13:26:13 -04:00
Joseph Schorr
fe79d5fb66 Check and handle NULL image_size 
Fixes #613
 2015-10-15 13:25:54 -04:00
Joseph Schorr
c9daf7d8a9 Add additional tests for repo visibility and further simplify the query for perf 2015-10-15 12:12:57 -04:00
Joseph Schorr
e8cb359d96 Unionize the mega query - It needed more performance-based benefits 2015-10-09 14:45:05 -07:00
Jimmy Zelinskie
9818481b08 limit logs to a maximum number of pages 2015-10-06 14:13:23 -04:00
Matt Jibson
87cc3289a0 Remove transaction from metric reporting 2015-10-06 01:28:43 -04:00
Joseph Schorr
8ca92d6828 Remove old search API and switch V1 search to use the new search system 2015-10-05 14:36:43 -04:00
Joseph Schorr
6e0ca735a5 Add a better redis health check that reads and writes 
This will hopefully catch issues earlier with Redis
 2015-09-30 15:23:19 -04:00
Joseph Schorr
35c35d9913 Load images and storage references in bulk during V1 synthesize 
Currently, we perform multiple queries for each layer, making it much slower (especially cross-region)

Fixes #413
 2015-09-29 17:53:39 -04:00
Silas Sewell
9000169b53 Revert "Merge pull request #491 from jakedt/migratebackp2" 
This reverts commit 7ad2522dbe, reversing
changes made to a0b191ffa1.
 2015-09-28 16:09:22 -04:00
josephschorr
7ad2522dbe Merge pull request #491 from jakedt/migratebackp2 
Migrate image data back phase 2
 2015-09-26 15:11:46 -04:00
Matt Jibson
4da66c1219 Move the metric put outside the transaction 2015-09-21 13:37:49 -04:00
Jimmy Zelinskie
2ff77df946 Merge pull request #518 from jzelinskie/fixmysqlssl 
move UseThenDisconnect into queueworker
 2015-09-21 13:35:35 -04:00
Jimmy Zelinskie
7c82e0b5b3 move UseThenDisconnect into queueworker 
This makes the tests pass while maintaining the same behavior.
 2015-09-21 13:34:12 -04:00
Jimmy Zelinskie
0de17627d5 Merge pull request #517 from jzelinskie/fixmysqlssl 
close connections after getting queue metrics
 2015-09-21 12:28:23 -04:00
Jimmy Zelinskie
98d6262a7f close connections after getting queue metrics 2015-09-21 12:21:39 -04:00
Matt Jibson
bba1557437 Monitor queue adds and EC2 node starts 
fixes #157
see #304
 2015-09-18 16:21:16 -04:00
Jake Moshenko
26cea9a07c Merge remote-tracking branch 'upstream/master' into python-registry-v2 2015-09-17 16:16:27 -04:00
Jake Moshenko
8baacd2741 Migrate old data to new locations, read only new. 2015-09-17 15:47:13 -04:00
Joseph Schorr
b807accfb5 Fix migration head 2015-09-16 18:34:42 -04:00
josephschorr
217779273f Merge pull request #503 from coreos-inc/ghmigrate 
Migrate all GitHub build triggers to use deploy keys
 2015-09-16 18:32:32 -04:00
Joseph Schorr
eff9ff7a66 Migrate all GitHub build triggers to use deploy keys 2015-09-16 17:55:51 -04:00
Silas Sewell
0a48f1cfb0 Merge pull request #495 from coreos-inc/quay-versions 
Add quay releases
 2015-09-16 17:29:58 -04:00
Silas Sewell
386c017d99 Add quay releases 2015-09-16 17:18:46 -04:00
Joseph Schorr
30379a2dd8 Fix interleaved repo delete with RAC via a transaction 
The RepositoryActionCount table can have entries added while a repository deletion is in progress. We now perform the repository deletion under a transaction and explicitly test for RAC entries in the deletion unit test (which doesn't test interleaving, but it was missing this check).

Fixes #494
 2015-09-16 15:34:32 -04:00
Jake Moshenko
502f5e4c8a Missed one place to duplicate metadata. 2015-09-15 15:57:55 -04:00
Jake Moshenko
b56de3355c Migrate data back to Image in preparation for v2 2015-09-15 11:53:31 -04:00
Matt Jibson
d36c7dcb4b Merge pull request #425 from mjibson/monitor-queue-size 
Monitor various sizes for queues
 2015-09-14 16:13:31 -04:00
Matt Jibson
39dc4c7d8d Monitor various sizes for queues 
see #304
 2015-09-14 15:57:08 -04:00
josephschorr
6d8752bdb5 Merge pull request #454 from coreos-inc/urlfor 
Remove uses of _external for url_for
 2015-09-14 15:54:42 -04:00
Jake Moshenko
9c3ddf846f Some fixes and tests for v2 auth 
Fixes #395
 2015-09-10 15:38:57 -04:00
Joseph Schorr
96d5bbb155 Fix exceptions raised by the diffs worker 
Fixes #465
 2015-09-10 14:12:16 -04:00
josephschorr
edef283697 Merge pull request #447 from coreos-inc/ronon 
Add support for Dex to Quay
 2015-09-10 11:42:01 -04:00
Joseph Schorr
474fffd01f Select the full RepositoryBuild record 
If we just return the ID, then peewee just fills in the other fields with defaults (such as UUID).
 2015-09-09 21:43:48 -04:00
Jimmy Zelinskie
ece08f6e88 Merge pull request #463 from jzelinskie/fixpagination 
fix pagination of tags in API
 2015-09-09 15:53:55 -04:00
Jimmy Zelinskie
ebdee55585 list_repository_tag_history fallback orderby name 
If tags are created at the same time (usually from a tight loop), it is
possible that they will be order nondeterministically unless we fallback
to another orderby.
 2015-09-09 15:52:25 -04:00
Joseph Schorr
3ee4147117 Switch the build logs archiver to a more performant query 
Fixes #459
 2015-09-09 13:59:45 -04:00
Joseph Schorr
f0c8552668 Remove uses of _external for url_for 
Fixes #439
 2015-09-08 10:29:28 -04:00
Joseph Schorr
c0286d1ac3 Add support for Dex to Quay 
Fixes #306

- Adds support for Dex as an OAuth external login provider
- Adds support for OIDC in general
- Extract out external logins on the JS side into a service
- Add a feature flag for disabling direct login
- Add support for directing to the single external login service
- Does *not* yet support the config in the superuser tool
 2015-09-04 17:05:06 -04:00
Jake Moshenko
210ed7cf02 Merge remote-tracking branch 'upstream/master' into python-registry-v2 2015-09-04 16:32:01 -04:00
Jake Moshenko
82efc746b3 Make our JWT checking more strict. 2015-09-04 15:18:57 -04:00
josephschorr
9889ca268a Merge pull request #432 from coreos-inc/oauthcheck 
Build the OAuth redirect URL ourselves, rather than relying on undocumented Flask behavior
 2015-09-02 13:35:44 -04:00
Joseph Schorr
b7f487da42 Build the OAuth redirect URL ourselves, rather than relying on undocumented Flask behavior 2015-09-02 13:32:11 -04:00
josephschorr
62ea4a6cf4 Merge pull request #191 from coreos-inc/carmen 
Add automatic storage replication
 2015-09-01 15:04:36 -04:00
Joseph Schorr
724b1607d7 Add automatic storage replication 
Adds a worker to automatically replicate data between storages and update the database accordingly
 2015-09-01 14:53:32 -04:00
Joseph Schorr
51c507d02d Add back the ability to retrieve information for an org member directly 
Fixes #427
 2015-08-31 16:45:24 -04:00
Joseph Schorr
c0c1da3232 Change build logs load to using streaming Gzip 2015-08-28 14:08:13 -04:00
Jake Moshenko
398202e6fc Implement some new methods on the storage engines. 2015-08-27 11:29:19 -04:00
Joseph Schorr
2b724509b9 Fix 500 error when pushing a manifest previously pushed 
Fixes #400
 2015-08-25 15:34:49 -04:00
Joseph Schorr
1450b7e84c Fix verbs support in V2 2015-08-24 12:05:09 -04:00
Joseph Schorr
cf030e2a98 Save the compressed image size on blob upload completion 2015-08-24 12:05:09 -04:00
Joseph Schorr
d246e68e68 Move shared V1/V2 code into common methods and fix verbs 2015-08-24 12:05:09 -04:00
Jake Moshenko
b998eca8e5 Fix the tests for registry v2 changes. 2015-08-24 11:59:12 -04:00
Jimmy Zelinskie
7787e1350d Merge pull request #386 from coreos-inc/missingmigration2 
Add missing migration
 2015-08-21 14:21:51 -04:00
Joseph Schorr
e7c405f56b Add missing migration 
Should have been in commit 84276ee945
 2015-08-21 14:21:11 -04:00
Joseph Schorr
e5d2083912 Add new carrier billing plan 
Fixes #370
 2015-08-21 14:10:48 -04:00
Matt Jibson
4cb4288672 Merge pull request #373 from mjibson/fix-metric-tests 
Fix test_queue.py tests
 2015-08-18 14:05:29 -04:00
Joseph Schorr
0854d20cbd SECURITY FIX FOR LDAP 
It appears the recent migration of the LDAP code and add of a check for the admin username/password being invalid *broke the LDAP password check*, allowing any password to succeed for login. This fixes the problem, add unit tests to verify the fix and add some tests to our other external auth test suite.

A release will be needed immediately along with an announcement
 2015-08-18 12:32:19 -04:00
Matt Jibson
fc671f3dde Fix test_queue.py tests 
This restores the reporter class as was before the metrics changes.
 2015-08-17 17:22:46 -04:00
Joseph Schorr
84276ee945 Better notifications UI 
Fixes #369
 2015-08-17 17:08:58 -04:00
Jake Moshenko
2fd1d5969e Merge pull request #351 from mjibson/more-metrics 
More metrics
 2015-08-17 13:09:08 -04:00
Jake Moshenko
e1b3e9e6ae Another huge batch of registry v2 changes 
Add patch support and resumeable sha
Implement all actual registry methods
Add a simple database generation option
 2015-08-12 16:41:12 -04:00
Jake Moshenko
ec6bee35b6 Allow a stripe plan to be superseded 
If a plan has a direct corrolary, show that one as the selected plan
instead of showing the plan as deprecated even though it has the same
details
 2015-08-12 15:01:15 -04:00
Matt Jibson
cfb6e884f2 Refactor metric collection 
This change adds a generic queue onto which metrics can be pushed. A
separate module removes metrics from the queue and adds them to Cloudwatch.
Since these are now separate ideas, we can easily change the consumer from
Cloudwatch to anything else.

This change maintains near feature parity (the only change is there is now
just one queue instead of two - not a big deal).
 2015-08-12 12:15:52 -04:00
Jake Moshenko
0cbc96a54b Unify the free trial period 
Fixes #263
 2015-08-10 17:36:13 -04:00
Joseph Schorr
ea25538646 MySQL and Postgres complain about the group by, so calculate dates ourselves 2015-08-06 12:52:55 -04:00
Joseph Schorr
d34afde954 Fix logs view and API 
- We needed to use an engine-agnostic way to extract the days
- Joining with the LogEntryKind table has *horrible* performance in MySQL, so do it ourselves
- Limit to 50 logs per page
 2015-08-05 17:47:03 -04:00
Joseph Schorr
d480a204f5 Revert change to queue 2015-08-05 15:27:33 -04:00
josephschorr
ee53c04a45 Merge pull request #309 from coreos-inc/fasterqueue 
Improve the performance of queue candidate queries.
 2015-08-04 18:24:28 -04:00
Jake Moshenko
ed62339f89 Improve the performance of queue candidate queries. 2015-08-04 18:20:54 -04:00
Joseph Schorr
9f2d6282bd Add missing index on retries_remaining 2015-08-04 18:01:28 -04:00
josephschorr
f772bd0c9e Merge pull request #300 from coreos-inc/toomanyutils 
Refactor the util directory to use subpackages.
 2015-08-03 16:18:55 -04:00
Jake Moshenko
18100be481 Refactor the util directory to use subpackages. 2015-08-03 16:04:19 -04:00
Jimmy Zelinskie
8e6a0fbbee Merge pull request #294 from coreos-inc/logsload 
Switch to using an aggregated logs query and infinite scrolling
 2015-08-03 14:52:04 -04:00
Joseph Schorr
3d6c92901c Switch to using an aggregated logs query and infinite scrolling 
This should allow users to work with large logs set.

Fixes #294
 2015-07-31 16:38:02 -04:00
Jake Moshenko
e133ea0962 Try not to throw any sets of data away when computing images to garbage collect. 2015-07-31 16:12:57 -04:00
Joseph Schorr
0fdc8b0f1f Fix spelling of ancestors 2015-07-28 15:30:04 -04:00
Joseph Schorr
ba7686af99 Switch back to the read-then-write tag deletion code 
We changed to this originally to avoid locks
 2015-07-28 15:30:04 -04:00
Joseph Schorr
70de107268 Make GC of repositories fully async for whitelisted namespaces 
This change adds a worker to conduct GC on repositories with garbage every 10s.

Fixes #144
 2015-07-28 15:30:04 -04:00
Joseph Schorr
acd86008c8 Switch tag deletion to use a single query 2015-07-28 15:30:04 -04:00
Joseph Schorr
378c83598d Fix subquery issues in MySQL 2015-07-28 15:28:00 -04:00
Joseph Schorr
66b3d45fbc Remove legacy.py that was misadded 2015-07-27 15:53:25 -04:00
Joseph Schorr
c3f269ee23 Add migration for BitBucket web hooks 
This needs to added only *after* we roll out #255
 2015-07-23 14:45:12 -04:00
Joseph Schorr
ac1b46e7ec Add missing migration 2015-07-22 16:19:10 -04:00
Joseph Schorr
687bab1c05 Support invite codes for verification of email 
Also changes the system so we don't apply the invite until it is called explicitly from the frontend

Fixes #241
 2015-07-22 13:41:27 -04:00
Jake Moshenko
5d86fa80e7 Merge pull request #197 from coreos-inc/keystone 
Add Keystone Auth
 2015-07-22 13:38:47 -04:00
Joseph Schorr
38a6b3621c Automatically link the superuser account to federated service for auth 
When the user commits the configuration, if they have chosen a non-DB auth system, we now auto-link the superuser account to that auth system, to ensure they can login again after restart.
 2015-07-22 13:37:23 -04:00
Joseph Schorr
a0c4e72f13 Clean up the repository list API and loads stars with it 
We load stars with the same list API now so that we get the extra metadata needed in the repo list (popularity and last modified)
 2015-07-22 13:05:02 -04:00
Joseph Schorr
7e4b23916a Small SQL query fix 
Fixes #248
 2015-07-20 14:17:26 -04:00
Joseph Schorr
33b54218cc Refactor the users class into their own files, add a common base class for federated users and add a verify_credentials method which only does the verification, without the linking. We use this in the superuser verification pass 2015-07-20 11:39:59 -04:00
Joseph Schorr
1245385808 Fix typo 2015-07-20 10:55:21 -04:00
Joseph Schorr
066637f496 Basic Keystone Auth support 
Note: This has been verified as working by the end customer
 2015-07-20 10:55:21 -04:00
Jake Moshenko
bc29561f8f Fix and templatize the logic for external JWT AuthN and registry v2 Auth. 
Make it explicit that the registry-v2 stuff is not ready for prime time.
 2015-07-17 11:56:15 -04:00
Jake Moshenko
3efaa255e8 Accidental refactor, split out legacy.py into separate sumodules and update all call sites. 2015-07-17 11:56:15 -04:00
Jake Moshenko
bea8b9ac53 More changes for registry-v2 in python. 
Implement the minimal changes to the local filesystem storage driver and feed them through the distributed storage driver.
Create a digest package which contains digest_tools and checksums.
Fix the tests to use the new v1 endpoint locations.
Fix repository.delete_instance to properly filter the generated queries to avoid most subquery deletes, but still generate them when not explicitly filtered.
 2015-07-17 11:50:41 -04:00
Jake Moshenko
acbcc2e206 Start of a v2 API. 2015-07-17 11:50:41 -04:00
Joseph Schorr
7a548ea101 Fix queries for repository list popularity and action count 
Before this change, we used extremely inefficient outer joins as part of a single query of lookup, which was spiking our CPU usage to nearly 100% on the query. We now issue two separate queries for popularity and action account, by doing a lookup of the previously found IDs. Interestingly enough, because of the way the queries are now written, MySQL can actually do both queries *directly from the indicies*, which means they each occur in approx 20ms!

Verified by local tests, postgres tests, and testing on staging with monitoring of our CPU usage during lookup
 2015-07-17 00:08:27 +03:00
Jimmy Zelinskie
2869e2a6ea model: add missing params to validate_database_url 2015-07-15 17:39:26 -04:00
Jake Moshenko
eec7886e01 Add a server default for the broken migration. 2015-07-14 16:58:58 -04:00
Jimmy Zelinskie
bde781c98b Merge pull request #205 from coreos-inc/delrobot 
Fix deletion of robot accounts when attached to builds
 2015-07-13 12:19:01 -04:00
Joseph Schorr
3a59c99b08 Add a secondary tab to Teams for managing org members 
Also adds the ability to completely remove a user from an organization (repo permissions and teams), in a single click

Fixes #212
 2015-07-02 17:06:36 +03:00
Joseph Schorr
b535e222b8 Have the fetch tag dialog show a warning for robot accounts without access 
Before this change, we'd show the squash pulling command with the proper credentials, but it then 403s on the end user.
 2015-07-01 19:37:52 +03:00
Joseph Schorr
3ba321934f Fix deletion of robot accounts when attached to builds 
Fixes #204
 2015-06-30 22:56:44 +03:00
josephschorr
7aeaf2344e Merge pull request #200 from coreos-inc/tagapilimit 
Add pagination support to tag history API
 2015-06-30 22:09:09 +03:00
Joseph Schorr
f7f10f4a6d Add pagination support to tag history API 
Fixes #198
 2015-06-30 19:44:43 +03:00
Jake Moshenko
38a5963afe Merge pull request #190 from coreos-inc/timezone 
Fromtimestamp needs to be in UTC for JWT auth
 2015-06-30 12:05:00 -04:00
Joseph Schorr
2b1bbcb579 Add a table view to the repos list page 
Fixes #104
 2015-06-29 21:12:53 +03:00
Jimmy Zelinskie
756d6784ca Merge pull request #192 from coreos-inc/sqlssl 
Allow SSL cert for the database to be configured
 2015-06-29 13:33:31 -04:00
Joseph Schorr
dc5af7496c Allow superusers to disable user accounts 2015-06-29 18:40:52 +03:00
Joseph Schorr
bb07d0965f Allow SSL cert for the database to be configured 
This change adds a field for the SSL cert for the database in the setup tool. Fixes #89
 2015-06-29 08:08:10 +03:00
Joseph Schorr
477e244eff Fromtimestamp needs to be in UTC for JWT auth 2015-06-28 11:37:09 +03:00
Joseph Schorr
e7915baf8c Have LDAP return a better error message if it fails to connect 
Currently, the error results in a 500 being raised when a user tries to login.
 2015-06-23 17:41:53 -04:00
Joseph Schorr
07439328a4 Remove user_exists endpoint from all auth systems 2015-06-23 17:33:51 -04:00
Jake Moshenko
b21a033ef3 Merge pull request #131 from coreos-inc/moveapp 
Refactor JWT auth to not import app locally
 2015-06-23 17:24:01 -04:00
Jake Moshenko
5f1d23c6e8 Use a UNION query instead of a multitude of left outer joins for performance reasons. 
Fixes #159
 2015-06-23 17:18:37 -04:00
Joseph Schorr
331c300893 Refactor JWT auth to not import app locally 2015-06-17 15:53:21 -04:00
Joseph Schorr
e7fa560787 Add support for custom fields in billing invoices 
Customers (especially in Europe) need the ability to add Tax IDs, VAT IDs, and other custom fields to their invoices.

Fixes #106
 2015-06-12 16:45:01 -04:00
Jake Moshenko
79f1181a63 Switch build-scheduled to an official build phase. 2015-06-10 16:19:51 -04:00
Jake Moshenko
e09d84b3c8 Merge pull request #55 from coreos-inc/oauthdeny 
Fix OAuth redirect for denial action when generating for internal tokens
 2015-06-05 14:00:16 -04:00
Joseph Schorr
8aac3fd86e Add support for an external JWT-based authentication system 
This authentication system hits two HTTP endpoints to check and verify the existence of users:

Existance endpoint:
GET http://endpoint/ with Authorization: Basic (username:) =>
    Returns 200 if the username/email exists, 4** otherwise

Verification endpoint:
GET http://endpoint/ with Authorization: Basic (username:password) =>
    Returns 200 and a signed JWT with the user's username and email address if the username+password validates, 4** otherwise with the body containing an optional error message

The JWT produced by the endpoint must be issued with an issuer matching that configured in the config.yaml, and the audience must be "quay.io/jwtauthn". The JWT is signed using a private key and then validated on the Quay.io side with the associated public key, found as "jwt-authn.cert" in the conf/stack directory.
 2015-06-05 13:20:10 -04:00
Jake Moshenko
42da017d69 Merge pull request #48 from coreos-inc/nobots 
Change API calls that expect non-robots to explicitly filter
 2015-06-02 12:31:19 -04:00
Joseph Schorr
5516911de9 Fix OAuth redirect for denial action when generating for internal tokens 2015-06-02 12:25:59 -04:00
Joseph Schorr
2a56790d38 Switch to using a named LDAP tuple for more readable code 2015-06-01 14:02:05 -04:00
Joseph Schorr
1aff701bc7 Fix LDAP referral and multiple pair handling 
Fixes two issues found with our LDAP handling code. First, we now follow referrals in both LDAP calls, as some LDAP systems will return a referral instead of the original record. Second, we now make sure to handle multiple search result pairs properly by further filtering based on the presence of the 'mail' attribute when we have multiple valid pairs. This CL also adds tests for all of the above cases.
 2015-05-27 15:04:34 -04:00
Joseph Schorr
fdd43e2490 Change API calls that expect non-robots to explicitly filter 
Before this change, we'd filter in the UI but calls to the API could allow robots accounts where we only expect real users
 2015-05-26 17:47:33 -04:00
Joseph Schorr
b0d763b5ff Fix encrypted password generator to use the LDAP username, not the Quay username. 
Currently, we use the Quay username via `verify_user` when we go to create the encrypted password. This is only correct if Quay has not generated its own different username for the LDAP user, and fails if it has. We therefore add a new method `confirm_existing_user`, which looks up the federated login for the LDAP user and then runs the auth flow using that username.
 2015-05-20 16:37:09 -04:00
Joseph Schorr
d5e70c6e2a Explicitly enable LDAP referrals 
Note: The mock LDAP system doesn't support referrals, so we can't add a unit test for this.
 2015-05-20 14:53:31 -04:00
Jimmy Zelinskie
3daa0e6a52 migration: use SA boolean value for server_default 2015-05-15 19:04:54 -04:00
Jimmy Zelinskie
e12d5af52c migration: server_default for boolean field 2015-05-15 17:04:50 -04:00
Joseph Schorr
6a52ffa942 Add missing default value on the new reversion column 2015-05-13 13:49:21 -04:00
Joseph Schorr
07b4fb9105 LDAP sometimes has multiple records for a user 2015-05-12 12:02:09 -04:00
Joseph Schorr
efab02ae47 LDAP improvements: 
- Better logging
  - Better error messages
  - Add unit tests
  - Clean up the setup tool for LDAP
 2015-05-11 21:23:18 -04:00
Joseph Schorr
3e1abba284 Add ability for super users to rename and delete organizations 2015-05-11 18:03:25 -04:00
Joseph Schorr
1c41d34b7c Add ability for superusers to change user emails 2015-05-11 14:38:10 -04:00
Joseph Schorr
1c83def15b LDAP should only show logs when asked. 2015-05-11 13:01:49 -04:00
Joseph Schorr
f858caf6cd Only return the team and repo permissions when listing robots when we absolutely need them. 2015-05-08 16:43:07 -04:00
Joseph Schorr
561f2c7db0 Remove unnecessary sort 2015-05-08 15:10:31 -04:00
Joseph Schorr
3b93854c0b MySQL returns a decimal type for the tuple, which we need to convert to an int 2015-05-08 14:55:54 -04:00
Joseph Schorr
36f33e8fd7 Merge branch 'master' of github.com:coreos-inc/quay 2015-05-08 14:19:39 -04:00
Joseph Schorr
f9c1f123c2 Add better debugging to LDAP 2015-05-08 14:19:32 -04:00
Joseph Schorr
8ed8367404 PR changes in response to comments 2015-05-08 13:38:34 -04:00
Joseph Schorr
c767aafcd6 Make the repository API faster by only checking the log entries table once for each kind of entry, rather than twice. We make use of a special subquery-like syntax, which allows us to count those entries that are both 30 days only and 1 day old in the same query. This was tested successfully on MySQL, Postgres and Sqlite. 2015-05-07 22:49:11 -04:00
Joseph Schorr
3627de103c Minimize the queries used when retrieve builds. Previously, we'd call out to SQL extra times per build. 2015-05-07 21:11:15 -04:00
Joseph Schorr
7b35555776 Make sure to test for unicode usernames, since the collate on the username field is latin1 2015-05-07 18:13:45 -04:00
Joseph Schorr
729249f42b Need to add the tables to the .select() to prevent N+1, not merely to the join 2015-05-07 16:59:13 -04:00
Joseph Schorr
b50600cde8 Make sure to join the Repo table on its namespace user and its visibility so we don't issue additional queries from the starred repos API call 2015-05-07 16:18:17 -04:00
Joseph Schorr
3cec22defe Make sorting by repo ID, rather than description 2015-05-07 13:10:26 -04:00
Joseph Schorr
73193e2ab5 Remove sort by description, which is slow (and fairly useless anyway) 2015-05-06 20:55:31 -04:00
Joseph Schorr
c89760278f Add migration to add gitlab trigger type 2015-05-03 10:49:13 -07:00
Joseph Schorr
a59100b231 Add the missing index on the peewee side. We already have the associated migration. 2015-04-30 15:56:40 -04:00
Joseph Schorr
17bc9b81bf Add bitbucket trigger type 2015-04-30 15:55:57 -04:00
Joseph Schorr
b96e35b28c Merge master into bitbucket 2015-04-30 15:52:08 -04:00
Joseph Schorr
31260d50f5 Rename the new images method to a slightly better name 2015-04-24 16:37:37 -04:00
Joseph Schorr
e70343d849 Faster cache lookup by removing a join with the ImagePlacementTable, removing the extra loop to add the locations and filtering the images looked up by the base image 2015-04-24 16:22:19 -04:00
Jimmy Zelinskie
fd65ca5916 migration: add custom-git service to database 2015-04-24 16:11:58 -04:00
Joseph Schorr
c480fb2105 Work in progress: bitbucket support 2015-04-24 15:13:08 -04:00
Jimmy Zelinskie
d25cc4db9c migrations: fix percona migration test 2015-04-21 18:01:40 -04:00
Jimmy Zelinskie
93cd459460 Merge branch 'master' into git 2015-04-20 10:58:49 -04:00
Joseph Schorr
43ff6839b8 Hide hidden tags in the tags timeline 2015-04-19 18:12:06 -04:00
Joseph Schorr
ed342ae831 Add migration for properly creating the repository_id+datetime index 2015-04-19 16:03:06 -04:00
Joseph Schorr
d1e2d072ea Add unit tests and a stronger restriction on the revert API call 2015-04-19 15:43:16 -04:00
Joseph Schorr
a8f8c317f9 Fix branch in alembic migrations 2015-04-19 15:20:01 -04:00
Joseph Schorr
2815ea2723 Merge branch 'master' of github.com:coreos-inc/quay 2015-04-19 15:16:39 -04:00
Joseph Schorr
8714eb207a Merge branch 'actioncount' 2015-04-19 15:12:44 -04:00
Jimmy Zelinskie
4133924813 migrations: rebase on top of master's changes 2015-04-17 13:19:04 -04:00
Jimmy Zelinskie
ba2cb08904 Merge branch 'master' into git 2015-04-16 17:38:35 -04:00
Joseph Schorr
f19d2f684e Add ability to revert tags via time machine 2015-04-16 17:18:00 -04:00
Joseph Schorr
f8c80f7d11 Add a history view to the tags page. Next step will add the ability to revert back in time 2015-04-15 15:21:09 -04:00
Joseph Schorr
3f1e8f3c27 Add a RepositoryActionCount table so we can use it (instead of LogEntry) when scoring repo search results 2015-04-13 13:31:07 -04:00
Joseph Schorr
1df025b57e Change search to use a set of queries for repo lookup rather than a single monolithic query, in the hopes that this will make things significantly faster and actually useable. The individual queries have been tested by hand on MySQL, but the real test will be staging 2015-04-10 15:27:37 -04:00
Joseph Schorr
0be0aed17d Move the repo sorting by pull count into the main matching query, to both make it more accurate and make the search faster 2015-04-09 14:41:59 -04:00
Joseph Schorr
396cba64e6 Fix search to return better results by searching for robots and namespaces in different queries. 2015-04-09 12:57:20 -04:00
Joseph Schorr
762fdaf319 Fix DB migration tests 2015-04-08 17:41:53 -04:00
Joseph Schorr
4f4bb05621 Fix search SQL issues 2015-04-08 17:41:08 -04:00
Joseph Schorr
19e25ac340 Merge branch 'master' into bing 2015-04-08 15:23:36 -04:00
Joseph Schorr
d09f2f6e22 Get the new context-sensitive new menu working 2015-04-07 18:33:43 -04:00
Joseph Schorr
40a6892a49 Add search tests 2015-04-07 14:05:12 -04:00
Joseph Schorr
1b56567268 Make sure also include teams from organizations that the user admins 2015-04-07 13:45:49 -04:00
Joseph Schorr
951b0cbab8 Start on new interactive search 2015-04-06 19:17:18 -04:00
Joseph Schorr
4cb7921c3a Make sure to show public repos on the user and orgs pages 2015-04-03 14:55:09 -04:00
Joseph Schorr
094f91fb8b Fix the tutorial's user events 2015-04-03 12:13:33 -04:00
Joseph Schorr
5cd500257d Merge branch 'master' into orgview 2015-04-01 13:56:49 -04:00
Joseph Schorr
fde9666647 Add the team membership to the robots view 2015-04-01 13:56:30 -04:00
Joseph Schorr
1f5e6df678 - Fix tests 
- Add new endpoints for retrieving the repo permissions for a robot account
- Have the robots list return the number of repositories for which there are permissions
- Other UI fixes
 2015-03-31 18:50:43 -04:00
Joseph Schorr
27a9b84587 Switch avatars to be built out of CSS and only overlayed with the gravatar when a non-default exists 2015-03-30 17:55:04 -04:00
Jimmy Zelinskie
da15eda2bf migration: make auth_token for builds nullable 2015-03-27 11:20:30 -04:00
Joseph Schorr
7b1d35737e Merge branch 'master' of github.com:coreos-inc/quay 2015-03-26 17:45:49 -04:00
Joseph Schorr
a7b6cb5c23 Fix handling of byte strings and large ints 2015-03-26 17:45:43 -04:00
Jimmy Zelinskie
4d4947edaf Merge branch 'master' of github.com:coreos-inc/quay 2015-03-26 16:31:20 -04:00
Jimmy Zelinskie
bcc7a9580b models: change UUID of user on password change 
This prevents old cookies from continuing to work after a password has
been changed.
 2015-03-26 16:30:41 -04:00
Joseph Schorr
c4a2574b0d Clarify unencrypted password error message 2015-03-26 16:23:28 -04:00
Joseph Schorr
f8afd8b5ce Make sure to parse the big int into a byte string 2015-03-26 16:13:35 -04:00
Joseph Schorr
4d1792db1c getrandbits creates an int, not a float 2015-03-26 15:47:44 -04:00
Joseph Schorr
aaf1b23e98 Address CL concerns and switch to a real encryption system 2015-03-26 15:10:58 -04:00
Joseph Schorr
d23bb6616d Fix error message to exactly match current output 2015-03-26 13:22:16 -04:00
Joseph Schorr
e4b659f107 Add support for encrypted client tokens via basic auth (for the docker CLI) and a feature flag to disable normal passwords 2015-03-25 18:43:12 -04:00
Jake Moshenko
3d44416016 Add the migration for the unique index which helps prevent tag deadlocks. 2015-03-24 18:00:04 -04:00
Jake Moshenko
201943ed1c Fix deadlocks with tags and garbage collection. 2015-03-24 18:00:04 -04:00
Jimmy Zelinskie
9c55aca011 migration: make resource_key nullable 2015-03-23 15:46:35 -04:00
Jimmy Zelinskie
93a9e9d01a migration: add private key to build triggers 2015-03-19 14:30:25 -04:00
Joseph Schorr
ab2331a486 Performance improvements for the repo API and the new repo UI 2015-03-18 14:47:53 -04:00
Joseph Schorr
fe8d006855 Fix subquery on MySQL 2015-03-17 13:54:51 -04:00
Joseph Schorr
44ff85d044 Remove migration (temporarily), fix a broken test, and make the aggregate size calculation use the entire image ancestry (for now). 2015-03-17 12:13:01 -04:00
Joseph Schorr
b8d88c0f4e Add aggregate size column and a migration to backfill it 2015-03-16 18:03:17 -04:00
Joseph Schorr
333e0acd6d Add the builds tab 2015-03-13 15:34:28 -07:00
Joseph Schorr
002dc083f2 Get the main repo page design working 2015-03-10 17:22:46 -07:00
Joseph Schorr
afc8e95e19 Start on new tag view 2015-03-09 22:03:39 -07:00