vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
8393 commits 2 branches 62 tags 205 MiB
Commit graph

1133 commits

Author SHA1 Message Date
Joseph Schorr
d63cca025a DNS name check got reversed; breaks wildcards 2017-01-29 11:51:37 -05:00
Charlton Austin
dae93dce78 feature(superuser panel): ability to view logs 
users would like the ability to view build logs in the superuser panel

[None]
 2017-01-26 13:54:03 -05:00
Joseph Schorr
a9791ea419 Have external login always make an API request to get the authorization URL 
This makes the OIDC lookup lazy, ensuring that the rest of the registry and app continues working even if one OIDC provider goes down.
 2017-01-23 19:06:19 -05:00
Joseph Schorr
fda203e4d7 Add proper and tested OIDC support on the server 
Note that this will still not work on the client side; the followup CL for the client side is right after this one.
 2017-01-23 17:53:34 -05:00
Charlton Austin
7854bf6b3a Making test independent of message ordering. 2017-01-23 14:32:34 -05:00
Joseph Schorr
19f7acf575 Lay foundation for truly dynamic external logins 
Moves all the external login services into a set of classes that share as much code as possible. These services are then registered on both the client and server, allowing us in the followup change to dynamically register new handlers
 2017-01-20 15:21:08 -05:00
Joseph Schorr
4755d08677 Refactor and rename the standard OAuth services 2017-01-19 15:23:15 -05:00
Joseph Schorr
7c7a07fb5a Allow namespaces to be between 2 and 255 characters in length 
[Delivers #137924329]
 2017-01-19 13:10:26 -05:00
josephschorr
e2748fccd9 Merge pull request #2282 from coreos-inc/motd-updates 
Severity and Markdown support in MOTD
 2017-01-18 17:41:27 -05:00
Joseph Schorr
3106504f39 Severity and Markdown support in MOTD 
[Delivers #133555165]
 2017-01-18 16:55:32 -05:00
Joseph Schorr
b3a74b94b3 Fix flaky port selection in registry tests 
Depends on https://github.com/jarus/flask-testing/pull/98

[Fixes #136705135]
 2017-01-18 15:06:11 -05:00
Joseph Schorr
462f47924e More detailed namespace validation 
Fixes namespace validation to use the proper regex for checking length, as well as showing the proper messaging if the entered namespace is invalid

[Delivers #137830461]
 2017-01-17 17:31:59 -05:00
josephschorr
aafcb592a6 Merge pull request #2257 from coreos-inc/clair-gc-take2 
feat(gc): Garbage collection for security scanning
 2017-01-17 14:49:36 -05:00
Joseph Schorr
8c4e86f48b Change queue to use state-field for claiming items 
Before this change, the queue code would check that none of the fields on the item to be claimed had changed between the time when the item was selected and the item is claimed. While this is a safe approach, it also causes quite a bit of lock contention in MySQL, because InnoDB will take a lock on *any* rows examined by the `where` clause of the `update`, even if they will ultimately thrown out due to other clauses (See: http://dev.mysql.com/doc/refman/5.7/en/innodb-locks-set.html: "A ..., an UPDATE, ... generally set record locks on every index record that is scanned in the processing of the SQL statement. It does not matter whether there are WHERE conditions in the statement that would exclude the row. InnoDB does not remember the exact WHERE condition, but only knows which index ranges were scanned").

As a result, we want to minimize the number of fields accessed in the `where` clause on an update to the QueueItem row. To do so, we introduce a new `state_id` column, which is updated on *every change* to the QueueItem rows with a unique, random value. We can then have the queue item claiming code simply check that the `state_id` column has not changed between the retrieval and claiming steps. This minimizes the number of columns being checked to two (`id` and `state_id`), and thus, should significantly reduce lock contention. Note that we can not (yet) reduce to just a single `state_id` column (which should work in theory), because we need to maintain backwards compatibility with existing items in the QueueItem table, which will be given empty `state_id` values when the migration in this change runs.

Also adds a number of tests for other queue operations that we want to make sure operate correctly following this change.

[Delivers #133632501]
 2017-01-17 13:29:26 -05:00
Joseph Schorr
939c122f70 Complete item queue test 2017-01-17 13:26:09 -05:00
Joseph Schorr
dcfd379b17 Queue cancelation test 2017-01-17 13:26:09 -05:00
Charlton Austin
8ca8c17e27 Merge pull request #2225 from charltonaustin/adding_in_new_indices 
Adding in new indices for queueitem table.
 2017-01-17 11:46:51 -05:00
josephschorr
9b65b37011 Merge pull request #2245 from coreos-inc/recaptcha 
Add support for recaptcha during the create account flow
 2017-01-17 11:34:23 -05:00
josephschorr
eb2cafacd4 Merge pull request #2249 from coreos-inc/notifier-fixes 
Security notification pagination fix
 2017-01-17 11:33:25 -05:00
Charlton Austin
ca832df975 Adding in new indices for queueitem table. 2017-01-17 10:04:31 -05:00
Joseph Schorr
7e0fbeb625 Custom SSL certificates config panel 
Adds a new panel to the superuser config tool, for managing custom SSL certificates in the config bundle

[Delivers #135586525]
 2017-01-13 14:34:35 -05:00
Joseph Schorr
3a24871422 Add SSL certificate utility and tests 2017-01-10 17:06:13 -05:00
Joseph Schorr
3eb17b7caa Add support for recaptcha during the create account flow 
If the feature is enabled and recaptcha keys are given in config, then a recaptcha box is displayed in the UI when creating a user and a recaptcha response code *must* be sent with the create API call for it to succeed.
 2017-01-09 11:08:21 -05:00
Joseph Schorr
ce21788da8 test(queue): delete_namespaced_items 
Add queue tests for delete_namespaced_items
 2017-01-09 11:05:39 -05:00
Joseph Schorr
d609e6a1c4 Security scanner garbage collection support 
Adds support for calling GC in the security scanner for any layers+storage removed by GC on the Quay side
 2016-12-22 14:55:26 -05:00
Joseph Schorr
5225642850 Garbage collection image+storage callback support 
Add support to GC to invoke a callback with the image+storages removed. Only images whose storage was also removed will be sent to the callback. This will be used by security scanning for its own GC in the followup change.
 2016-12-22 14:27:42 -05:00
Joseph Schorr
ef80471a39 fix(136521333): Handle None email_or_id in avatar code 
Fixes https://www.pivotaltracker.com/story/show/136521333
 2016-12-21 15:00:55 -05:00
josephschorr
732ab67b57 Merge pull request #2252 from coreos-inc/parallel-tests 
Fix pytests and enable parallel registry tests
 2016-12-20 16:56:52 -05:00
Joseph Schorr
481cebe46b Fix pytests and enable parallel registry tests 2016-12-20 15:42:04 -05:00
Joseph Schorr
001691e579 Fix whitespace 2016-12-20 13:25:23 -05:00
Joseph Schorr
5b3212ea0e Change security notification code to use the new stream diff reporters 
This ensures that even if security scanner pagination sends Old and New layer IDs on different pages, they will properly be handled across the entire notification.

Fixes https://www.pivotaltracker.com/story/show/136133657
 2016-12-20 12:50:19 -05:00
Joseph Schorr
ced0149520 Implement helper classes for tracking streaming diffs, both indexed and non-indexed 
These classes will be used to handle the Layer ID paginated diffs from Clair.
 2016-12-20 12:50:18 -05:00
Joseph Schorr
e2efb6c458 Add default and configurable LDAP timeouts 
Fixes https://www.pivotaltracker.com/story/show/135885019
 2016-12-19 11:53:06 -05:00
josephschorr
e58e04b0e9 Merge pull request #2242 from coreos-inc/clair-exceptions 
Security scanner flow changes and auto-retry
 2016-12-16 15:54:52 -05:00
Joseph Schorr
405eca074c Security scanner flow changes and auto-retry 
Changes the security scanner code to raise exceptions now for non-successful operations. One of the new exceptions raised is MissingParentLayerException, which, when raised, will cause the security worker to perform a full rescan of all parent images for the current layer, before trying once more to scan the current layer. This should allow the system to be "self-healing" in the case where the security scanner engine somehow loses or corrupts a parent layer.
 2016-12-16 15:38:09 -05:00
josephschorr
f72185f527 Merge pull request #2240 from coreos-inc/wrong-email-invite-accept 
Fix attempts to confirm team invite for mismatched email address
 2016-12-16 14:30:37 -05:00
josephschorr
9fa16679f8 Merge pull request #2238 from coreos-inc/fake-clair 
Add a fake security scanner class for easier testing
 2016-12-15 20:51:24 -05:00
Joseph Schorr
785c74de52 Fix attempts to confirm team invite for mismatched email address 
Currently, if a user tries to confirm an invite sent to them on an account with a mismatching email address, we simply redirect to the org (where they get a 403). This change ensures they get the proper error response message, and restyles the error page to be nicer.

Fixes #2227
Fixes https://www.pivotaltracker.com/story/show/136088507
 2016-12-15 17:15:11 -05:00
Joseph Schorr
15041ac5ed Add a fake security scanner class for easier testing 
The FakeSecurityScanner mocks out all calls that Quay is expected to make to the security scanner API, and returns faked data that can be adjusted by the calling test case
 2016-12-14 17:11:45 -05:00
EvB
0a5d4990e6 test(endpoints/api): ensure empty 202 resp 2016-12-14 16:32:06 -05:00
Joseph Schorr
6871eb95b1 Send notifications for previously unscannable layers in QSS 
Following this change, if an image was previously indexed unsuccessfully, then we will send notifications once successfully indexed
 2016-12-14 11:25:45 -05:00
Joseph Schorr
a9a75cd4cf Add a test for selecting images to be scanned 2016-12-14 00:07:48 -05:00
Joseph Schorr
624b2a8385 Have security scanner analyze only send notifications for *new* layers 
Following this change, anytime a layer is indexed by the security scanner, we only send notifications out if the layer previously had a security_indexed_engine value of `-1`, thus ensuring it has *never* been indexed previously. This will allow us to change to version of the security scanner upwards, and have all the images be re-indexed, without firing off notifications in a spammy manner.
 2016-12-13 23:17:11 -05:00
Evan Cordell
5686c80af1 Revert "Add GC of layers in Clair" 
This reverts 49872838ab
 2016-12-13 18:40:58 -05:00
Joseph Schorr
1302fd2fbd Switch csrf token check to use compare_digest to prevent timing attacks 
Also adds some additional tests for CSRF tokens
 2016-12-08 23:46:31 -05:00
Joseph Schorr
dbdcb802b1 Add end-to-end OAuth login and attach tests 2016-12-08 18:35:42 -05:00
josephschorr
410b9d74fc Merge pull request #2214 from coreos-inc/clair-gc 
Add GC of layers in Clair
 2016-12-07 17:58:21 -05:00
josephschorr
111b7b0788 Merge pull request #2206 from coreos-inc/ldap-user-search-fix 
Fix external auth returns for query_user calls
 2016-12-07 17:53:04 -05:00
Jimmy Zelinskie
00eafff747 Merge pull request #2204 from jzelinskie/429builds 
add rate limiting to build queues
 2016-12-07 15:03:31 -05:00
Joseph Schorr
3203fd6de1 Fix external auth returns for query_user calls 
Adds the missing field on the query_user calls, updates the external auth tests to ensure it is returned properly, and adds new end-to-end tests which call the external auth engines via the *API*, to ensure this doesn't break again
 2016-12-07 14:28:42 -05:00
Jimmy Zelinskie
ebbe58d311 replace prefix w/ canonical name list 2016-12-07 12:56:56 -05:00
Joseph Schorr
49872838ab Add GC of layers in Clair 
Fixes https://www.pivotaltracker.com/story/show/135583207
 2016-12-06 19:52:56 -05:00
Jimmy Zelinskie
eb69abff8b build rate limiting: tests 2016-12-06 16:30:12 -05:00
Jake Moshenko
21e3001446 Add a bulk insert for queue and notifications. 
Use it for Clair spawned notifications.
 2016-12-06 14:00:16 -05:00
Joseph Schorr
97d150e281 Have QSS only add security scanner notifications once 2016-12-05 19:08:20 -05:00
Joseph Schorr
a565251b58 Remove check that breaks under full db tests 2016-12-02 17:46:01 -05:00
Charlton Austin
0a6322015c Fix the queue item delete. 2016-12-02 15:30:35 -05:00
Antoine Legrand
784c5f4fc7 Merge pull request #2160 from ant31/use_pytest 
Add pytest, tox  and code-coverage to run tests
 2016-12-02 15:53:40 +01:00
Joseph Schorr
fdff0bee4e Add configurable Docker host in full db tests 2016-12-01 15:45:08 -05:00
josephschorr
64c954dc58 Merge pull request #2182 from coreos-inc/fix-full-db-tests 
Fix full database test script to not fail randomly
 2016-12-01 14:33:22 -05:00
Charlton Austin
1f03fcb146 Adding in notification type for notification kind. 2016-12-01 12:26:18 -05:00
Joseph Schorr
e6ee538e15 Fix full database test script to not fail randomly 
- Switches database schema creation to alembic, which solves the MySQL issue (and makes sure we test migrations as well)
- Adds a few time.sleep(1) to work around MySQL's second-precision issue when adding items to queues and then immediately retrieving them
- Disables the storage proxy tests when running against non-SQLite databases, as it causes failures with the multiple process and multiple transactions
- Changes initdb to support only populating the database, as well as fixing a few small items around the test data when working with non-SQLite data
 2016-11-30 18:24:08 -05:00
Charlton Austin
2c637fe5ce Merge pull request #2173 from charltonaustin/adding_in_build_cancel_notifications 
Adding in cancel notifications
 2016-11-30 15:03:17 -05:00
Charlton Austin
4103a0b75f Adding in cancel notifications 2016-11-30 14:38:34 -05:00
Joseph Schorr
236655adb4 Fix config validator for storage and add a test suite 
Note that the test suite doesn't fully verify that each validation succeeds; rather, it ensures that the proper system (storage, security scanning, etc) is called with the configuration and returns at all (usually with an expected error). This should prevent us from forgetting to update these code paths when we change config-based systems. Longer term, we might want to have these tests stand up fake/mock versions of the endpoint services as well, for end-to-end testing.
 2016-11-30 11:58:41 -05:00
Joseph Schorr
402ad25690 Change team invitation acceptance to join all invited teams under the org 
Fixes #1989
 2016-11-28 18:39:28 -05:00
Evan Cordell
b4ace1dd29 registry auth tests: test more access types 2016-11-28 14:02:08 -05:00
Evan Cordell
9e96e6870f Add support for * (admin) permission to registry auth v2 endpoint 2016-11-28 14:02:08 -05:00
ant31
2eaa8a4a1b Add pytest and tox to run tests 2016-11-28 13:13:07 +01:00
Jimmy Zelinskie
498d7fc15e Merge pull request #2143 from jakedt/makebuildmanasyncagain 
Make buildman async again
 2016-11-21 15:08:06 -05:00
Charlton Austin
2fe74e4057 Adding in UI for cancel anytime. 2016-11-21 10:58:32 -05:00
Jake Moshenko
f0ef4347e5 Make the redis client use AsyncWrapper and coroutines 
Change all log messages to be synchronous
 2016-11-18 15:59:14 -05:00
Jake Moshenko
5935e93eb8 Linter fixes. 2016-11-18 15:56:08 -05:00
Joseph Schorr
0b549125d9 Fix 500 on get label endpoint and add a test 
Fixes #2133
 2016-11-17 14:55:14 -05:00
Joseph Schorr
69e2cfad70 Fix github trigger when submitting a webhook without a head_commit 
Fixes #2125
 2016-11-16 14:14:17 -05:00
josephschorr
1346b7fb63 Merge pull request #2105 from coreos-inc/frack-swift 
Fix swift exception reporting on deletion and add async chunk cleanup
 2016-11-15 17:59:48 -05:00
Joseph Schorr
5f99448adc Add a chunk cleanup queue for async GC of empty chunks 
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
 2016-11-15 15:07:41 -05:00
Joseph Schorr
1a61ef4e04 Report the user's name and company to Marketo 
Also fixes the API to report the other changes (username and email) as well
 2016-11-14 17:34:50 -05:00
Joseph Schorr
59cb6bd216 Make sure to not log exceptions if Swift deletes fail 2016-11-11 14:17:32 -05:00
Joseph Schorr
3d221bcdd7 Add tests for the Swift storage layer using a fake swift engine 2016-11-10 15:43:03 -05:00
josephschorr
45b1148118 Merge pull request #2086 from coreos-inc/user-info 
Add collection of user metadata: name and company
 2016-11-09 13:15:07 -05:00
Joseph Schorr
bf2804bd4d Add a test for deleting a user with a user prompt 2016-11-08 18:27:12 -05:00
Joseph Schorr
0f2eb61f4a Add collection of user metadata: name and company 2016-11-08 16:15:02 -05:00
josephschorr
233b2be5c2 Merge pull request #2066 from coreos-inc/select-username 
Add support for temp usernames and an interstitial to confirm username
 2016-11-03 16:22:16 -04:00
Joseph Schorr
1e3b354201 Add support for temp usernames and an interstitial to confirm username 
When a user now logs in for the first time for any external auth (LDAP, JWT, Keystone, Github, Google, Dex), they will be presented with a confirmation screen that affords them the opportunity to change their Quay-assigned username.

Addresses most of the user issues around #74
 2016-11-03 15:59:14 -04:00
Joseph Schorr
3fd92aef35 Fix entity search API to not IndexError 2016-11-02 16:22:35 -04:00
josephschorr
840ea4e768 Merge pull request #2047 from coreos-inc/external-auth-email-optional 
Make email addresses optional in external auth if email feature is turned off
 2016-10-31 14:16:33 -04:00
Joseph Schorr
d7f56350a4 Make email addresses optional in external auth if email feature is turned off 
Before this change, external auth such as Keystone would fail if a user without an email address tried to login, even if the email feature was disabled.
 2016-10-31 13:50:24 -04:00
Joseph Schorr
bab17932ac Fix namespace lookup in V1 registry search 
Fixes #2053
 2016-10-31 13:24:40 -04:00
josephschorr
934cdecbd6 Merge pull request #1905 from coreos-inc/external-auth-search 
Add support for entity search against external auth users not yet linked
 2016-10-27 16:06:42 -04:00
Joseph Schorr
b3d1d7227c Add support to Keystone Auth for external user linking 
Also adds Keystone V3 support
 2016-10-27 15:42:03 -04:00
Joseph Schorr
fbb524e34e Add support to ExternalJWT Auth for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
f9ee8d2bef Add support to LDAP for external user linking 2016-10-27 15:42:03 -04:00
Joseph Schorr
d145222812 Add support for linking to external users in entity search 2016-10-27 15:42:03 -04:00
Jimmy Zelinskie
30821569a4 key server: fix tests by exporting jwk_with_kid 2016-10-25 16:14:18 -04:00
josephschorr
500a218768 Merge pull request #1875 from coreos-inc/max-chunk-size 
Make sure we don't generate chunk sizes larger than 5 GB.
 2016-10-25 14:12:31 -04:00
Joseph Schorr
bfe2646a50 Make sure we don't generate chunk sizes larger than 5 GB. 
Amazon S3 does not allow for chunk sizes larger than 5 GB; we currently don't handle that case at all, which is why large uploads are failing. This change ensures that if a storage engine specifies a *maximum* chunk size, we write multiple chunks no larger than that size.
 2016-10-25 13:57:49 -04:00
Jake Moshenko
6f815907a4 Merge pull request #2030 from jakedt/twooh 
Prepare the changelog for v2.0.0
 2016-10-24 16:30:49 -04:00
Charlton Austin
dc35769396 Merge pull request #2022 from charltonaustin/refactor_for_cancel_anytime 
Making some refactors to make it easier to cancel the build at any time.
 2016-10-24 16:17:55 -04:00
Jake Moshenko
45bacbabaa s/Regions/Deployments 2016-10-24 16:04:04 -04:00
Charlton Austin
1cde22e76c Making some refactors to make it easier to cancel the build at any time. 2016-10-24 15:59:33 -04:00
Joseph Schorr
19393a8619 Add a test for deleting a user with federated login 2016-10-21 17:55:22 -04:00
Joseph Schorr
5ed13da2e6 Add missing security test for delete org 2016-10-21 17:37:49 -04:00
Joseph Schorr
73eb66eac5 Add support for deleting namespaces (users, organizations) 
Fixes #102
Fixes #105
 2016-10-21 15:41:09 -04:00
Joseph Schorr
864c44501e Fix global messages by removing "extra" method 
I think this happened due to a bad merge.
 2016-10-20 13:53:51 -04:00
josephschorr
67dde6e154 Merge pull request #1852 from coreos-inc/underscore_orgs 
Better handling of namespace validation to fix a number of issues
 2016-10-20 13:36:32 -04:00
Joseph Schorr
3a68740ff7 Better handling of namespace validation to fix a number of issues 
- Fixes a bug which allows for underscores at the beginning of namespaces: Fixes #1849
- Allows dots and dashes for newer Docker clients: Fixes #1188
- Has the UI display better messaging associated with namespace entry
 2016-10-20 13:32:22 -04:00
Joseph Schorr
2eabf1a291 Fix tests and test provider for real license format 2016-10-18 23:44:08 -04:00
Joseph Schorr
67f828279d Switch the license validator to use config_provider and have a test license 
Fixes the broken tests currently which try (and fail) to read the license file
 2016-10-18 11:44:13 -04:00
Joseph Schorr
7a6fb7554d Only attempt to load the license for the setup tool once there is a valid user 
Prevents the 401 session expired box from appearing
 2016-10-17 21:57:17 -04:00
Joseph Schorr
ee96693252 Add superuser config section for updating license 2016-10-17 21:44:25 -04:00
Jimmy Zelinskie
0c5400b7d1 enforce license across registry blueprints 2016-10-17 21:43:45 -04:00
Joseph Schorr
8fe29c5b89 Add license upload step to the setup flow 
Fixes #853
 2016-10-17 21:43:15 -04:00
Charlton Austin
f45aac063e Merge pull request #2005 from charltonaustin/fix_spacing_for_motd 
Moving the messages endpoint to something more generic, and making th…
 2016-10-17 17:21:03 -04:00
Charlton Austin
8e5dc8d3db Moving the messages endpoint to something more generic, and making the get visible all the time. 2016-10-17 16:23:48 -04:00
Joseph Schorr
3439f814b6 Fix quoting of scopes in WWW-Authenticate header 
Fixes part of #2002
 2016-10-17 14:32:43 -04:00
Joseph Schorr
18097a1bd6 Fix Link headers for pagination to match RFC 
Fixes part of #2002
 2016-10-17 13:57:05 -04:00
josephschorr
b4dd04cca4 Merge pull request #1996 from coreos-inc/aci-con-test-fix 
Add a sleep to fix ACI conversion tests
 2016-10-14 16:25:57 -04:00
Joseph Schorr
886489c666 Fix NPE raised if a vulnerability notification doesn't have a level filter 
Fixes #1990
 2016-10-14 14:23:50 -04:00
Joseph Schorr
c3ce491f02 Add a sleep to fix ACI conversion tests 
This ensures that the cache is cleared before the second request is made
 2016-10-14 13:36:47 -04:00
Charlton Austin
97d644d95d Adding in the delete api and the delete and create UI. 2016-10-13 10:40:52 -04:00
charltonaustin
5546fa6214 Adding in messages table 2016-10-11 10:55:12 -04:00
Charlton Austin
be916fb6ed Merge pull request #1966 from charltonaustin/j_code_review_comments 
Adding in security tests and docs.
 2016-10-11 09:50:47 -04:00
charltonaustin
5a4b702888 Adding in security tests and docs. 2016-10-11 09:30:37 -04:00
Jake Moshenko
7a3ee86e53 Merge pull request #1957 from jakedt/absolutecorruption 
Always use absolute URLs in Location headers.
 2016-10-10 18:25:29 -04:00
Jake Moshenko
df1f35e9f9 Always use absolute URLs in Location headers. 
This works around docker/docker#15048
 2016-10-10 16:30:24 -04:00
josephschorr
7fc33a9a57 Merge pull request #1965 from coreos-inc/condense-slack-notifications 
Less verbose notifications for QSS
 2016-10-10 15:38:12 -04:00
Joseph Schorr
ebf4120326 Less verbose notifications for QSS 
Fixes #1914
 2016-10-10 15:18:49 -04:00
charltonaustin
fa10d799b2 Adding in one more unit test. 2016-10-10 14:00:20 -04:00
charltonaustin
14eb3005b6 Some fixes for code review. 2016-10-10 12:55:00 -04:00
charltonaustin
1e733ddffb Adding in a new message data model and the corresponding methods to in the API. 2016-10-07 15:56:58 -04:00
charltonaustin
002f533bf8 Creating message api. 2016-10-07 10:22:30 -04:00
josephschorr
6b33503d8c Merge pull request #1939 from coreos-inc/unicode-search 
Handle unicode in entity search
 2016-10-04 22:19:59 +03:00
Joseph Schorr
ff0a292548 Handle unicode in entity search 
Fixes #1934
 2016-10-04 21:56:47 +03:00
josephschorr
768459ef86 Merge pull request #1938 from coreos-inc/v2-missing-parent-test 
Add parent mis-ordered registry test
 2016-10-04 21:53:13 +03:00
josephschorr
d8bef56e68 Merge pull request #1933 from coreos-inc/test-notifications-api 
Add a test for issuing test notifications
 2016-10-04 20:15:11 +03:00
Joseph Schorr
a046141708 Add parent mis-ordered registry test 2016-10-04 19:26:12 +03:00
Joseph Schorr
f4e1748bb7 Fix parent rewrite bug in schema1 manifest code and add a bunch more tests 
Before this change, if you ended up writing a middle layer whose parent is not in the database, the manifest would fail to rewrite. We now just lookup the parent image in the manifest given to us, ignoring whether it is in the database or not (as it doesn't actually matter if not present; it'll be created if necessary).
 2016-10-04 09:15:27 -04:00
Joseph Schorr
fdcedafe91 Add a test for issuing test notifications 2016-10-04 10:57:32 +03:00
Evan Cordell
42ebb0a6c3 Record metrics in a separate etcd record 2016-10-03 16:11:37 -04:00
Joseph Schorr
f72cb1d2ba Fix tags API pagination and add a test 2016-10-03 22:06:31 +03:00
Joseph Schorr
0b7bb6d6c6 Fix issue in V1 registry code with accessing locations under HEAD 
Fixes #1922
 2016-10-03 17:09:12 +03:00
Joseph Schorr
95b7b47501 Add a registry tests for numeric tags 2016-10-03 16:06:49 +03:00
Evan Cordell
68c5384473 Fixes prometheus start metric 2016-09-30 13:09:03 -04:00
josephschorr
2d9ce6dbe3 Merge pull request #1906 from coreos-inc/gitlab-bug 
Gitlab trigger payload bug fixes
 2016-09-30 17:52:49 +02:00
Joseph Schorr
f50bb8a1ce Add missing call to set_phase when a build doesn't start 
This change fixes the build manager ephemeral executor to tell the overall build server to call set_phase when a build never starts. Before this change, we'd properly adjust the queue item, but not the repo build row or the logs, which is why users just saw "Preparing Build Node", with no indicating the node failed to start.

Fixes #1904
 2016-09-30 14:54:49 +02:00
Joseph Schorr
26e8e241da Fix handling of Gitlab payloads with multiple commits 
Gitlab sends multiple commits in the order reversed from Github. As this only broke recently, I suspect that they may have changed the ordering. This change makes the code order-agnostic to hopefully remove the problem going forward.

Fixes #1900
 2016-09-30 12:14:32 +02:00
Joseph Schorr
c43173576a Fix Gitlab trigger payload bug when commits is empty 
Gitlab will occasionally send trigger payloads with an empty commit list (and a null checkout_ha) for branches that have been deleted. Properly handle that case.
 2016-09-30 12:03:08 +02:00
josephschorr
684ace3b5a Merge pull request #1761 from coreos-inc/nginx-direct-download 
Add feature flag to force all direct download URLs to be proxied
 2016-09-29 22:46:57 +02:00
Evan Cordell
832ee89923 Add duration metric collector decorator (#1885) 
Track time-to-start for builders
Track time-to-build for builders
Track ec2 builder fallbacks
Track build time
 2016-09-29 15:44:06 -04:00
Jimmy Zelinskie
31b77cf232 rename auth.auth to auth.process 
This fixes some ambiguity around imports.
 2016-09-29 15:24:57 -04:00
Joseph Schorr
6ae3faf7fc Add explicit config parameter to the JWT auth methods 2016-09-29 11:15:20 +02:00
Joseph Schorr
dd2e086a20 Add feature flag to force all direct download URLs to be proxied 
Fixes #1667
 2016-09-29 11:13:41 +02:00
Jimmy Zelinskie
fc7301be0d *: fix legacy imports 
This change reorganizes imports and renames the legacy flask extensions.
 2016-09-28 20:17:14 -04:00
Jimmy Zelinskie
ae16d24fd1 license: validate via key instance rather than PEM 2016-09-28 15:44:28 -04:00
Joseph Schorr
fac9d9fc5d Fix broken test after the recent 404 change 2016-09-27 17:14:56 +02:00
josephschorr
e1771abe58 Merge pull request #739 from coreos-inc/license 
Add license checking to Quay
 2016-09-27 16:52:08 +02:00
Joseph Schorr
476576bb70 Add license checking to Quay 
Based off of mjibson's changes

Fixes #499
 2016-09-27 10:31:34 +02:00
Joseph Schorr
3c8b87e086 Fix verbs in manifestlist 
All registry_tests now pass
 2016-09-26 14:49:58 -04:00
Jimmy Zelinskie
2e5a94bc0b create key server data interface 2016-09-26 14:49:23 -04:00
Joseph Schorr
db60df827d Implement V2 interfaces and remaining V1 interfaces 
Also adds some tests to registry tests for V1 stuff.
Note: All *registry* tests currently pass, but as verbs are not yet converted, the verb tests in registry_tests.py currently fail.
 2016-09-26 14:49:04 -04:00
Joseph Schorr
de990253bc Fix the build manager tests for recent change 2016-09-26 17:28:09 +02:00
Joseph Schorr
a5fef119c9 Add an end-to-end test for the notifications queue 2016-09-21 15:15:35 -04:00
Joseph Schorr
6fffc22b8a Fix build should_perform for empty JSON 2016-09-21 15:00:47 -04:00
Joseph Schorr
af79fde50d Fix build notifications 2016-09-21 14:37:23 -04:00
Joseph Schorr
03d4445a02 Add notification filtering for builds based on ref regex 
Fixes #1835
 2016-09-14 16:48:17 -04:00
Jake Moshenko
d56f570d3b Improve the imagetree test. 2016-09-07 13:25:19 -04:00
Jake Moshenko
cf83c9a16a Improve the garbage collection tests. 2016-09-07 13:25:19 -04:00
Jake Moshenko
584a5a7ddd Reduce database bandwidth by tracking gc candidate images. 2016-09-07 13:25:19 -04:00
Jake Moshenko
0815f6b6c4 Fix indentation for DB queries. 2016-09-07 10:48:58 -04:00
josephschorr
cd8b45e25b Merge pull request #1754 from coreos-inc/team-add-perms 
Better UI and permissions handling for robots and teams
 2016-09-06 17:21:19 -04:00
Joseph Schorr
1b7b3ea41d Make sure to filter starred repos to those visible to the user 
Fixes #1793
 2016-08-31 14:08:31 -04:00
Joseph Schorr
b4939a3cd0 Fix filtering of repos only visible to org admins 2016-08-31 13:51:53 -04:00
Joseph Schorr
1864196d8d TAR creation is not deterministic, so we can't test repush consistently 2016-08-29 16:38:12 -04:00
Joseph Schorr
357005e33f Raise a 409 if we try to insert a tag twice at the same time 
Also fixes handling of labels for existing manifests

Fixes #1775
 2016-08-29 16:03:35 -04:00
Joseph Schorr
1a2666be07 Fix deletion of labels and add tests 2016-08-26 16:07:49 -04:00
Joseph Schorr
608ffd9663 Basic labels support 
Adds basic labels support to the registry code (V2), and the API. Note that this does not yet add any UI related support.
 2016-08-26 15:24:26 -04:00
Joseph Schorr
391d70d9ec Add repo permissions dialog for existing teams and robots 
Fixes #1686
 2016-08-22 14:43:12 -04:00
josephschorr
2caa82d091 Merge pull request #1713 from coreos-inc/enable-iam 
Enable IAM support for S3 storage
 2016-08-16 16:13:29 -04:00
Joseph Schorr
742e153133 Fix watch of the jobs key in the build manager 2016-08-16 15:43:09 -04:00
Joseph Schorr
06718d1237 Fix S3 tests for IAM 2016-08-15 20:34:17 -04:00
josephschorr
de9be6e993 Merge pull request #1730 from coreos-inc/fix-pagination 
Fix pagination of repositories
 2016-08-15 17:14:32 -04:00
Joseph Schorr
d78361b041 Cleanup old executions that never start 
Fixes #1727
 2016-08-15 16:54:02 -04:00
Joseph Schorr
7f5b536ddb Fix pagination of repositories 
Fixes #1725
 2016-08-15 16:48:04 -04:00
Joseph Schorr
4f5b4e63f2 Really fix the hack (for now) on public repo pagination 2016-08-13 14:40:11 -04:00
josephschorr
d9b65b88e9 Merge pull request #1716 from coreos-inc/unicode-tags 
Add a test for unicode tags to ensure they cannot be set
 2016-08-11 18:34:44 -04:00
Joseph Schorr
19fb8fcf7c Add a test for unicode tags to ensure they cannot be set 
Fixes #1324
 2016-08-11 18:21:01 -04:00
Joseph Schorr
bab5cf69c7 Add a test for a deleted ref for the Github trigger 
Fixes #1047
 2016-08-11 18:01:04 -04:00
Joseph Schorr
4a2acac5dc Fix pagination of public repos, make more efficient and add test 2016-08-10 15:08:06 -04:00
Jimmy Zelinskie
22a25ac2d3 Revert "Merge pull request #1678 from coreos-inc/delete-repo-fix" 
This reverts commit df64caf133, reversing
changes made to 0d1e453566.
 2016-08-08 12:38:15 -04:00
josephschorr
df64caf133 Merge pull request #1678 from coreos-inc/delete-repo-fix 
Have repo deletion not lock all the things
 2016-08-04 16:48:03 -04:00
Joseph Schorr
c4daf1cc3d Change permissions model so that non-admins do not get org-wide read 
Fixes #1684
 2016-08-04 16:47:28 -04:00
Joseph Schorr
0b5cd95693 Have repo deletion not lock all the things 2016-08-04 16:45:59 -04:00
josephschorr
8bc0080aeb Merge pull request #1672 from coreos-inc/off-by-one 
Fix off-by-one error in repo tags pagination
 2016-08-03 15:00:23 -04:00
josephschorr
de58c1e38b Merge pull request #1661 from coreos-inc/buildman-timeout 
Fix TTL on heartbeat in etcd
 2016-08-03 11:18:32 -04:00
Joseph Schorr
c29f9ccc7f Fix TTL on heartbeat in etcd 
Until now, once the heartbeat has expired, we would issue a TTL that is negative, which causes etcd to either raise an exception or simply ignore the expiration (depending on the version of etcd). This change ensures that once the key is expired, it is removed immediately via a set of a TTL of 0. Also adds tests for this case and the normal expiration case.
 2016-08-03 11:15:03 -04:00
Joseph Schorr
b1b0da7afd Fix off-by-one error in repo tags pagination 
Fixes #1665
 2016-08-02 14:17:33 -04:00
Jake Moshenko
8ac88facab Add a test to make sure the random policy function runs. 2016-08-01 18:42:55 -04:00
Jake Moshenko
05e2773fa7 Get rid of remaining slow query for garbage collection. 2016-08-01 18:22:38 -04:00
josephschorr
b0bffe56ca Merge pull request #1638 from coreos-inc/swift-retry-seek 
Add retry support to Swift
 2016-08-01 14:04:54 -04:00
josephschorr
46a28617e8 Merge pull request #1651 from coreos-inc/fix-branches 
Fix handling of multi-part branches in the build triggers
 2016-07-26 16:00:21 -07:00
josephschorr
0162d3da30 Merge pull request #1645 from coreos-inc/gc-query-optimize 
Optimize GC query for looking up deletable storages
 2016-07-26 16:00:17 -07:00
Joseph Schorr
9e4f8cac03 Optimize GC query for looking up deletable storages 2016-07-26 13:47:15 -07:00
Joseph Schorr
06d52f2c83 Fix handling of multi-part branches in the build triggers 
Fixes #1360
 2016-07-26 13:41:13 -07:00
Joseph Schorr
a41ccf0356 Add retry support to Swift 
Fixes #1636
 2016-07-26 09:56:00 -07:00
Joseph Schorr
0fe3e6510a Prevent invalid tags on builds 
Fixes #1632
 2016-07-25 17:50:35 -07:00
Joseph Schorr
5de1e98d3c Fix LDAP DN building for empty RDN list 2016-07-22 14:40:53 -04:00
Joseph Schorr
392242d20b Another fix for the record keeping in buildman 
Adds some more mocked tests as well
 2016-07-22 12:01:30 -04:00
josephschorr
cf630838f0 Merge pull request #1624 from coreos-inc/builder-cleanup-tests 
Bug fixes, refactoring and "new" tests for the build manager
 2016-07-21 13:50:41 -04:00
Jimmy Zelinskie
2ed5723ca9 test_secscan: add a second before reads from queue 
Because of the granularity of MySQL's clock, we need to wait a second
before an item becomes available.
 2016-07-18 14:19:36 -04:00
Joseph Schorr
2c1880b944 Bug fixes, refactoring and "new" tests for the build manager 
- Fixes various bugs introduced in the most recent build system commit
- Refactors state management in the build manager to be cleaner and more contained
- Adds back in the mock-based tests, fixed to not use threads and adjusted for the refactoring
- Adds some more simplified unit tests around non-etch related flows
 2016-07-18 13:46:48 -04:00
Joseph Schorr
b0b7b63be9 Fix queue tests for MySQL 
MySQL's date time's appear to have a 1 second threshold, so we need to make sure the queue items added for the tests are available as soon as they are added. Before this change, the available_after was set to `datetime.utcnow()`, and, if the `get` was called within 1 second, then its check would fail.
 2016-07-15 13:27:50 -04:00
Joseph Schorr
1ed1bc9ed3 Disable prometheus in tests 2016-07-14 15:48:21 -04:00
Joseph Schorr
74b87fa813 Build manager cleanup and more logging 2016-07-14 14:33:14 -04:00
Joseph Schorr
3d558f6090 Disable ACI tests when build under Docker 
We need to figure out why they fail on our build fleet
 2016-07-13 14:23:30 -04:00
josephschorr
a69266c282 Merge pull request #1605 from coreos-inc/kubernetes-builder 
Kubernetes builder
 2016-07-12 14:49:10 -04:00
josephschorr
3143da6392 Merge pull request #1608 from coreos-inc/storage-rep 
Fix storage replication for CAS and add tests
 2016-07-12 13:56:36 -04:00
Joseph Schorr
5cd793331e Fix storage replication for CAS and add tests 2016-07-12 13:46:06 -04:00
Joseph Schorr
c1e4bf79b7 Fix delete team error message for admin teams 2016-07-11 15:47:05 -04:00
Joseph Schorr
811413fe9c Add multiple executor and whitelist support to build manager 2016-07-08 15:50:51 -04:00
Joseph Schorr
6bdbe25cdc Cleanup unused imports 2016-07-08 15:50:51 -04:00
Colin Hom
bc13333f20 Kubernetes build worker 2016-07-08 15:50:51 -04:00
Joseph Schorr
adaeeba5d0 Allow for multiple user RDNs in LDAP 
Fixes #1600
 2016-07-07 14:46:38 -04:00
Joseph Schorr
e252ee07cb Fix popularity metrics on list repos API 2016-07-06 16:15:54 -04:00
Joseph Schorr
1eec6f53b2 Fix SQL error with pagination around Repositories 
Fixes #1591
 2016-06-30 17:31:35 -04:00
Joseph Schorr
9558c0e937 Fix handling of Github API paths and add tests 2016-06-30 14:10:22 -04:00
Joseph Schorr
2f771304fa Disable flaky mock-based tests 2016-06-24 16:04:34 -04:00
Joseph Schorr
30ede029d5 Fix GeneratorFile for working with BufferedReader 
The user files system uses a BufferedReader along with the magic library to determine the mime type of the user file being served. Currently, BufferedReader fails with an exception on Swift storage, because Swift storage returns a GeneratorFile, which is missing the `readable()` method.
 2016-06-23 13:40:57 -04:00
josephschorr
9e6a264f5f Merge pull request #1523 from coreos-inc/verb-tag-cache-fix 
Add a uniqueness hash to derived image storage to break caching over …
 2016-06-20 16:38:25 -04:00
Joseph Schorr
a43b741f1b Add a uniqueness hash to derived image storage to break caching over tags 
This allows converted ACIs and squashed images to be unique based on the specified tag.

Fixes #92
 2016-06-20 16:34:52 -04:00
Jake Moshenko
a1cf12e460 Add a sitemap.txt for popular public repos 
and reference it from the robots.txt
 2016-06-17 14:34:20 -04:00
josephschorr
614b9124ae Merge pull request #1512 from coreos-inc/optimize-queries 
Optimize various queries
 2016-06-16 14:22:59 -04:00
Joseph Schorr
fea47bdaed Increase test counter to 10 2016-06-13 17:31:42 -04:00
josephschorr
58bef472d9 Merge pull request #1526 from coreos-inc/superuser-grant 
Add ability for super users to take ownership of namespaces
 2016-06-13 16:23:10 -04:00
Joseph Schorr
20816804e5 Add ability for super users to take ownership of namespaces 
Fixes #1395
 2016-06-13 16:22:52 -04:00
josephschorr
9263aad2ac Merge pull request #1534 from coreos-inc/flakey-test 
Fix flaky tests
 2016-06-13 16:12:15 -04:00
Joseph Schorr
4747dea395 Fix flaky tests 2016-06-13 16:00:55 -04:00
josephschorr
92f0db8056 Merge pull request #1514 from coreos-inc/better-logs 
Only send heavy log-based stats for repository where required
 2016-06-09 14:52:30 -04:00
Joseph Schorr
c3701cea7a Only send heavy log-based stats for repository where required 2016-06-09 14:52:15 -04:00
Joseph Schorr
8887f09ba8 Use the instance service key for registry JWT signing 2016-06-07 11:58:10 -04:00
Joseph Schorr
53538f9001 Optimize get_tag_image query 
No caller uses the image placements or locations, so no need to load them.
 2016-06-02 16:36:38 -04:00
josephschorr
cad8746f9d Merge pull request #1502 from coreos-inc/image-replication 
Enable storage replication for V2 and add backfill tool
 2016-06-02 15:02:53 -04:00
Joseph Schorr
12924784ce Enable storage replication for V2 and add backfill tool 
Fixes #1501
 2016-06-02 14:36:08 -04:00
josephschorr
ec492bb683 Merge pull request #1323 from coreos-inc/secworkerreturn 
Move security notification work into its own method to allow for retu…
 2016-06-02 13:59:25 -04:00
Joseph Schorr
48213f9ff9 Reject manifest 2 earlier to make pushes faster 2016-06-02 12:46:20 -04:00
Jimmy Zelinskie
2317938bfa Merge pull request #1496 from jzelinskie/ripRMS 
dockerfile: add check for GPL pip packages
 2016-06-02 12:28:18 -04:00
Jimmy Zelinskie
e5241c6d88 tests: simple test for BuildRequest w/ archive URL 2016-06-02 12:27:49 -04:00
Joseph Schorr
a18c4dd210 Make exponential back off test try multiple times 
Slower runtime environments require multiple calls before we hit the 429
 2016-06-01 15:00:10 -04:00
josephschorr
a85c3ebff7 Merge pull request #1457 from coreos-inc/xauth 
Add support for direct granting of OAuth tokens and add tests
 2016-06-01 12:07:12 -04:00
Jimmy Zelinskie
6178371cf5 Merge pull request #1493 from jzelinskie/noorder 
queue: explicitly declare ordering requirement
 2016-05-31 15:46:39 -04:00