Commit graph

766 commits

Author SHA1 Message Date
Joseph Schorr
29058201e5 Fix bug in modelutil pagination that caused us to load far more results than necessary
Also adds tests for the modelutil pagination
2018-05-14 11:40:31 -04:00
Joseph Schorr
f06eec8a35 Fix issue where we didn't delete robots immediately under a namespace
This could result in "hanging" robot accounts, although that would only leak the names of said accounts. Now we delete them immediately AND we proactively delete them before replacing the namespace (just to be sure)
2018-05-09 17:53:30 +03:00
Joseph Schorr
3309daa32e Add support for reduced initial build count for new possible abusing users
If configured, we now check the IP address of the user signing up and, if they are a possible threat, we further reduce their number of allowed maximum builds to the configured value.
2018-04-20 18:46:32 +03:00
Brad Ison
8e5c67e207
Use get_or_create in place of create_or_get
Peewee v2.9.0 removes the create_or_get method.
2018-04-04 14:21:16 -04:00
Joseph Schorr
82da2042fd Add tests for image model, fix a small bug and remove an unused function 2018-04-04 13:13:52 -04:00
Joseph Schorr
8146646761 Simplifying queries around images and placements
Only verbs needs to load placements for multiple images, so we can vastly simplify and optimize most queries by making it two-step, and having the rest of the image loads not worry about placements
2018-04-03 16:23:49 -04:00
Joseph Schorr
f6b647bd61 Default for robot account metadata should be an empty dict, not a string 2018-03-27 17:29:55 -04:00
josephschorr
6c43b7ff0d
Merge pull request #3024 from coreos-inc/manageable-robots
Manageable robots epic
2018-03-21 18:50:17 -04:00
Joseph Schorr
2ea13e86a0 Add last_accessed information to User and expose for robot accounts
Fixes https://jira.coreos.com/browse/QUAY-848
2018-03-21 15:28:34 -04:00
Joseph Schorr
c4debe011c Fix team add/invite logic around when an invite is required
We were accidentally skipping the invite if the user was a member of *any* organization, rather than the specific organization (as intended)

Fixes https://jira.coreos.com/browse/QUAY-880
2018-03-16 17:13:27 -04:00
Joseph Schorr
254cdfe43a Add support for metadata on robot accounts
Fixes https://jira.coreos.com/browse/QUAY-847
Fixes https://jira.coreos.com/browse/QUAY-816
2018-03-12 20:32:05 -04:00
Joseph Schorr
a693771345 Add creation date information to robots API and UI
Fixes https://jira.coreos.com/browse/QUAY-846
2018-03-09 13:55:19 -05:00
Joseph Schorr
ed84835d23 Changes for code review 2018-03-01 16:49:52 -05:00
Joseph Schorr
3837a7c2ed Add datetime of when a trigger was disabled 2018-03-01 16:49:51 -05:00
Joseph Schorr
93d79e777e Automatically disable build triggers with successive failures or internal errors
We allow users to reenable them manually once disabled
2018-03-01 16:49:51 -05:00
Joseph Schorr
c35eec0615 Add ability for triggers to be disabled
Will be used in the followup commit to automatically disable broken triggers
2018-03-01 16:49:28 -05:00
Joseph Schorr
8bc55a5676 Make namespace deletion asynchronous
Instead of deleting a namespace synchronously as before, we now mark the namespace for deletion, disable it, and rename it. A worker then comes along and deletes the namespace in the background. This results in a *significantly* better user experience, as the namespace deletion operation now "completes" in under a second, where before it could take 10s of minutes at the worse.

Fixes https://jira.coreos.com/browse/QUAY-838
2018-02-27 13:12:51 -05:00
josephschorr
d77aa9228f
Merge pull request #3002 from coreos-inc/joseph.schorr/QUAY-822/gc-app-tokens
Add a worker to automatically GC expired app specific tokens
2018-02-20 17:21:48 -05:00
Joseph Schorr
9a452ace11 Add configurable limits for number of builds allowed under a namespace
We also support that limit being increased automatically once a successful billing charge has gone through
2018-02-20 16:54:22 -05:00
Brad Ison
62971b7f20
Merge pull request #2999 from bison/user-location
Add user location metadata filed
2018-02-20 16:48:37 -05:00
Joseph Schorr
e220b50543 Refactor auth code to be cleaner and more extensible
We move all the auth handling, serialization and deserialization into a new AuthContext interface, and then standardize a registration model for handling of specific auth context types (user, robot, token, etc).
2018-02-14 15:35:27 -05:00
Joseph Schorr
e1a3857de9 Make sure to filtered already-expired tokens from the "expiring soon" tokens 2018-02-12 17:07:50 -05:00
Joseph Schorr
d45161b120 Add a worker to automatically GC expired app specific tokens
Fixes https://jira.coreos.com/browse/QUAY-822
2018-02-12 14:56:01 -05:00
Brad Ison
3de6b4a646 Add location metadata field for users 2018-02-06 16:06:17 -05:00
Joseph Schorr
da0fa2e0d2 Make sure to add primary repo permissions under a transaction
Should prevent a repository from being created under a user's namespace without a corresponding admin permission

Fixes https://jira.coreos.com/browse/QUAY-826
2018-02-06 11:20:40 -05:00
Joseph Schorr
6228ab5a51 We cannot cache the full expiration for an app specific token, as it would include the datetime when the cache is created, rather than now
Fixes https://jira.coreos.com/browse/QUAY-819
2018-02-02 16:11:59 -05:00
Joseph Schorr
bbdf9e074c Add metrics for tracking when instance key renewal succeeds and fails, as well as when instance key *lookup* fails 2018-02-02 11:14:42 -05:00
josephschorr
fd1237cff9
Merge pull request #2974 from coreos-inc/joseph.schorr/QS-118/manifest-write-query
Audit the number of SQL queries we make in writing manifests, and significantly reduce in the common case
2018-01-31 11:08:33 -05:00
Joseph Schorr
9e16a989f5 Audit the number of SQL queries we make in writing manifests, and significantly reduce in the common case
Instead of 41 queries now for the simple manifest, we are down to 14.

The biggest changes:
  - Only synthesize the V1 image rows if we haven't already found them in the database
  - Thread the repository object through to the other model method calls, and use it instead of loading again and again
2018-01-25 11:10:43 -05:00
Joseph Schorr
208dc38d25 Allow expired app specific tokens to be deleted 2018-01-23 11:40:51 -05:00
josephschorr
b17c05fbd5
Merge pull request #2968 from coreos-inc/joseph.schorr/QS-110/user-login-lock
Reduce lock contention on invalid user login calls
2018-01-09 12:41:00 -05:00
Joseph Schorr
81be47486c Reduce lock contention on invalid user login calls
Fixes https://jira.coreos.com/browse/QS-110
2018-01-08 16:57:50 -05:00
josephschorr
13b738c43c
Merge pull request #2954 from coreos-inc/joseph.schorr/QS-102/user-api-filter
Add ability to filter users list to enabled users
2018-01-05 15:40:50 -05:00
josephschorr
d8fde005d8
Merge pull request #2961 from coreos-inc/joseph.schorr/QS-107/create-repo-opt
Small optimizations around create repository code
2018-01-05 15:40:30 -05:00
Joseph Schorr
888b564a9b Add a banner to the Quay UI when an app specific token is about to expire 2018-01-04 15:27:42 -05:00
Joseph Schorr
524d77f527 Add an AppSpecificAuthToken data model for app-specific auth tokens. These will be used for the Docker CLI in place of username+password 2018-01-04 15:27:41 -05:00
Joseph Schorr
3f99e864e1 Reduce the number of DB queries for creating a repo 2018-01-04 14:32:39 -05:00
Joseph Schorr
1e1bec0afe Remove extra update call on create repo 2018-01-04 13:42:05 -05:00
Joseph Schorr
8e473b9779 Add filter for disabled users to superuser user list API
Fixes https://jira.coreos.com/browse/QS-102
2017-12-22 16:45:49 -05:00
Joseph Schorr
60bc655695 Fix flakiness in a test when comparing date times 2017-12-14 14:00:20 -05:00
Joseph Schorr
51e67ab7f5 Fix get_blob_path to not make any database calls and add a test
This will be supported by caching, hopefully removing the need to hit the database when the blob object is cached
2017-12-13 16:27:46 -05:00
josephschorr
b2db266747
Merge pull request #2935 from coreos-inc/joseph.schorr/QS-80/password-reset-expire
Add maximum lifetime of 30m on password recovery tokens
2017-12-07 14:21:32 -05:00
Joseph Schorr
1d1c6f0606 Invalidate all session tokens when a user signs out
Fixes https://jira.coreos.com/browse/QS-85
2017-12-07 13:03:11 -05:00
Joseph Schorr
5dd95038cf Add maximum lifetime of 30m on password recovery tokens
Fixes https://jira.coreos.com/browse/QS-80
2017-12-06 17:06:03 -05:00
Joseph Schorr
32255f122b Simplify and further optimize handling of unfiltered search results
Using the DB-side limit is much faster
2017-11-30 16:56:01 -05:00
Joseph Schorr
eea026be52 Fix bug around search pagination with non-filtered searches
Also further optimizes the queries
2017-11-30 16:13:42 -05:00
Joseph Schorr
0440cca3ef Fix typo in how we order search results
`.desc()` needs to be on the field
2017-11-30 14:53:23 -05:00
Joseph Schorr
c767c88b82 Optimize searching of repositories when there is no query 2017-11-30 14:10:22 -05:00
Joseph Schorr
2ced523313 Add Explore tab and query-less searching
Allows for exploration of all visible repositories, in paginated form.

This change also fixes the layout of the header on different viewport sizes to be consistently a single line in height.

Fixes https://jira.coreos.com/browse/QS-63
2017-11-28 16:50:23 +02:00
Joseph Schorr
2e28976d7c Change exception to debug on expected RAC integrity error
Fixes https://jira.prod.coreos.systems/browse/QS-44
2017-10-30 12:55:24 -04:00
Joseph Schorr
9f804de23d Fix bug in deletion of repos with OCI-style linked tags
MySQL does not allow rows in the same table referencing other rows to be deleted in a single statement. We now do a two-pass deletion, and add a test to make sure.

Fixes https://jira.prod.coreos.systems/browse/QS-18
2017-10-18 17:03:27 -04:00
Joseph Schorr
c1ce84822d Fix tuple for health check 2017-10-12 16:25:31 -04:00
Joseph Schorr
c6aad5fef0 Add option to disable partial autocompletion of users 2017-09-12 15:55:37 -04:00
Charlton Austin
6c29ec873a refactor(endpoints/api/superuser*): refactored code behind db model
this moves all the db model code behind an interface in prep for v2-2

Issue: https://coreosdev.atlassian.net/browse/QUAY-750

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-08-01 11:27:55 -04:00
Joseph Schorr
dfe371286a Optimize purging of a repository by skipping the unreferenced check 2017-07-28 13:41:14 -04:00
Jake Moshenko
0011ddda16 Release database connections after updating build statuses. 2017-07-26 12:29:51 -04:00
Charlton Austin
be206a8b88 Merge pull request #2814 from charltonaustin/create_data_interface_for_subsystem_api/repository_633
refactor(endpoints/api/repository*): added in pre_oci_model abstraction
2017-07-25 13:59:52 -04:00
Charlton Austin
8f1200b00d style(data, endpoints, test): ran yapf against changed files
### Description of Changes

Issue: https://coreosdev.atlassian.net/browse/QUAY-633

## Reviewer Checklist

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-07-24 11:05:15 -04:00
Joseph Schorr
aecec02b6c Change common_login to take in a user uuid, instead of a user DB object 2017-07-20 16:01:39 -04:00
Joseph Schorr
7d4fed6892 Change error message when trying to pull a deleted or expired tag
Will let the users know they can recover the tag via time machine

Note: This was tested with the Docker protocol, but the new error code is *technically* out of spec; we should make sure its okay.
2017-07-19 17:13:48 -04:00
Joseph Schorr
99d7fde8ee Add UI for viewing and changing the expiration of tags 2017-07-19 17:13:48 -04:00
Joseph Schorr
4663bf4194 Add additional test for tag expiration 2017-07-19 17:13:06 -04:00
Joseph Schorr
4ad3682b9c Make health check failures report their reasons
Note that we add a new block with expanded service info, to avoid breaking compatibility with existing callers of the health endpoint
2017-07-19 16:17:02 +03:00
josephschorr
fdb21aa5dc Merge pull request #2777 from coreos-inc/joseph.schorr/QUAY-618/notificationworker-data-interface
Change notificationworker to use data interface
2017-07-13 00:23:15 +03:00
josephschorr
d00c7d299c Merge pull request #2778 from coreos-inc/gc-long-chain-test
Add a long-chain test for GC
2017-07-13 00:23:03 +03:00
Charlton Austin
82488c9102 Merge pull request #2753 from charltonaustin/api_tag_v22
refactor(endpoints/api/tag): refactor code for v22
2017-07-12 15:51:48 -04:00
Charlton Austin
131acde317 refactor(data+endpoints): code review changes
this puts the view logic on the object and adds a parameter for logging

[TESTING->locally with docker compose]

Issue: https://coreosdev.atlassian.net/browse/QUAY-632

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-07-12 14:10:21 -04:00
Joseph Schorr
2b51265b48 Add a long-chain test for GC 2017-07-12 18:03:27 +03:00
Joseph Schorr
b6f1782642 Change notificationworker to use a data interface 2017-07-12 17:40:45 +03:00
Joseph Schorr
bdab367285 Change blobuploadcleanupworker to use a data interface 2017-07-11 16:58:09 +03:00
Charlton Austin
897a091692 style(data+endpoints): ran yapf
### Description of Changes

ran yapf for the branch

[TESTING->locally using docker compose]

Issue: https://coreosdev.atlassian.net/browse/QUAY-632

## Reviewer Checklist

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-07-10 09:46:02 -04:00
Charlton Austin
fc4b3642d3 refactor(endpoints/api/tag): refactor code for v22
this decouples the database models from the api

[TESTING->locally with docker compose]

Issue: https://coreosdev.atlassian.net/browse/QUAY-632

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-07-10 09:40:33 -04:00
EvB
ccca0c9655 refactor(util/tufmetadata/test): move app test to gc suite 2017-07-07 15:14:14 -04:00
josephschorr
89ab94ffbb Merge pull request #2720 from coreos-inc/gc-opt
GC optimizations
2017-06-29 20:55:14 +03:00
Joseph Schorr
8dcea30d58 Fix build by pre-calling the caches
They were being called in a test-dependent order, which caused any tests which relied on query count to fail
2017-06-27 18:11:46 +03:00
Joseph Schorr
cdd7cb9321 Remove directly referenced images from the candidate set before starting GC iteration
Makes the lookup query underneath the transaction smaller if there are a lot of images referenced directly by tag. We still must do the direct referenced check within the transaction, but this should reduce the scope of the search space a bit.
2017-06-27 17:33:58 +03:00
Joseph Schorr
45c7008078 Change Repo GC to be iterative
This prevents us from creating a massive join when there are a large number of tags in the repository, which can result in locking the entire DB for long periods of time. Instead of the join, we just iteratively lookup any images found to be directly referenced by a tag or found as a parent of another image, both of which should be indexed lookups. Once done, we only remove those images and then iterate until the working set stops changing.
2017-06-27 17:33:58 +03:00
Joseph Schorr
e9a95874ee Move GC tests into pytest 2017-06-27 17:33:58 +03:00
Joseph Schorr
11659f73bf Fix log for reenabling a notification
We forgot to log the event and method names
2017-06-27 15:26:40 +03:00
Charlton Austin
5e6aa6648b fix(endpoints): added in proper error handling
before we would return a 400 without a message because the errors were not being caught

Issue: https://www.pivotaltracker.com/story/show/145459707

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-06-21 16:59:48 -04:00
josephschorr
b269cb84a3 Merge pull request #2688 from coreos-inc/build-badge-query
Remove expensive call in build badge
2017-06-12 15:06:13 -04:00
Joseph Schorr
a949a44cb2 Remove expensive call in build badge
We don't need to list all the tags to determine if any exist, and showing the repo is ready when it is empty is probably correct behavior anyway
2017-06-12 14:53:54 -04:00
Joseph Schorr
d68b65d90c Optimize GC query join a bit by reducing the surface
We remove the directly referenced images from the join across ancestors, as they will be covered by the first part of the union clause. For some large repositories, this will result in a significantly reduced set of images that have to be joined NxM.
2017-06-12 12:55:32 -04:00
Charlton Austin
993f2a174c feat(full-stack): disable notifications after 3 failures
This stops notifications from firing over and over again if they are repeatedly failing.

[TESTING -> locally with docker compose, DATABASE MIGRATION -> there is a single migration]

Issue: https://www.pivotaltracker.com/story/show/b144646649n

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-05-19 16:58:46 -04:00
Joseph Schorr
db767b3610 Optimize lookup of org membership on prototype and perms APIs
Fixes a major slowdown when working with permissions under organizations with a lot of members

Fixes https://www.pivotaltracker.com/story/show/144076113
2017-05-08 14:03:59 -04:00
Joseph Schorr
9d97e053b3 Make sure to re-sort the filtered repositories in search
The filtering breaks the ordered we expected, so we need to re-sort
2017-05-03 18:38:46 -04:00
Joseph Schorr
0164b48a24 Switch repository search to order matches in repo names higher
Helps push better results to the top of the results list
2017-05-03 17:02:24 -04:00
josephschorr
19f67bfa1b Merge pull request #2607 from coreos-inc/faster-security-notify
Batch the tag lookups in the security notification worker in an attempt to significant reduce load
2017-05-03 13:49:13 -04:00
Joseph Schorr
68d473bff4 Clarify comment and make pairs code a little nicer 2017-05-03 11:41:39 -04:00
Joseph Schorr
977bbc20a2 Add filtering onto the images query in get_matching_tags_for_images
Should make the query even faster in the security notification case
2017-05-02 18:29:14 -04:00
Joseph Schorr
d77463cc8b Add sharing to the images lookup in get_matching_tags_for_images
Should prevent an issue if we ever get a request with thousands and thousands of images where we would exceed the DB's max packet size
2017-05-02 15:50:31 -04:00
Joseph Schorr
74dd0ef8e8 Add a batch get_matching_tags_for_images method
This will be used in the security notification worker to retrieving the tags needed in a set of batch calls, rather than multiple calls per image
2017-05-02 15:38:25 -04:00
Joseph Schorr
e583be3914 Remove inner query for ancestors lookup on get_matching_tags 2017-04-28 20:10:54 -04:00
Joseph Schorr
8b2e4d3bcf Add a test for get_matching_tags 2017-04-28 19:57:24 -04:00
Joseph Schorr
e9ffe0e27b Implement new search UI
We now have both autocomplete-based searching for quick results, as well as a full search page for a full listing of results
2017-04-28 13:57:28 -04:00
josephschorr
8b148bf1d4 Merge pull request #2576 from coreos-inc/full-db-tests-tox
Reenable full database testing locally and in concourse
2017-04-27 18:09:15 -04:00
Joseph Schorr
d7f3ef96ce Small fixes found by running full db tests 2017-04-24 16:45:15 -04:00
Joseph Schorr
7debd44b54 Switch fixture imports to wildcard in prep for full db test fixes 2017-04-24 16:45:14 -04:00
Jake Moshenko
a159bd3e77 Resolve race condition between multiple log archivers 2017-04-24 13:41:08 -04:00
Joseph Schorr
e46e668cc5 Fix search in postgres
Stupid missing group_by again
2017-04-21 17:21:17 -04:00