Commit graph

1525 commits

Author SHA1 Message Date
Joseph Schorr
89582438cd Fix the V22 phase 1 migrations to use new tables for mapping rather than editing existing tables
The ALTER TABLE operations previously used were causing the DB to die when run on the production TagManifest table which has 7 million rows. We instead now use new mapping tables, which is less nice, but these are temporary anyway, so hopefully we only have to deal with their ugliness for a short duration.
2018-08-06 16:58:27 -04:00
Joseph Schorr
7068010751
Merge pull request #3186 from quay/joseph.schorr/QUAY-1013-1015/manifest-migration
First phase of V2_2 data model migrations
2018-08-06 13:56:14 -04:00
Joseph Schorr
a46660a06f Add new Manifest, ManifestLabel, ManifestLegacyImage and ManifestBlob tables and start writing and GCing to/from them
This change also starts passing in the manifest interface, rather than the raw data, to the model for writing.

Note that this change does *not* backfill the existing rows in to the new tables; that will occur in a followup PR. The new columns in `tagmanifest` and `tagmanifestlabel` will be used to track the backfill, as it will occur in a worker.
2018-08-06 13:01:11 -04:00
Sida Chen
6efe262217
Merge pull request #3188 from KeyboardNerd/sidchen/notification_datetime_bug
Fixes empty "created" field in notification
2018-08-06 12:58:38 -04:00
Sida Chen
415631153a Fixes empty "created" field in notification
Fixes https://jira.coreos.com/browse/QUAY-1022
2018-08-01 17:17:47 -04:00
Joseph Schorr
7325b22c90 Change id column in LogEntry to a BigInt and migrate back to using that table 2018-07-30 12:42:56 -04:00
Joseph Schorr
8e425ee559 Fix for merge 2018-07-23 12:43:23 -04:00
Joseph Schorr
9a40e99a8f Enhancements for validation of DB urls
We now turn off retries and DB pooling, and make sure to always close the connection
2018-07-23 12:43:23 -04:00
Joseph Schorr
d15dcae505 Reenable retry wrapper on database operations 2018-07-23 12:43:23 -04:00
Brad Ison
d3d9cca182 Upgrade Peewee to latest 3.x
This requires a number of small changes in the data model code, as well as additional testing.
2018-07-23 12:43:23 -04:00
Joseph Schorr
487edf0ba1 Phase 3 of Appr migration
Deletes the old models and their code
2018-07-21 15:43:00 -04:00
Joseph Schorr
acb7d626a4 Phase 2 of Appr migration
Backfills the new tables from the old ones, and switches all code to read and write from the new tables
2018-07-21 15:25:04 -04:00
Joseph Schorr
113bb96f29 Phase 1 of migrating APPR-specific tables to tables with the Appr prefix
Fixes https://jira.coreos.com/browse/QUAY-950
2018-07-20 13:19:01 -04:00
Joseph Schorr
6622f27c93 Rename oci_model to appr_model 2018-07-20 12:46:19 -04:00
Joseph Schorr
3f3a0946f6 Move repo visibility tests into pytest 2018-07-19 11:23:11 -04:00
Joseph Schorr
f84cb05792
Merge pull request #3164 from quay/move-queue-tests
Move queue tests to pytest
2018-07-18 17:27:38 -04:00
Joseph Schorr
2d7b4dd1bf Move public repo test to pytest 2018-07-18 17:22:05 -04:00
Joseph Schorr
f648b3311b Move queue tests to pytest 2018-07-18 15:22:38 -04:00
Joseph Schorr
928a5d604a Move build query tests to pytest 2018-07-18 11:26:18 -04:00
Sam Chow
f5a8116f5a Remove password confirmation in config app
Small fix to manually clean up temp dir when creating new temp dir,
small fix to font awesome icons, change the jwt/keystone
validators to not use username/password
2018-07-17 16:00:55 -04:00
Sam Chow
496d94138c
Merge pull request #3139 from quay/spike/install-certs
Install certs in the config app, small refactor to LDAP validation
2018-07-16 12:50:36 -04:00
Joseph Schorr
eb8e325f94 Move image sharing model tests to pytest 2018-07-16 09:58:21 -04:00
Joseph Schorr
dda083abb1 Move blob model tests to pytest 2018-07-13 13:19:46 +03:00
Sam Chow
9024419896 Modify ldap validator to just check user existence
Remove auth user check from updating config app config

remove duplicate certs install script
2018-07-11 16:49:13 -04:00
Joseph Schorr
924dda296f Fully migrate API security tests into the pytest test suite
Also adds an additional test that ensures that at least one security test exists for every (api endpoint, http method) pair.
2018-07-08 18:33:21 +03:00
Brad Ison
73cb7f3228
endpoints/api: Allow null fields in user metadata
The user metadata fields are nullable in the database, but were not in
the json sechema.  This prevented users from updating some of their
information on the site if they hadn't set the metadata fields.
2018-06-27 15:34:55 -04:00
Sam Chow
872be8605a Fix error case in uploading tar, more comments 2018-06-22 13:23:08 -04:00
Sam Chow
aff1a08a83 Fix main app migration pathway 2018-06-21 15:33:26 -04:00
Sam Chow
b5f630ba29 Fix alembic migrations importing app
Ensure we connect to loaded config db
2018-06-20 17:17:35 -04:00
Joseph Schorr
371f6f8946
Merge pull request #3118 from quay/joseph.schorr/QUAY-977/catalog-efficiency
Catalog efficiency improvements
2018-06-20 16:40:37 -04:00
Joseph Schorr
2caaf84f31 Add caching support to catalog
We will now cache the results of the catalog for 60s and not hit the database at all if cached
2018-06-20 14:58:01 -04:00
Joseph Schorr
7604e9842b Change repo filtering for users to use a user ID reference, rather than the username
While this means we need an additional query for initial lookup, it makes the *filtering* query (which is the heavy part) require far fewer joins, thus making it more efficient.

Also adds a new unit test to verify that our filter filters to the correct set of repositories.
2018-06-19 10:51:30 -04:00
Joseph Schorr
f6ff0d6ca0 Add support for populating test data during migration testing
This change ensures that the tables in the database during migration have at least one row of "real" data, which should help catch issues in the future where we forget to set column defaults and other such schema oversights that can only be truly tested with non-empty tables

Fixes https://jira.coreos.com/browse/QUAY-913
2018-06-18 14:14:32 -04:00
Joseph Schorr
0fa1a1d5fd Implement support for blob mounting via the mount parameter on blob uploads
Fixes https://jira.coreos.com/browse/QUAY-893
2018-06-06 11:31:28 -04:00
Joseph Schorr
b2262eaf46 Add feature flag to disable username confirmation
Fixes https://jira.coreos.com/browse/QUAY-914
2018-06-01 13:30:50 -04:00
Joseph Schorr
8ee98c1c48 Show starred public repositories in the starred repos list
We forgot to check if the repository was public in the filter

Fixes https://jira.coreos.com/browse/QUAY-857
2018-05-31 17:34:17 -04:00
Joseph Schorr
6ffafe44d3
Merge pull request #3059 from quay/joseph.schorr/QUAY-906/reg-tests
Move registry integration tests to py.test
2018-05-22 17:09:11 -04:00
Joseph Schorr
861e81cccd Allow team syncing if user creation is disabled
Before this change, if user creation was disabled, team sync would fail to sync over users that had not yet been invited/logged in, because their accounts could not be created. Following this change, team syncing of users not yet in the system will create those user accounts, allowing users to be "auto invited" via team sync.

Fixes https://jira.coreos.com/browse/QUAY-910
2018-05-22 14:09:40 -04:00
Joseph Schorr
a007332d4c Temporarily change to storing logs in a new LogEntry2 table
This will prevent us from running out of auto-incrementing ID values until such time as we can upgrade to peewee 3 and change the field type to a BigInt

Fixes https://jira.coreos.com/browse/QUAY-943
2018-05-18 20:15:16 -04:00
Joseph Schorr
f5fafc480f Notifications can target robots, but the model did not reflect that
This fixes a breakage when deleting robots that have notifications targeting them.
2018-05-15 16:19:46 -04:00
josephschorr
58612955d6
Merge pull request #3079 from quay/joseph.schorr/QUAY-927/robots-limit
Limit robots displayed in entity search
2018-05-15 13:59:48 -04:00
josephschorr
6780a1f06b
Merge pull request #3077 from quay/joseph.schorr/QUAY-931/log-slowdown
Logs slowdown fixes
2018-05-15 13:28:05 -04:00
Joseph Schorr
5c50161d85 Limit robots displayed in entity search
Before, we'd load *all* the robots, which can be a huge issue in namespaces with a large number of robots. Now, we only load the top-20 robots (as per recency in login), and we also limit the information returned to the entity search to save some bandwidth.

Fixes https://jira.coreos.com/browse/QUAY-927
2018-05-15 11:00:57 -04:00
Joseph Schorr
b98c65b3a3 Filter out deleted users and organizations from the superuser list
Superusers were getting confused because the users/orgs were being disabled and renamed, but still appeared in the list until they were GCed by the background worker. Now we just hide them.

Fixes https://jira.coreos.com/browse/QUAY-936
2018-05-15 10:42:41 -04:00
Joseph Schorr
e3248bde47 Small fixes to make loading of logs faster
Removes filtering of log types where not necessary, removes filtering based on namespace when filtering based on repository (superfluous check that was causing issues in MySQL preventing the use of the correct index) and fix some other small issues around the API

Fixes https://jira.coreos.com/browse/QUAY-931
2018-05-14 11:41:49 -04:00
Joseph Schorr
29058201e5 Fix bug in modelutil pagination that caused us to load far more results than necessary
Also adds tests for the modelutil pagination
2018-05-14 11:40:31 -04:00
Joseph Schorr
49699fe251 Fix MySQL bug in full text search
Fixes https://jira.coreos.com/browse/QUAY-937
2018-05-14 08:41:40 -04:00
Joseph Schorr
f06eec8a35 Fix issue where we didn't delete robots immediately under a namespace
This could result in "hanging" robot accounts, although that would only leak the names of said accounts. Now we delete them immediately AND we proactively delete them before replacing the namespace (just to be sure)
2018-05-09 17:53:30 +03:00
Joseph Schorr
178c8e7cb0 Fix bug in in-memory data cache
Previously, if we didn't find a key, we'd empty the entire cache, making it essentially a single-key cache. We skip clearing now, although this does mean we won't GC expired entries (not a problem for tests, though)
2018-05-01 13:28:24 +03:00
Joseph Schorr
3309daa32e Add support for reduced initial build count for new possible abusing users
If configured, we now check the IP address of the user signing up and, if they are a possible threat, we further reduce their number of allowed maximum builds to the configured value.
2018-04-20 18:46:32 +03:00