vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
8717 commits 2 branches 62 tags 205 MiB
Commit graph

309 commits

Author SHA1 Message Date
Joseph Schorr
71ec23b550 Switch QueueItem state_id to be unique after a backfill 2017-01-18 17:43:41 -05:00
josephschorr
e2748fccd9 Merge pull request #2282 from coreos-inc/motd-updates 
Severity and Markdown support in MOTD
 2017-01-18 17:41:27 -05:00
Joseph Schorr
3106504f39 Severity and Markdown support in MOTD 
[Delivers #133555165]
 2017-01-18 16:55:32 -05:00
Joseph Schorr
af23d2bedd Remove unique from queue item state_id 2017-01-18 15:04:26 -05:00
Joseph Schorr
8c4e86f48b Change queue to use state-field for claiming items 
Before this change, the queue code would check that none of the fields on the item to be claimed had changed between the time when the item was selected and the item is claimed. While this is a safe approach, it also causes quite a bit of lock contention in MySQL, because InnoDB will take a lock on *any* rows examined by the `where` clause of the `update`, even if they will ultimately thrown out due to other clauses (See: http://dev.mysql.com/doc/refman/5.7/en/innodb-locks-set.html: "A ..., an UPDATE, ... generally set record locks on every index record that is scanned in the processing of the SQL statement. It does not matter whether there are WHERE conditions in the statement that would exclude the row. InnoDB does not remember the exact WHERE condition, but only knows which index ranges were scanned").

As a result, we want to minimize the number of fields accessed in the `where` clause on an update to the QueueItem row. To do so, we introduce a new `state_id` column, which is updated on *every change* to the QueueItem rows with a unique, random value. We can then have the queue item claiming code simply check that the `state_id` column has not changed between the retrieval and claiming steps. This minimizes the number of columns being checked to two (`id` and `state_id`), and thus, should significantly reduce lock contention. Note that we can not (yet) reduce to just a single `state_id` column (which should work in theory), because we need to maintain backwards compatibility with existing items in the QueueItem table, which will be given empty `state_id` values when the migration in this change runs.

Also adds a number of tests for other queue operations that we want to make sure operate correctly following this change.

[Delivers #133632501]
 2017-01-17 13:29:26 -05:00
Charlton Austin
ca832df975 Adding in new indices for queueitem table. 2017-01-17 10:04:31 -05:00
Joseph Schorr
624b2a8385 Have security scanner analyze only send notifications for *new* layers 
Following this change, anytime a layer is indexed by the security scanner, we only send notifications out if the layer previously had a security_indexed_engine value of `-1`, thus ensuring it has *never* been indexed previously. This will allow us to change to version of the security scanner upwards, and have all the images be re-indexed, without firing off notifications in a spammy manner.
 2016-12-13 23:17:11 -05:00
Charlton Austin
0aa6e6cd58 Merge pull request #2203 from charltonaustin/fix_build_component_cleanup 
Adding in a cancel method to the build component so we can properly c…
 2016-12-06 14:13:10 -05:00
Charlton Austin
c6be12e31e Adding in a cancel method to the build component so we can properly clean up the job task. 2016-12-06 13:37:49 -05:00
Jake Moshenko
7c490b46c8 Only save dirty fields on Queue queries. 2016-12-05 18:12:14 -05:00
Charlton Austin
fd7c566d31 Adding in cancel for a build that is building. 2016-11-16 17:40:24 -05:00
Joseph Schorr
1a61ef4e04 Report the user's name and company to Marketo 
Also fixes the API to report the other changes (username and email) as well
 2016-11-14 17:34:50 -05:00
Jake Moshenko
b5834a8a66 Collapse all migrations prior to 2.0.0 into one. 2016-11-10 17:31:00 -05:00
Joseph Schorr
0f2eb61f4a Add collection of user metadata: name and company 2016-11-08 16:15:02 -05:00
Joseph Schorr
1e3b354201 Add support for temp usernames and an interstitial to confirm username 
When a user now logs in for the first time for any external auth (LDAP, JWT, Keystone, Github, Google, Dex), they will be presented with a confirmation screen that affords them the opportunity to change their Quay-assigned username.

Addresses most of the user issues around #74
 2016-11-03 15:59:14 -04:00
josephschorr
edc2bc8b93 Merge pull request #1698 from coreos-inc/delete-namespace 
Add support for deleting namespaces (users, organizations)
 2016-10-21 16:54:52 -04:00
Joseph Schorr
73eb66eac5 Add support for deleting namespaces (users, organizations) 
Fixes #102
Fixes #105
 2016-10-21 15:41:09 -04:00
Joseph Schorr
715fc27474 Add a default database connect timeout 
Fixes #1760
 2016-10-17 13:33:30 -04:00
Charlton Austin
97d644d95d Adding in the delete api and the delete and create UI. 2016-10-13 10:40:52 -04:00
charltonaustin
4ae6e6efa9 Fixing some database integration errors 2016-10-10 10:51:30 -04:00
charltonaustin
1e733ddffb Adding in a new message data model and the corresponding methods to in the API. 2016-10-07 15:56:58 -04:00
Jimmy Zelinskie
a1a930b833 database: fix indices post-rebase 2016-09-26 14:49:58 -04:00
Jimmy Zelinskie
ca883e5662 port label support to refactored v2 registry 2016-09-26 14:49:58 -04:00
Joseph Schorr
3c8b87e086 Fix verbs in manifestlist 
All registry_tests now pass
 2016-09-26 14:49:58 -04:00
Jimmy Zelinskie
783c9e7a73 stop exporting experimental database models 2016-09-26 14:49:23 -04:00
Joseph Schorr
b775458d4b lifetimes on Tags should now be in milliseconds 
Fixes #1779
 2016-09-26 14:49:04 -04:00
Jimmy Zelinskie
e6c99bb471 re-ordered BlobUploading fields 2016-09-26 14:48:05 -04:00
Joseph Schorr
94d71f2166 Fix model to actually initialize 2016-09-26 14:47:06 -04:00
Jimmy Zelinskie
9cfd6ec452 database: initial manifestlist schema changes 2016-09-26 14:47:06 -04:00
Jake Moshenko
584a5a7ddd Reduce database bandwidth by tracking gc candidate images. 2016-09-07 13:25:19 -04:00
Jake Moshenko
1d8b72235a Add a helper method to Image to parse ancestor string. 2016-09-07 10:48:58 -04:00
Joseph Schorr
608ffd9663 Basic labels support 
Adds basic labels support to the registry code (V2), and the API. Note that this does not yet add any UI related support.
 2016-08-26 15:24:26 -04:00
Joseph Schorr
0f46230493 Add an index for lookup by account to log entries 
Also fixes the query to require one less join
 2016-08-12 17:39:31 -04:00
josephschorr
1a137ee7b3 Merge pull request #1643 from coreos-inc/db-retry 
Enable automatic retry for the database
 2016-08-08 15:04:25 -04:00
Joseph Schorr
700e7b74e4 Enable automatic retry for the database 2016-08-08 15:02:42 -04:00
Jimmy Zelinskie
22a25ac2d3 Revert "Merge pull request #1678 from coreos-inc/delete-repo-fix" 
This reverts commit df64caf133, reversing
changes made to 0d1e453566.
 2016-08-08 12:38:15 -04:00
Jimmy Zelinskie
052c31752b MIGRATION: drop foreign keys on logentry table 
This migration generates the following for MySQL:

BEGIN;

-- Running upgrade 1093d8b212bb -> 6243159408b5

ALTER TABLE logentry DROP FOREIGN KEY fk_logentry_account_id_user;

ALTER TABLE logentry DROP FOREIGN KEY
fk_logentry_repository_id_repository;

ALTER TABLE logentry DROP FOREIGN KEY fk_logentry_performer_id_user;

UPDATE alembic_version SET version_num='6243159408b5' WHERE
alembic_version.version_num = '1093d8b212bb';

COMMIT;
 2016-08-08 12:38:15 -04:00
josephschorr
6716a2562b Merge pull request #1680 from coreos-inc/add-missing-index 
Add various missing indexes
 2016-08-08 12:34:58 -04:00
Joseph Schorr
80a37fd295 Add various missing indexes 
Indexes added:

Image::repository - Needed for model.image.get_repository_images_without_placements
RepositoryTag::image - Needed for model.tag.get_tags_for_image
RepositoryTag::repository - Needed for repository deletion
RepositoryBuild::phase - Needed for model.build.list_repository_builds sorting
RepositoryBuild::started - Needed for model.build.list_repository_builds sorting
RepositoryBuild::repository+started+phase - Needed for model.build.list_repository_builds
RepositoryBuild::started+logs_archived+phase - Needed for model.build.get_archivable_build lookup
 2016-08-08 12:34:45 -04:00
Joseph Schorr
0b5cd95693 Have repo deletion not lock all the things 2016-08-04 16:45:59 -04:00
Joseph Schorr
06d52f2c83 Fix handling of multi-part branches in the build triggers 
Fixes #1360
 2016-07-26 13:41:13 -07:00
Joseph Schorr
a43b741f1b Add a uniqueness hash to derived image storage to break caching over tags 
This allows converted ACIs and squashed images to be unique based on the specified tag.

Fixes #92
 2016-06-20 16:34:52 -04:00
Jimmy Zelinskie
4a521f5844 database: revert logentry foreign key proxy 2016-04-29 14:10:33 -04:00
Evan Cordell
2242c6773d Add 'Automatic' ServiceKeyApprovalType 2016-04-29 14:10:33 -04:00
Joseph Schorr
4f63a50a17 Change account-less logs to use a user and not null 
This allows us to skip the migration
 2016-04-29 14:09:37 -04:00
Jimmy Zelinskie
370ac3ecd0 service keys: add rotation_duration field 2016-04-29 14:05:16 -04:00
Joseph Schorr
11ff3e9b59 keys ui WIP 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
dff59b4a39 service key migration 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
42b5196b21 add notification path and use for service keys 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
f406942984 converging on proper rotation 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
35ed73e195 rework superuser api 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
4079dba167 service keys: do all the right stuff 2016-04-29 13:38:25 -04:00
Jimmy Zelinskie
499bb16306 service key server wip 2016-04-29 13:38:25 -04:00
Joseph Schorr
c0374d71c9 Refactor the security worker and API calls and add a bunch of tests 2016-02-25 12:29:41 -05:00
Joseph Schorr
abd2e3c234 V1 Docker ID <-> V2 layer SHA mismatch fix 
Fix handling of V1 Docker ID <-> V2 layer SHA mismatch by dynamically rewriting the manifest to use new synthesized IDs for all layers above the mismatch. Also adds a bunch of tests for this and other use cases, fixes a bug around manifest digest uniqueness and fixes the 5.5 migration for MySQL.
 2016-02-12 17:39:27 +02:00
Jake Moshenko
fe2bdeb6cb Require some data from all models in initdb 2016-01-19 15:30:27 -05:00
Jake Moshenko
8ab6c8a22d Fix torrent hash generation to work in mixed stacks 2016-01-11 16:43:46 -05:00
Jake Moshenko
1ae101c917 Address torrent feature review comments. 2016-01-08 16:38:21 -05:00
Jake Moshenko
77aa58996a Fix the db definition for torrentinfo and add migration 2016-01-06 14:04:03 -05:00
Jake Moshenko
ce8fcbeaae Update the pieces to use base64 encoded binary 2016-01-04 16:17:51 -05:00
Jake Moshenko
fe87d3c796 Hash and track layer file chunks for torrenting 2016-01-04 16:17:51 -05:00
Joseph Schorr
10efa96009 Add support for custom billing invoice email address 
Fixes #782
 2015-12-28 13:59:50 -05:00
Joseph Schorr
94ece129d4 Remove remaining recursive queries on repo delete and add test 2015-12-18 16:04:03 -05:00
Joseph Schorr
2e7835c372 Fix user deletion under MySQL 
Fixes #973
 2015-12-17 15:05:15 -05:00
Joseph Schorr
9698d6f6a0 Add created column to blob upload 
Fixes first half of #1054
 2015-12-14 15:27:48 -05:00
Joseph Schorr
54095eb5cb Handle the common case of one chunk when calculating the uncompressed size 
Reference #992
 2015-12-14 15:27:48 -05:00
Joseph Schorr
0f7c8105b0 Remove DerivedImageStorage table 2015-11-25 11:46:59 -05:00
Joseph Schorr
762cd56e64 Change derived storage to be based on image 
Fixes #971
 2015-11-24 12:44:07 -05:00
Jake Moshenko
18b14001b4 Add indices for the security worker fields on Image 
Fixes #906
 2015-11-18 13:29:51 -05:00
Jake Moshenko
206e18d160 Image parents do not have to be nulled transitively on repo delete 2015-11-17 16:48:26 -05:00
Jake Moshenko
e252397292 Switch parent back to a ForeignKeyField without a constraint 2015-11-17 16:09:33 -05:00
Jake Moshenko
a1ccd860e7 Merge pull request #823 from coreos-inc/phase3-11-07-2015 
Phase3 11 07 2015
 2015-11-11 14:22:19 -05:00
Joseph Schorr
2d2662f53f Fix deleting repos and images under MySQL 
MySQL doesn't handle constraints at the end of transactions, so deleting images currently fails. This removes the constraint and just leaves parent_id as an int
 2015-11-09 14:42:05 -05:00
Jake Moshenko
ad93425ead Stop writing to v1 checksum on ImageStorage 2015-11-06 16:40:04 -05:00
Joseph Schorr
0f3db709ea Add a vulnerability_found event for notice when we detect a vuln 
Fixes #637

Note: This PR does *not* actually raise the event; it merely adds support for it
 2015-11-06 15:22:18 -05:00
Quentin Machu
f59e35cc81 Add support for Quay's vulnerability tool 2015-11-06 15:22:18 -05:00
Jake Moshenko
3d0bcbaaeb Move v1 checksums to image and track v2 separately 2015-11-06 15:17:55 -05:00
Joseph Schorr
bbf4a1fac4 Remove the used_legacy_github column 2015-11-06 15:17:55 -05:00
Jake Moshenko
cb7ec2f239 Backport remaining v2 changes to phase4 2015-10-24 15:00:13 -04:00
Jake Moshenko
9da64f3aba Stop writing to deprecated columns for image data. 2015-10-24 14:45:15 -04:00
josephschorr
217779273f Merge pull request #503 from coreos-inc/ghmigrate 
Migrate all GitHub build triggers to use deploy keys
 2015-09-16 18:32:32 -04:00
Joseph Schorr
eff9ff7a66 Migrate all GitHub build triggers to use deploy keys 2015-09-16 17:55:51 -04:00
Silas Sewell
0a48f1cfb0 Merge pull request #495 from coreos-inc/quay-versions 
Add quay releases
 2015-09-16 17:29:58 -04:00
Silas Sewell
386c017d99 Add quay releases 2015-09-16 17:18:46 -04:00
Joseph Schorr
30379a2dd8 Fix interleaved repo delete with RAC via a transaction 
The RepositoryActionCount table can have entries added while a repository deletion is in progress. We now perform the repository deletion under a transaction and explicitly test for RAC entries in the deletion unit test (which doesn't test interleaving, but it was missing this check).

Fixes #494
 2015-09-16 15:34:32 -04:00
Jake Moshenko
b56de3355c Migrate data back to Image in preparation for v2 2015-09-15 11:53:31 -04:00
Joseph Schorr
724b1607d7 Add automatic storage replication 
Adds a worker to automatically replicate data between storages and update the database accordingly
 2015-09-01 14:53:32 -04:00
Joseph Schorr
84276ee945 Better notifications UI 
Fixes #369
 2015-08-17 17:08:58 -04:00
Joseph Schorr
9f2d6282bd Add missing index on retries_remaining 2015-08-04 18:01:28 -04:00
Joseph Schorr
3d6c92901c Switch to using an aggregated logs query and infinite scrolling 
This should allow users to work with large logs set.

Fixes #294
 2015-07-31 16:38:02 -04:00
Joseph Schorr
ac1b46e7ec Add missing migration 2015-07-22 16:19:10 -04:00
Jake Moshenko
3efaa255e8 Accidental refactor, split out legacy.py into separate sumodules and update all call sites. 2015-07-17 11:56:15 -04:00
Jake Moshenko
bea8b9ac53 More changes for registry-v2 in python. 
Implement the minimal changes to the local filesystem storage driver and feed them through the distributed storage driver.
Create a digest package which contains digest_tools and checksums.
Fix the tests to use the new v1 endpoint locations.
Fix repository.delete_instance to properly filter the generated queries to avoid most subquery deletes, but still generate them when not explicitly filtered.
 2015-07-17 11:50:41 -04:00
Jake Moshenko
acbcc2e206 Start of a v2 API. 2015-07-17 11:50:41 -04:00
Joseph Schorr
3ba321934f Fix deletion of robot accounts when attached to builds 
Fixes #204
 2015-06-30 22:56:44 +03:00
Jimmy Zelinskie
756d6784ca Merge pull request #192 from coreos-inc/sqlssl 
Allow SSL cert for the database to be configured
 2015-06-29 13:33:31 -04:00
Joseph Schorr
dc5af7496c Allow superusers to disable user accounts 2015-06-29 18:40:52 +03:00
Joseph Schorr
bb07d0965f Allow SSL cert for the database to be configured 
This change adds a field for the SSL cert for the database in the setup tool. Fixes #89
 2015-06-29 08:08:10 +03:00
Jake Moshenko
79f1181a63 Switch build-scheduled to an official build phase. 2015-06-10 16:19:51 -04:00
Joseph Schorr
a59100b231 Add the missing index on the peewee side. We already have the associated migration. 2015-04-30 15:56:40 -04:00