vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
2604 commits 2 branches 62 tags 205 MiB
Commit graph

499 commits

Author SHA1 Message Date
Jake Moshenko
f4681f2c18 Merge branch 'master' into nomenclature 
Conflicts:
	test/data/test.db
 2014-11-17 17:59:59 -05:00
Joseph Schorr
58ca76239b Add ability to one-click generate an authorization access token in the applications panel 2014-11-17 14:54:07 -05:00
Joseph Schorr
178c5a7ac0 Add an index to the docker_image_id for faster lookup 2014-11-13 12:51:50 -05:00
Joseph Schorr
d73747ce1d - Fix some other group_by clauses 
- Fix garbage_collect for an empty list (fixes a test)
- Add a script which runs the full test suite against mysql and postgres (note: QueueTest's are broken for MySQL, but they obviously work in production, so they need to be fixed)
 2014-11-13 12:51:37 -05:00
Joseph Schorr
a2e2dcb010 Fix group_by clauses for postgres 2014-11-12 20:32:06 -05:00
Joseph Schorr
079289c53c HACK: Don't join on the visibility table because it is horrendously slow for some odd reason. 2014-11-12 14:51:24 -05:00
Joseph Schorr
eddcc02ea6 Make repository deletes much faster by adding custom deletion code and have additional tests to verify the deletion code paths 2014-11-10 23:05:20 -05:00
Joseph Schorr
e7cbda86f7 Merge branch 'perf' 2014-11-10 21:52:27 -05:00
Joseph Schorr
091f821a6a - Rename get_repo_image to get_repo_image_extended and get_repo_image_directly to get_repo_image 
- Remove the configure call from CloseForLongOperation
- Other small fixes
 2014-11-10 13:44:36 -05:00
Joseph Schorr
72fedef097 Move the robot deletion code into a delete_instance method on the User object 2014-11-10 13:18:17 -05:00
Jake Moshenko
ca435fc7a6 Rename robots when we rename a user. Do not use the namespace from the path to check permissions from the incoming webhooks since the namespace may have changed and we cannot recreate them in remote services easily. 2014-11-09 17:50:57 -05:00
Joseph Schorr
17f605a9ef Select only a single token. 2014-11-09 15:50:50 -05:00
Joseph Schorr
691be49817 Fix issues with the perf updated code 2014-11-07 14:36:32 -05:00
Joseph Schorr
158acd4f41 - Turn on foreign key constraint checking in the tests 
- Change all ForeignKeyField's that refer to users to use our custom class, and mark those that allow robots
- Change robot delete to only execute the subset of queries necessary to actually delete robots
 2014-11-07 12:05:21 -05:00
Joseph Schorr
d5bbb57481 Change registry code to disconnect from the DB before long I/O operations 2014-11-06 18:00:52 -05:00
Joseph Schorr
23d9bd2b42 Change verbs to use short lived database connections 2014-11-06 17:50:48 -05:00
Joseph Schorr
c569299e5c Database optimizations around image creation and logs lookup 2014-11-06 14:48:16 -05:00
Joseph Schorr
a35bc11912 Add perf comments 2014-11-05 12:27:38 -05:00
Joseph Schorr
29c30b336e Move the uncompressed image size migration call outside of alembic, since it will sometimes deadlock with certain kinds of DBs (because alembic is running things inside a transaction) 2014-11-03 21:01:41 -05:00
Joseph Schorr
9aa72c5cc2 Fix migration issues: 
- MySQL 5.5 doesn't support the now() call as a default
  - Postgres migration isn't auto-committed, so we have to check if the table exists first
 2014-11-03 15:25:55 -05:00
Joseph Schorr
4eedd54b66 - Make usage language more accurate by stating "repositories" 
- Have usage counter be based on a 4 weeks TTL
- Add a simple usage counter breakage test
 2014-10-30 13:26:02 -04:00
Joseph Schorr
c1398c6d2b - Add a log entry for repo verb handling and make the container usage calculation take it into account 
- Move all the repo push/pull/verb logging into a central track_and_log method
- Readd images accidentally deleted in the last CL
- Make the uncompressed size migration script better handle exceptions
 2014-10-29 15:42:44 -04:00
Joseph Schorr
e9c2e54dbc Change delete to once again not perform everything under a transaction. A recent revision accidentally moved everything back under it. 2014-10-29 11:59:02 -04:00
Joseph Schorr
109850b428 Add a basic usage counter for enterprise 2014-10-28 16:33:13 -04:00
Joseph Schorr
c06f57a6e7 Make sure builders close the db handle when no work comes in and make the metrics transaction smaller in scope 2014-10-24 11:40:02 -04:00
Jake Moshenko
1461310ab8 Merge remote-tracking branch 'origin/master' into nomenclature 
Conflicts:
	endpoints/common.py
	endpoints/notificationhelper.py
	test/data/test.db
	workers/dockerfilebuild.py
 2014-10-23 13:25:37 -04:00
Joseph Schorr
5db9cd948b Add better (jinja-based) messaging to the notifications and add some fixes for the email templates 2014-10-22 19:01:56 -04:00
Joseph Schorr
28a463f998 Make the 'created' field optional, in case the user does not specify it via the API 2014-10-20 12:07:49 -04:00
Jake Moshenko
dc5ee43a3a MySQL still doesn't allow us to have empty IN clauses. 2014-10-17 17:49:18 -04:00
Jake Moshenko
6ca0115b5e MySQL doesnt allow us to use the table we are modifying in a subquery. 2014-10-17 17:01:55 -04:00
Jake Moshenko
efe8825a15 We can't even use an empty in query for images. 2014-10-17 16:37:05 -04:00
Jake Moshenko
acd5a1e07e Only garbage collect storage when it might result in something. Return the count of the removed images to fulfill the contract expected by some callers. 2014-10-17 16:13:32 -04:00
Jake Moshenko
18a944427f Merge branch 'master' of bitbucket.org:yackob03/quay 2014-10-17 15:26:58 -04:00
Jake Moshenko
c093e5a326 Add a whitelist of candidate storages which will speed up the orphan queries and limit the damage of GC run amok. 2014-10-17 15:26:51 -04:00
Joseph Schorr
26a9bb50d5 Fix spelling mistake 2014-10-17 14:44:17 -04:00
Jake Moshenko
d8149295ab Update the GC code to do everything with subqueries, making each GC run a bounded finite number of queries with a fixed length. 2014-10-17 14:33:04 -04:00
Jake Moshenko
380eb49e58 Split out the GC code into smaller transactions. Remove from the actual storage component after the fact, outside of the transactions. 2014-10-17 11:42:09 -04:00
Jake Moshenko
fa6a06502d Change the default redis host to localhost. Fix some whitespace issues in the userevents module. 2014-10-14 14:37:02 -04:00
Jake Moshenko
328db8b660 Split the app into separate backends, which can use different worker types and different timeouts. 2014-10-14 13:58:08 -04:00
Joseph Schorr
bb9502ee77 Make sure to raise the exception for WSGI and make the delete method's intent more clear 2014-10-08 13:51:50 -04:00
Joseph Schorr
d16fdde528 Fix bug in dockerloadformat and make sure we handle exceptions properly in the verb call 2014-10-08 13:43:12 -04:00
Jake Moshenko
c34a8b6727 Fix the squashed image migration to insert the proper image transformation type. 2014-10-08 11:47:02 -04:00
Jake Moshenko
1a8287af55 Missed a table import. 2014-10-07 17:33:20 -04:00
Jake Moshenko
bbcdf18fdd Update the migration to reflect the state of the database. 2014-10-07 16:19:05 -04:00
Jake Moshenko
1968c67cba Merge remote-tracking branch 'origin/laffa' into nomenclature 2014-10-07 16:09:45 -04:00
Jake Moshenko
626d984fb2 Fixes to the migration generator script. 2014-10-07 16:09:30 -04:00
Jake Moshenko
fe59ad4fb5 Rectify our prod database, previous migrations, and database definition files. 2014-10-07 16:03:57 -04:00
Jake Moshenko
c77a3dc6ea Update the migration for translating queue names to take into account previous migrations. 2014-10-07 15:57:17 -04:00
Jake Moshenko
a0d94f9d59 Merge remote-tracking branch 'origin/laffa' into nomenclature 
Conflicts:
	test/data/test.db
 2014-10-07 15:56:28 -04:00
Jake Moshenko
ee9973a395 Merge remote-tracking branch 'origin/master' into nomenclature 2014-10-07 15:43:59 -04:00
Joseph Schorr
f4daa5e97b - Update the migrations tool to verify migrations work up and down for both MySQL and PostgresSQL. 
- Add migrations for the squashed image tables and for backfilling the uncompressed sizes
- Make sure gzip stream uses a max length when determining the uncompressed size
 2014-10-07 15:29:56 -04:00
Joseph Schorr
f38ce51943 Merge master into laffa 2014-10-07 14:03:17 -04:00
Jake Moshenko
4ad592e7ce Add an index to the image storage uuid to improve performance. 2014-10-06 18:44:37 -04:00
Jake Moshenko
c4266140e2 Fix all of the upgrades and downgrades to work on both mysql and postgres. 2014-10-06 16:41:49 -04:00
Jake Moshenko
ed8bcff39e Merge remote-tracking branch 'origin/master' into nomenclature 
Conflicts:
	test/data/test.db
	workers/dockerfilebuild.py
 2014-10-06 10:29:39 -04:00
Jake Moshenko
5c18ffe67d Allow the namespace column to be null, and also non-unique. Fix the uncompressed size clobbering the size on the wire field. Add metadata constraints so that foreign key constraints get predictable names. Fix all downgrade migrations. 2014-10-02 10:46:20 -04:00
Jake Moshenko
e8b3d1cc4a Phase 4 of the namespace to user migration: actually remove the column from the db and remove the dependence on serialized namespaces in the workers and queues 2014-10-01 14:23:46 -04:00
Joseph Schorr
2c5cc7990f Allow for additional REDIS config such as password and port 2014-10-01 14:16:42 -04:00
Joseph Schorr
6c520b8b0b Fixed broken query 2014-09-30 13:19:32 -04:00
Joseph Schorr
5f776771d1 Merge branch 'master' of https://bitbucket.org/yackob03/quay 2014-09-29 17:01:07 -04:00
Joseph Schorr
474add0fb1 Have the layer PUT method calculate the uncompressed size in realtime, as trusting the JSON is fraught with complications 2014-09-29 17:00:47 -04:00
Jake Moshenko
fbc7ee3bce Phase 3 of the namespace migration, remove write references to the namespace string column. 2014-09-29 15:44:25 -04:00
Jake Moshenko
ae9026ee91 MySQL will not allow us to revoke the possibility for the namespace_user column to be nullable. 2014-09-26 14:50:26 -04:00
Jake Moshenko
21d8c41320 Add the existing types to the alter column to make MySQL happy. 2014-09-26 12:58:03 -04:00
Jake Moshenko
e9745dbc96 Fix the SQL migration. 2014-09-26 11:17:42 -04:00
Jake Moshenko
03190efde3 Phase 2 of migrating repo namespaces to referencing user objects, backfilling the rows without a value for namespace_user, and changing all accesses to go through the namespace_user object. All tests are passing, manual testing still required. 2014-09-24 18:01:35 -04:00
Jake Moshenko
98d8e0fe37 Fixes for the uncompressed size backfill script. 2014-09-23 16:06:59 -04:00
Joseph Schorr
ec484e3efc Move size and checksum updates into the metadata call 2014-09-23 15:49:28 -04:00
Jake Moshenko
74e35f917e Switch socket_timeout to socket_connect_timeout for redis. 2014-09-23 14:59:00 -04:00
Jake Moshenko
798c13aec8 Fix the migration to use the proper column name that peewee expects. 2014-09-23 14:20:15 -04:00
Jake Moshenko
b6d635b648 Fix the migration to use the proper base. 2014-09-23 14:04:20 -04:00
Joseph Schorr
86dfca2e3e Add uncompressed size field to the image storage and add a backfill script (which is not yet automatically called) 2014-09-23 14:01:27 -04:00
Joseph Schorr
87bc37f6c8 Merge branch 'sunday' 2014-09-23 11:29:03 -04:00
Joseph Schorr
ba0963a81c Update the worker code to better handle exceptions, fix the utcdate issue and make sure we send the proper retry. Also updates notification workers to send JobExceptions rather than returning true or false 2014-09-23 11:01:50 -04:00
Joseph Schorr
f3b03ebc34 Add a feature flag for disabling all emails 2014-09-22 19:11:48 -04:00
Jake Moshenko
c68d49dfce Fix the get_matching_users query to work with peewee 2.3+ 2014-09-22 18:42:48 -04:00
Jake Moshenko
a6225ad34a Set an aggressive timeout on the redis connections to allow the unit tests to run when communication with redis is disabled. 2014-09-22 18:42:22 -04:00
Jake Moshenko
3259cda000 The new strategy is to do a three phase migration. This is the first phase: getting the namespace user in the db and written for all new repositories. 2014-09-22 17:27:02 -04:00
Joseph Schorr
297c8ad29c Add migration to backfill uncompressed image sizes on the storage 2014-09-22 15:04:28 -04:00
Joseph Schorr
1658475ac1 Previous revision should not have the image storage col in it 2014-09-22 14:39:44 -04:00
Joseph Schorr
f16878cce9 Add migration for synthetic image tables 2014-09-22 14:38:42 -04:00
Joseph Schorr
70e0aba257 Add a script for generating schema migrations. Should be run from the root quay directory. 2014-09-22 14:36:52 -04:00
Joseph Schorr
f23038c6ee Update the worker code to better handle exceptions, fix the utcdate issue and make sure we send the proper retry. Also updates notification workers to send JobExceptions rather than returning true or false 2014-09-22 12:52:57 -04:00
Jake Moshenko
8626d1cd70 Initial changes to move repositories from using a namespace string to referencing a user object. Also stores the user id in the cookie rather than the username, to allow users to be renamed. This commit must not be used unmodified because the database migration is too aggressive for live migration. 2014-09-19 10:17:23 -04:00
Jake Moshenko
8c00eabedd Merge remote-tracking branch 'origin/better-emails' 2014-09-19 10:04:02 -04:00
Joseph Schorr
6b05b55225 Add unicode password support 2014-09-18 17:36:26 -04:00
Jake Moshenko
11bb8e6448 Actually store the generated image storage in the database, and allow it to be garbage collected when the parent image storage is collected. 2014-09-18 17:26:40 -04:00
Joseph Schorr
b212dbb2ab Merge branch 'master' into better-emails 2014-09-18 13:20:32 -04:00
Jake Moshenko
8b5e7621d2 Add the required notification kind for team invites to the migration that was previously run. 2014-09-16 10:11:18 -04:00
Joseph Schorr
e3c52fa0eb Work in progress. This is currently broken! 2014-09-16 00:18:57 -04:00
Jake Moshenko
e13c027631 Add a migration for team member invites. 2014-09-15 23:57:05 -04:00
Jake Moshenko
75d2ef377e Merge remote-tracking branch 'origin/master' into comewithmeifyouwanttowork 
Conflicts:
	data/model/legacy.py
 2014-09-15 17:52:17 -04:00
Jake Moshenko
2b59a0cbe1 Merge branch 'master' of bitbucket.org:yackob03/quay 2014-09-15 15:59:18 -04:00
Jake Moshenko
efb66f7c1e Select the random row function based on DB driver. 2014-09-15 15:58:56 -04:00
Joseph Schorr
913b3e472f Add ability to detach external login services 2014-09-15 12:01:02 -04:00
Joseph Schorr
e8ad01cb41 Lots of small NPE and other exception fixes 2014-09-15 11:27:33 -04:00
Joseph Schorr
10faa7de84 Only allow users matching the team invite to accept, if the invite was specified for a user (rather than an email) 2014-09-12 14:29:01 -04:00
Jake Moshenko
c01de4a916 Set redis logs entries to expire rather than to immediately delete them to make the logs archiver idempotent. 2014-09-12 13:13:14 -04:00
Jake Moshenko
353da13b16 Set up the service monitoring for the log migration script. Add a database migration to add the logs_archived column. 2014-09-12 11:57:47 -04:00
Jake Moshenko
b55e79e5d3 Merge remote-tracking branch 'origin/master' into waltermitty 2014-09-12 11:45:07 -04:00