Joseph Schorr
693e47e063
Add additional assertions and further checks in storage code
2019-02-20 12:12:56 -05:00
Joseph Schorr
f75f315037
Optimize lookup of shared global blobs
...
Currently, we only have one (the shared empty layer), but this should make the blob lookups for repositories significantly faster, as we won't need to do the massive join.
2019-02-14 12:46:42 -05:00
Joseph Schorr
e309508776
Change lookup of blobs to not use a subquery, which is murder on MySQL
2019-01-03 15:41:21 -05:00
Joseph Schorr
adccdd30ca
Read blobs from new manifest blob table where relevant
2018-11-25 17:31:09 +02:00
Joseph Schorr
366bddd20a
Fix NPE bug in torrentinfo set call
...
Fixes https://sentry.io/coreos/backend-production/issues/747494304
2018-10-30 16:47:59 -04:00
Joseph Schorr
a172de4fdc
Reimplement cache support for blobs in the registry data model
2018-10-18 14:25:40 -04:00
Joseph Schorr
f252b0b16f
Move verbs endpoint to use new registry data model
2018-08-29 16:38:54 -04:00
Joseph Schorr
6c5c2f1a75
Add new methods to registry data model interface in prep for moving verbs to using it
2018-08-29 16:38:54 -04:00
Joseph Schorr
a46660a06f
Add new Manifest, ManifestLabel, ManifestLegacyImage and ManifestBlob tables and start writing and GCing to/from them
...
This change also starts passing in the manifest interface, rather than the raw data, to the model for writing.
Note that this change does *not* backfill the existing rows in to the new tables; that will occur in a followup PR. The new columns in `tagmanifest` and `tagmanifestlabel` will be used to track the backfill, as it will occur in a worker.
2018-08-06 13:01:11 -04:00
Joseph Schorr
487edf0ba1
Phase 3 of Appr migration
...
Deletes the old models and their code
2018-07-21 15:43:00 -04:00
Joseph Schorr
113bb96f29
Phase 1 of migrating APPR-specific tables to tables with the Appr prefix
...
Fixes https://jira.coreos.com/browse/QUAY-950
2018-07-20 13:19:01 -04:00
Joseph Schorr
51e67ab7f5
Fix get_blob_path to not make any database calls and add a test
...
This will be supported by caching, hopefully removing the need to hit the database when the blob object is cached
2017-12-13 16:27:46 -05:00
Joseph Schorr
74dd0ef8e8
Add a batch get_matching_tags_for_images method
...
This will be used in the security notification worker to retrieving the tags needed in a set of batch calls, rather than multiple calls per image
2017-05-02 15:38:25 -04:00
Joseph Schorr
bdda74d6df
Make sure GC checks new Blob table as well before deleting CAS storage
2017-03-22 23:53:21 -04:00
Joseph Schorr
62312e6461
Add warning when CAS paths are skipped and ensure we are under a transaction
2017-03-08 17:01:07 -05:00
Joseph Schorr
69e550d125
Fix GC handling around CAS paths
...
Adds code to ensure we never GC CAS paths that are shared amongst multiple ImageStorage rows, as well as an associated pair of tests to catch the positive and negative cases.
2017-03-07 13:48:07 -05:00
Joseph Schorr
5225642850
Garbage collection image+storage callback support
...
Add support to GC to invoke a callback with the image+storages removed. Only images whose storage was also removed will be sent to the callback. This will be used by security scanning for its own GC in the followup change.
2016-12-22 14:27:42 -05:00
Joseph Schorr
855cc36057
Remove unneeded imports
2016-08-11 17:16:31 -04:00
Joseph Schorr
9e4f8cac03
Optimize GC query for looking up deletable storages
2016-07-26 13:47:15 -07:00
Joseph Schorr
4d6f96cd6c
Add missing pass keyword
2016-07-19 22:24:27 -04:00
Joseph Schorr
b8d2570725
Don't raise an error on duplicate placements
...
This can happen if two pushes are racing on the same storage.
2016-07-19 16:44:05 -04:00
josephschorr
bda5d7ae29
Merge pull request #1511 from coreos-inc/location-cache
...
Use a cache for ImageStorageLocation
2016-06-09 14:03:07 -04:00
Joseph Schorr
7aa6b812e2
Use a cache for ImageStorageLocation
...
No need to reload it from the DB or join as it is a static set only changed during migration
2016-06-09 14:02:42 -04:00
Joseph Schorr
12924784ce
Enable storage replication for V2 and add backfill tool
...
Fixes #1501
2016-06-02 14:36:08 -04:00
Jake Moshenko
1b392dcb9a
Remove dependent signatures before removing image storages
2016-01-19 14:56:02 -05:00
Jake Moshenko
7d0be20842
Formatting and syntax improvements
2016-01-19 14:56:02 -05:00
Joseph Schorr
e4da61a05d
Fix piece hash calculation
2016-01-12 17:44:19 -05:00
Jake Moshenko
96c72e73df
Clean up torrents before removing referenced storages
2016-01-12 11:43:07 -05:00
Joseph Schorr
c36a7c21c8
Order sadly matters with this check in peewee
2016-01-11 15:10:46 -05:00
Jake Moshenko
1ae101c917
Address torrent feature review comments.
2016-01-08 16:38:21 -05:00
Jake Moshenko
fd1e5f2407
Remove an unnecessary outer join
2016-01-05 14:43:40 -05:00
Jake Moshenko
8f80d7064b
Hash v1 uploads for torrent chunks
2016-01-05 14:43:40 -05:00
Jimmy Zelinskie
9b0a84c02f
implement get_torrent_info
2016-01-04 16:17:51 -05:00
Jake Moshenko
fe87d3c796
Hash and track layer file chunks for torrenting
2016-01-04 16:17:51 -05:00
Jake Moshenko
9c1a2e7e1b
Improve performance by removing unnecessary group by fields
2015-12-22 11:35:49 -05:00
Joseph Schorr
f59f4e51e8
Remove check for derived image storages on image storage
...
Derived image storages are now 1-to-1 with image storages, so we know they have already been removed at this point
Fixes #1067
2015-12-16 13:41:25 -05:00
Joseph Schorr
f07b940bc5
Optimize blob lookup
...
Fixes #1013
2015-12-04 14:47:09 -05:00
Joseph Schorr
762cd56e64
Change derived storage to be based on image
...
Fixes #971
2015-11-24 12:44:07 -05:00
Jake Moshenko
3a29dfc535
Reducing in a tree to avoid recursion depth limits
2015-11-23 15:57:13 -05:00
Joseph Schorr
f4266d08d2
Fix handling of aggregate size in V2
...
Fixes #931
2015-11-20 11:44:03 -05:00
Jake Moshenko
7ae94f414c
Alias our subqueries to appease the MySQL beast
2015-11-19 12:58:06 -05:00
Jake Moshenko
b564492ea7
Improve the performance of fetching manifest blobs by checksum.
2015-11-19 11:01:47 -05:00
Quentin Machu
605ed1fc77
Refactor security worker
2015-11-18 14:38:32 -05:00
Jake Moshenko
52125bbfed
Fix gc by using the v1/v2 storage location helper everywhere
2015-11-16 14:13:37 -05:00
Jake Moshenko
c2fcf8bead
Merge remote-tracking branch 'upstream/phase4-11-07-2015' into python-registry-v2
2015-11-06 18:18:29 -05:00
Jake Moshenko
cb7ec2f239
Backport remaining v2 changes to phase4
2015-10-24 15:00:13 -04:00
Joseph Schorr
35c35d9913
Load images and storage references in bulk during V1 synthesize
...
Currently, we perform multiple queries for each layer, making it much slower (especially cross-region)
Fixes #413
2015-09-29 17:53:39 -04:00
Jake Moshenko
210ed7cf02
Merge remote-tracking branch 'upstream/master' into python-registry-v2
2015-09-04 16:32:01 -04:00
Joseph Schorr
724b1607d7
Add automatic storage replication
...
Adds a worker to automatically replicate data between storages and update the database accordingly
2015-09-01 14:53:32 -04:00
Joseph Schorr
1450b7e84c
Fix verbs support in V2
2015-08-24 12:05:09 -04:00
