Commit graph

13 commits

Author SHA1 Message Date
Joseph Schorr
6a9634dffb Harden the storage replication worker to failures by explicitly catching certain errors, having better backoff and explicitly ensuring a layer has been replicated to a region before adding the placement into the database
Also adds tests for the various failure cases
2018-10-31 14:13:25 -04:00
Joseph Schorr
52ddf6a308 Fix exception in storage replication worker
Fixes https://jira.coreos.com/browse/QUAY-957
2018-05-31 13:58:04 -04:00
Joseph Schorr
7b72cf8b27 Small fix for georeplication and add better logs
Previously, if we attempted to georeplicate storage from the existing location and, somehow, that existing location did not exist, we'd still mark the new location as invalid. This is a major problem for storage engines that are not consistent. Now, we first try a back off strategy to find the image in the existing storage and, as well, if the replication fails in any way, we log it.
2017-06-23 17:07:05 -04:00
Antoine Legrand
3c99928a27 Add log JSON formatter 2017-06-07 00:02:52 +02:00
Joseph Schorr
b05ebbf2c0 Have storage replication wait up to 20 minutes before trying again
Copying a file can be a long operation, so make this configurable and far above the default 5 minutes
2017-03-21 16:58:36 -04:00
Joseph Schorr
e6ee538e15 Fix full database test script to not fail randomly
- Switches database schema creation to alembic, which solves the MySQL issue (and makes sure we test migrations as well)
- Adds a few time.sleep(1) to work around MySQL's second-precision issue when adding items to queues and then immediately retrieving them
- Disables the storage proxy tests when running against non-SQLite databases, as it causes failures with the multiple process and multiple transactions
- Changes initdb to support only populating the database, as well as fixing a few small items around the test data when working with non-SQLite data
2016-11-30 18:24:08 -05:00
Joseph Schorr
b8d2570725 Don't raise an error on duplicate placements
This can happen if two pushes are racing on the same storage.
2016-07-19 16:44:05 -04:00
Joseph Schorr
5cd793331e Fix storage replication for CAS and add tests 2016-07-12 13:46:06 -04:00
Jake Moshenko
2f626f2691 Unify the database connection lifecycle across all workers 2015-12-04 15:51:53 -05:00
Jake Moshenko
e7a6176594 Merge remote-tracking branch 'upstream/v2-phase4' into python-registry-v2 2015-10-22 16:59:28 -04:00
Jake Moshenko
ce94931540 Stop writing to deprecated columns for image data. 2015-10-22 12:14:39 -04:00
Silas Sewell
03f5fe6143 workers.storagereplication: error on LocalStorage
Ensure we don't start when LocalStorage is in the config.

Fixes #502
2015-10-20 19:04:31 -04:00
Joseph Schorr
724b1607d7 Add automatic storage replication
Adds a worker to automatically replicate data between storages and update the database accordingly
2015-09-01 14:53:32 -04:00