Commit graph

26 commits

Author SHA1 Message Date
Joseph Schorr
8d722dee81
Merge pull request #3356 from quay/tag-backfill-skip-improvements
Tag backfill improvements
2019-02-07 13:52:32 -05:00
Joseph Schorr
0bfbccdd44 Add a handler for broken tags in the tag backfill system
This will generate a tag pointing to an empty manifest; the tag will be broken, but as it is *already* broken, at least the backfill can complete
2019-02-07 13:36:47 -05:00
Tom McKay
56ceef823a 1316 - exportactionlogsworker period to 60sec 2019-02-05 16:06:29 -05:00
Joseph Schorr
9f09d68ad8
Merge pull request #3323 from quay/joseph.schorr/QUAY-1282/log-interfacing
Interface out all action log data model operations
2019-01-28 15:09:25 -05:00
Joseph Schorr
b773a18ed8 Interface out all action log data model operations
This will allow us to reimplement the logs data model against a non-database system in the near future
2019-01-25 15:52:22 -05:00
Joseph Schorr
74bc161b2e Add an additional validation test to the tag backfill worker tests 2019-01-21 14:06:30 -05:00
Joseph Schorr
2a4646b1ca Add extra verification that backfilling hits all tags in a repository 2019-01-16 11:30:01 -05:00
Joseph Schorr
d49fd1c727 Make sure we backfilled at least one dead tag 2019-01-16 11:24:33 -05:00
Joseph Schorr
171c7e5238 Further fixes for unicode handling in manifests
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
2019-01-09 15:14:41 -05:00
Joseph Schorr
cdb49dbfd3 Add LogEntry3 table without the extra indexes and switch to writing to it 2019-01-03 13:50:43 -05:00
Joseph Schorr
b6db002729 Fix references to LogEntry model used and add support for a maximum page of results on the logs API 2019-01-02 16:04:15 -05:00
Joseph Schorr
8a212728a3 Implement a worker for batch exporting of usage logs
This will allow customers to request their usage logs for a repository or an entire namespace, and we can export the logs in a manner that doesn't absolutely destroy the database, with every step along the way timed.
2018-12-18 15:33:03 -05:00
Joseph Schorr
eb7591183d Add a tag backfill worker to fully backfill the new-style Tag's in the background 2018-12-10 15:36:25 -05:00
Joseph Schorr
57e93a82c9 Remove manifest backfill worker 2018-12-10 15:36:25 -05:00
Joseph Schorr
114e2c3bf2 Have all tag code add, modify and delete both old and new style tags
This preps us for being able to use the new data model with existing repositories
2018-11-01 17:59:10 -04:00
Joseph Schorr
6a9634dffb Harden the storage replication worker to failures by explicitly catching certain errors, having better backoff and explicitly ensuring a layer has been replicated to a region before adding the placement into the database
Also adds tests for the various failure cases
2018-10-31 14:13:25 -04:00
Joseph Schorr
aeceea0f97 Add a worker for backfilling labels on manifests that have already been backfilled 2018-09-26 14:55:14 -04:00
Joseph Schorr
e30b746aef Fix TagManifests with shared digests under the same repository.
TagManifests can (apparently, in very rare scenarios) share manifests with the exact same digests, so we need to support that case in the backfill worker. We also need to remove a unique constraint on the manifest column in the mapping table to support this case.
2018-08-20 11:32:59 -04:00
Joseph Schorr
70df10dade Change manifest backfill to lookup missing content checksums in the repository
After discussion, we decided the best solution for the missing content checksum problem was to lookup the proper blobs in the repository and, if not present, mark the manifest as broken, as this would reflect the actual issue the user faces if they pull the repository tag today via V2
2018-08-14 14:36:19 -04:00
Joseph Schorr
96e0fc4ad6 Fix manifest backfill for manifests pointing to V1 images
V1 images don't have checksums, so we just always use the full storage set for the manifest, rather than a checksum map
2018-08-13 15:51:18 -04:00
Joseph Schorr
03ea3a3250 Add worker to backfill the new manifest tables from the tagmanifest table 2018-08-13 14:56:31 -04:00
Joseph Schorr
1a2e2f54e1 Move storagereplication test under the workers test directory 2018-07-19 15:56:20 -04:00
Joseph Schorr
cc09e8738e Remove extra whitespace 2017-04-24 17:04:09 -04:00
Joseph Schorr
7debd44b54 Switch fixture imports to wildcard in prep for full db test fixes 2017-04-24 16:45:14 -04:00
Joseph Schorr
80693d6b8c Fix NPE bug in RAC worker
We need to return `None`, not `0` if there are no additional repositories to measure
2017-04-11 15:42:11 -04:00
Joseph Schorr
df3f47c79a Add a RepositorySearchScore table and calculation to the RAC worker
This will be used in a followup PR to order search results instead of the RAC join. Currently, the join with the RAC table in search results in a lookup of ~600K rows, which causes searching to take ~6s. This PR denormalizes the data we need, as well as allowing us to score based on a wider band (6 months vs the current 1 week).
2017-04-10 14:29:02 -04:00