vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9132 commits 2 branches 62 tags 205 MiB
Commit graph

397 commits

Author SHA1 Message Date
Sida Chen
d67bbd4917
Merge pull request #3352 from kleesc/elastic_logs_model_implementation 
Implement logs model using Elasticsearch
 2019-02-13 15:51:13 -05:00
Kenny Lee Sin Cheong
035541c6f2 Implement logs model using Elasticsearch 
- Implement logs model using Elasticsearch with tests
- Implement transition model using both elasticsearch and database model
- Add LOGS_MODEL configuration to choose which to use.

Co-authored-by: Sida Chen <sidchen@redhat.com>
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
 2019-02-12 16:33:42 -05:00
Joseph Schorr
8d722dee81
Merge pull request #3356 from quay/tag-backfill-skip-improvements 
Tag backfill improvements
 2019-02-07 13:52:32 -05:00
Joseph Schorr
0bfbccdd44 Add a handler for broken tags in the tag backfill system 
This will generate a tag pointing to an empty manifest; the tag will be broken, but as it is *already* broken, at least the backfill can complete
 2019-02-07 13:36:47 -05:00
Joseph Schorr
f8dd8b2494 Have tag backfill worker continue processing ranges that have errors 2019-02-06 11:41:50 -05:00
Tom McKay
56ceef823a 1316 - exportactionlogsworker period to 60sec 2019-02-05 16:06:29 -05:00
Joseph Schorr
9f09d68ad8
Merge pull request #3323 from quay/joseph.schorr/QUAY-1282/log-interfacing 
Interface out all action log data model operations
 2019-01-28 15:09:25 -05:00
Joseph Schorr
b773a18ed8 Interface out all action log data model operations 
This will allow us to reimplement the logs data model against a non-database system in the near future
 2019-01-25 15:52:22 -05:00
Joseph Schorr
4e9949f69d Fix a bug in the tag backfill that prevented copying lifetime_start_ts when it is zero 2019-01-23 15:28:05 -05:00
Joseph Schorr
74bc161b2e Add an additional validation test to the tag backfill worker tests 2019-01-21 14:06:30 -05:00
Joseph Schorr
c1ac12ff8d Some small cleanup to the tag backfill worker 2019-01-21 13:37:11 -05:00
Joseph Schorr
92e6e06e90 Make the timeout for the tags backfill worker configurable 2019-01-16 13:30:36 -05:00
Joseph Schorr
1cda16f055 Add additional logs 2019-01-16 11:39:57 -05:00
Joseph Schorr
0f18b3a70a Allow filtering of the backfill by namespace 2019-01-16 11:36:09 -05:00
Joseph Schorr
2a4646b1ca Add extra verification that backfilling hits all tags in a repository 2019-01-16 11:30:01 -05:00
Joseph Schorr
d49fd1c727 Make sure we backfilled at least one dead tag 2019-01-16 11:24:33 -05:00
Joseph Schorr
b8d60152c4 A few small improvements to the tag backfill worker 
1) Remove the join on the min ID lookup. This join is incredibly slow and taxing on the database, so we simply set the minimum to the min database ID.
2) Increase the timeout on the worker.
3) Have the manifest backfill verify the contents of the backfilled manifests, to ensure we didn't mis-copy bytes due to the previous unicode issues
 2019-01-14 13:43:42 -05:00
Joseph Schorr
171c7e5238 Further fixes for unicode handling in manifests 
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
 2019-01-09 15:14:41 -05:00
Joseph Schorr
cdb49dbfd3 Add LogEntry3 table without the extra indexes and switch to writing to it 2019-01-03 13:50:43 -05:00
Joseph Schorr
b6db002729 Fix references to LogEntry model used and add support for a maximum page of results on the logs API 2019-01-02 16:04:15 -05:00
Joseph Schorr
204eb74c4f Put aggregated log query and log exports behind feature flags 2019-01-02 16:04:15 -05:00
Joseph Schorr
58d7dd07b9 Revert the changes to drop LogEntry2 table definition 2019-01-02 13:29:35 -05:00
Joseph Schorr
8a212728a3 Implement a worker for batch exporting of usage logs 
This will allow customers to request their usage logs for a repository or an entire namespace, and we can export the logs in a manner that doesn't absolutely destroy the database, with every step along the way timed.
 2018-12-18 15:33:03 -05:00
Joseph Schorr
6a94eba1a2 Remove all references to LogEntry2 from the code 
The migration to actually drop the table will go in after this code has been pushed to production
 2018-12-11 15:54:16 -05:00
Joseph Schorr
d734adb8c3 Add missing method 2018-12-10 16:56:19 -05:00
Joseph Schorr
6963107ec6 Fix after merge 2018-12-10 15:37:26 -05:00
Joseph Schorr
eb7591183d Add a tag backfill worker to fully backfill the new-style Tag's in the background 2018-12-10 15:36:25 -05:00
Joseph Schorr
57e93a82c9 Remove manifest backfill worker 2018-12-10 15:36:25 -05:00
Joseph Schorr
3c2e050593 Support pulling of schema2 manifests directly via a manifest list tag 
This change ensures that if a manifest list is requested with an accepts header for a *schema 2* manifest, the legacy manifest (if any) is returned as schema 2 if it was pushed as a schema 2 manifest (rather than being auto-converted to schema 1)
 2018-12-07 11:57:13 -05:00
Joseph Schorr
cdfaca8e78 Disable GC worker in its entirety 2018-12-03 10:42:53 -05:00
Joseph Schorr
4e1ff90cb2 Unify the get_layers calls across all implements of manifest schemas to ensure we have a common type returned 
Also renames some methods to make it more clear what kind of information they return
 2018-11-26 17:58:48 +02:00
Joseph Schorr
adccdd30ca Read blobs from new manifest blob table where relevant 2018-11-25 17:31:09 +02:00
Joseph Schorr
c233760007 Fix conversion of schema 2 manifests to schema 1 manifests 
Also adds a number of conversion tests and clarify the interfaces a bit more
 2018-11-21 18:06:16 +02:00
Joseph Schorr
1eaf5b18dd Adjustments based on code review feedback 2018-11-15 13:51:48 +02:00
Joseph Schorr
c46b11bac1 Implement UI support for manifest lists 2018-11-14 18:40:57 +02:00
Joseph Schorr
37b20010aa Add support for pushing and pulling schema 2 manifests with remote layers 
This is required for windows image support
 2018-11-14 13:21:50 +02:00
Joseph Schorr
d97055e2ba Code review fixes 2018-11-14 09:15:58 +02:00
Joseph Schorr
e344d4a5cf Enhancements for Docker schema implementations in preparing for supporting schema 2 in the OCI model 
This adds additional required properties and methods to the Docker schema interface to allow us to treat both schema1 and schema2 manifests and lists logically equivalent from the OCI mode perspective
 2018-11-12 23:27:01 +02:00
Joseph Schorr
114e2c3bf2 Have all tag code add, modify and delete both old and new style tags 
This preps us for being able to use the new data model with existing repositories
 2018-11-01 17:59:10 -04:00
Joseph Schorr
6a9634dffb Harden the storage replication worker to failures by explicitly catching certain errors, having better backoff and explicitly ensuring a layer has been replicated to a region before adding the placement into the database 
Also adds tests for the various failure cases
 2018-10-31 14:13:25 -04:00
Joseph Schorr
0ae062be62 Add manifest creation to new registry data model interface 2018-10-18 14:25:39 -04:00
Joseph Schorr
aeceea0f97 Add a worker for backfilling labels on manifests that have already been backfilled 2018-09-26 14:55:14 -04:00
Joseph Schorr
a4fff886a6 Handle the case where the log metadata JSON cannot be parsed 
This can happen if the JSON overflowed the text field in the table, for example

Fixes https://sentry.io/coreos/backend-production/issues/670957592/
 2018-09-07 13:34:22 -04:00
Joseph Schorr
a04658085b Add configuration flag to control whether we backfill manifests 
This will allow us to easily turn off the backfill once it is initially complete, but also reenable it if necessary
 2018-08-27 17:22:43 -04:00
Joseph Schorr
ed897626a2 Handle data model exceptions when loading parent images in manifest backfill 2018-08-20 11:32:59 -04:00
Joseph Schorr
e30b746aef Fix TagManifests with shared digests under the same repository. 
TagManifests can (apparently, in very rare scenarios) share manifests with the exact same digests, so we need to support that case in the backfill worker. We also need to remove a unique constraint on the manifest column in the mapping table to support this case.
 2018-08-20 11:32:59 -04:00
Joseph Schorr
fa58f3b1d2 Fix handling of manifests with unicode in the backfill 
Also adds a bunch of tests around manifests to ensure we get the same information in and out
 2018-08-15 11:41:15 -04:00
Joseph Schorr
70df10dade Change manifest backfill to lookup missing content checksums in the repository 
After discussion, we decided the best solution for the missing content checksum problem was to lookup the proper blobs in the repository and, if not present, mark the manifest as broken, as this would reflect the actual issue the user faces if they pull the repository tag today via V2
 2018-08-14 14:36:19 -04:00
Joseph Schorr
96e0fc4ad6 Fix manifest backfill for manifests pointing to V1 images 
V1 images don't have checksums, so we just always use the full storage set for the manifest, rather than a checksum map
 2018-08-13 15:51:18 -04:00
Joseph Schorr
f2d50b3f8e Add run commands for backfill worker 2018-08-13 14:56:32 -04:00