vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9129 commits 2 branches 62 tags 205 MiB
Commit graph

1662 commits

Author SHA1 Message Date
Joseph Schorr
f917de8c70
Merge pull request #3357 from quay/joseph.schorr/QUAY-1327/dont-scan-uploading 
Make sure images in the process of being uploaded are not scanned by Clair
 2019-02-13 16:04:34 -05:00
Sida Chen
d67bbd4917
Merge pull request #3352 from kleesc/elastic_logs_model_implementation 
Implement logs model using Elasticsearch
 2019-02-13 15:51:13 -05:00
Kenny Lee Sin Cheong
035541c6f2 Implement logs model using Elasticsearch 
- Implement logs model using Elasticsearch with tests
- Implement transition model using both elasticsearch and database model
- Add LOGS_MODEL configuration to choose which to use.

Co-authored-by: Sida Chen <sidchen@redhat.com>
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
 2019-02-12 16:33:42 -05:00
Joseph Schorr
0b706bc7c4 Make sure images in the process of being uploaded are not scanned by Clair 2019-02-08 16:20:44 -05:00
Joseph Schorr
0bfbccdd44 Add a handler for broken tags in the tag backfill system 
This will generate a tag pointing to an empty manifest; the tag will be broken, but as it is *already* broken, at least the backfill can complete
 2019-02-07 13:36:47 -05:00
Joseph Schorr
bbaee0d3f2 Add additional logging for missing parent images 2019-02-06 17:34:18 -05:00
Joseph Schorr
82897a2bba Add a log for an integrity error in writing a manifest 2019-02-06 11:41:50 -05:00
Joseph Schorr
0d1ac7a4bb Small optimization in manifest loading in the OCI data model 2019-01-29 13:11:48 -05:00
Joseph Schorr
ec9913326a Change bulk logs downloader to use datetime ranges as well 2019-01-28 18:15:55 -05:00
Joseph Schorr
d4c74bc1d3 Optimize listing of logs by changing to bucket by datetime, over which we have an index 2019-01-28 17:45:45 -05:00
Joseph Schorr
9f09d68ad8
Merge pull request #3323 from quay/joseph.schorr/QUAY-1282/log-interfacing 
Interface out all action log data model operations
 2019-01-28 15:09:25 -05:00
Joseph Schorr
b773a18ed8 Interface out all action log data model operations 
This will allow us to reimplement the logs data model against a non-database system in the near future
 2019-01-25 15:52:22 -05:00
Joseph Schorr
bbfb0211dc Fix bug where we weren't properly caching enum values and we were always looking up the media type when constructing manifests 2019-01-22 14:46:38 -05:00
Joseph Schorr
0035a1ec6d Fix a bug in manifest backfill that made it only work on active tags 2019-01-21 16:59:37 -05:00
Joseph Schorr
7c241a93f6 Allow namespaces to be whitelisted for OCI, but not V22 2019-01-21 14:40:26 -05:00
Joseph Schorr
570d974067 Add support for retargeting a tag to all schema 1 manifests 
Schema version 1 manifests contain the tag name, and we have a check to ensure we don't point a tag at a manifest with the wrong name embedded. However, this also means that we cannot retarget to that manifest, which will break the UI once we get rid of legacy images.

This change means we can retarget to those manifests, and the OCI model does the work of rewriting the manifest when necessary.
 2019-01-15 16:00:06 -05:00
Joseph Schorr
fe45797490 Optimize the list active tags query 
Some customers are hitting this endpoint rapidly for repositories with many, many tags. This change drops the unnecessary joins, which should reduce database load somewhat.
 2019-01-14 13:34:25 -05:00
Joseph Schorr
eb9ca8e8a8 Fix handling of four byte utf8 manifests 
- Adds the charset: utf-8 to all the manifest responses
- Makes sure we connect to MySQL in utf8mb4 mode, to ensure we can properly read and write 4-byte utf8 strings
- Adds tests for all of the above
 2019-01-10 16:34:56 -05:00
Joseph Schorr
b1dd053b02 Fix an NPE when trying to pull a manifest without a legacy image via V1 2019-01-10 13:59:35 -05:00
Joseph Schorr
171c7e5238 Further fixes for unicode handling in manifests 
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
 2019-01-09 15:14:41 -05:00
Joseph Schorr
b7b64d84d3 Disable verification of schemas temporarily while we determine why some older schemas don't validate 2019-01-08 14:25:35 -05:00
Joseph Schorr
c510759df3 Remove the foreign key constraint from the kind field on the logentry3 table 2019-01-04 13:35:14 -05:00
Joseph Schorr
e309508776 Change lookup of blobs to not use a subquery, which is murder on MySQL 2019-01-03 15:41:21 -05:00
Joseph Schorr
cdb49dbfd3 Add LogEntry3 table without the extra indexes and switch to writing to it 2019-01-03 13:50:43 -05:00
Joseph Schorr
b6db002729 Fix references to LogEntry model used and add support for a maximum page of results on the logs API 2019-01-02 16:04:15 -05:00
Joseph Schorr
4ba4d9141b Put user last accessed updating behind a feature flag 2019-01-02 16:04:15 -05:00
Joseph Schorr
a6b5b4d6e3 Switch to writing logs to LogEntry2 table 2019-01-02 16:04:15 -05:00
Joseph Schorr
58d7dd07b9 Revert the changes to drop LogEntry2 table definition 2019-01-02 13:29:35 -05:00
Joseph Schorr
442312402f
Merge pull request #3299 from quay/joseph.schorr/QUAY-1233/batch-logs-download 
Batch download of logs
 2018-12-19 15:38:22 -05:00
Joseph Schorr
9190c046c7 Remove unusable sitemap endpoint 
It calls a data model method which is extremely slow and heavy, and it isn't even used anyway
 2018-12-18 16:37:09 -05:00
Joseph Schorr
8a212728a3 Implement a worker for batch exporting of usage logs 
This will allow customers to request their usage logs for a repository or an entire namespace, and we can export the logs in a manner that doesn't absolutely destroy the database, with every step along the way timed.
 2018-12-18 15:33:03 -05:00
Joseph Schorr
9e2c5417b8
Merge pull request #3310 from quay/joseph.schorr/QUAY-1235/drop-logentry2 
Remove all references to LogEntry2 from the code
 2018-12-18 15:18:22 -05:00
Joseph Schorr
fc691cefb4
Merge pull request #3306 from quay/fix-manifest-ui 
Fix manifest UI page to properly show the layers of manifests and show manifest lists
 2018-12-11 16:53:36 -05:00
Joseph Schorr
6a94eba1a2 Remove all references to LogEntry2 from the code 
The migration to actually drop the table will go in after this code has been pushed to production
 2018-12-11 15:54:16 -05:00
Joseph Schorr
a9b8586423 Make sure list_manifest_layers is not callable for manifest lists 2018-12-11 13:10:24 -05:00
Joseph Schorr
7fa60d5802 Fix manifest UI display for manifests with empty layers 
Also fixes the command for schema 2 manifests
 2018-12-10 17:10:54 -05:00
Joseph Schorr
eb7591183d Add a tag backfill worker to fully backfill the new-style Tag's in the background 2018-12-10 15:36:25 -05:00
Joseph Schorr
4106f5ce51 Fix manifest UI page to properly show the layers of manifests and show manifest lists 2018-12-10 15:33:59 -05:00
Joseph Schorr
3c2e050593 Support pulling of schema2 manifests directly via a manifest list tag 
This change ensures that if a manifest list is requested with an accepts header for a *schema 2* manifest, the legacy manifest (if any) is returned as schema 2 if it was pushed as a schema 2 manifest (rather than being auto-converted to schema 1)
 2018-12-07 11:57:13 -05:00
Joseph Schorr
c3710a6a5e Add ability for specific geographic regions to be blocked from pulling images within a namespace 2018-12-06 17:14:45 -05:00
Joseph Schorr
d59bea3569 Enable a configurable whitelist of namespaces for V22 
If a namespace is present in the whitelist, all calls are sent to the OCI model instead of the Pre OCI model

Note that this does increase overhead for registry calls (since we need to lookup the namespace for every single call), but it should only be temporary until we've migrated all users over to the OCI data model
 2018-12-03 17:37:28 -05:00
Joseph Schorr
492934de3c Remove redundant call 2018-11-27 18:44:34 +02:00
Joseph Schorr
4e1ff90cb2 Unify the get_layers calls across all implements of manifest schemas to ensure we have a common type returned 
Also renames some methods to make it more clear what kind of information they return
 2018-11-26 17:58:48 +02:00
Joseph Schorr
180d8847db Fix image replication for images with remote layers 2018-11-26 16:15:48 +02:00
Joseph Schorr
adccdd30ca Read blobs from new manifest blob table where relevant 2018-11-25 17:31:09 +02:00
Joseph Schorr
4985040d31 Properly handle the empty layer when pushing schema 2 manifests 
Docker doesn't send us the contents of this layer, so we are forced to synthesize it ourselves
 2018-11-25 16:16:59 +02:00
Joseph Schorr
ffd567e2c2 Rename accessed items in the data model 2018-11-21 18:06:16 +02:00
Joseph Schorr
1f03fdb27e Fix pulling of squashed versions of the legacy image in a manifest lists 2018-11-20 16:37:15 +02:00
Joseph Schorr
001768c043 Make security status for manifests without legacy images more clear in the UI 2018-11-20 16:37:15 +02:00
Joseph Schorr
e972e4088b Fix lookup of manifests referenced solely by a manifest list 
We need to ensure we can find them if there is an active tag pointing to the parent list
 2018-11-19 23:31:41 +02:00