vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9192 commits 2 branches 62 tags 205 MiB
Commit graph

831 commits

Author SHA1 Message Date
Joseph Schorr
9d3543a353
Merge pull request #3315 from quay/joseph.schorr/QUAY-1266/manifestlist-validation 
Add architecture validation to manifest lists that contain schema 1 manifests
 2019-02-27 18:32:21 -05:00
Joseph Schorr
91e9fe8050 Optimize the logs archiving worker to not issue extremely long queries 
Fixes https://jira.coreos.com/browse/QUAY-1371
 2019-02-26 14:37:42 -05:00
Joseph Schorr
f0f2d9cdf4 Add architecture validation to manifest lists that contain schema 1 manifests 
Fixes https://jira.coreos.com/browse/QUAY-1266
 2019-02-22 13:39:54 -05:00
Joseph Schorr
77fa28a749 Always create a new tag manifest row when backfilling in the old data model 
This is temporary anyway, but it will allow us to finish the backfill without having to worry about newer tags being displayed with just their V1IDs

Fixes https://jira.coreos.com/browse/QUAY-1353
 2019-02-20 16:04:31 -05:00
Joseph Schorr
693e47e063 Add additional assertions and further checks in storage code 2019-02-20 12:12:56 -05:00
Joseph Schorr
2cebf828a2
Merge pull request #3368 from quay/security-scan-set-opt 
Change the security scan update query to only change the indexed row
 2019-02-18 11:59:45 -05:00
Joseph Schorr
59e4896dbf Make absolutely sure we don't overwrite an existing storage checksum 
Old V1 code would occasionally do so, which was never correct, but we never hit it because we didn't (practically) share storage rows before. Now that we explicitly do, we were occasionally de-checksum-ing storage rows on V1 pushes (which are extremely rare as-is). This change makes sure we don't do that, and makes sure we always set a proper digest and checksum on storage rows.
 2019-02-16 12:40:53 -05:00
Joseph Schorr
10ee6be442 Change the security scan update query to only change the indexed row 
This will mean we try to re-index other similar rows, but that's a no-op on the Clair side, and this update query is extremely heavy and easily interleaving locks
 2019-02-15 18:13:42 -05:00
Joseph Schorr
f75f315037 Optimize lookup of shared global blobs 
Currently, we only have one (the shared empty layer), but this should make the blob lookups for repositories significantly faster, as we won't need to do the massive join.
 2019-02-14 12:46:42 -05:00
Joseph Schorr
7beac643ec Add an additional test for lookup repo blobs by digest 2019-02-14 11:22:06 -05:00
Joseph Schorr
ec35750f2c Add a lookup test for get_or_create_shared_blob 2019-02-14 10:27:34 -05:00
Joseph Schorr
f917de8c70
Merge pull request #3357 from quay/joseph.schorr/QUAY-1327/dont-scan-uploading 
Make sure images in the process of being uploaded are not scanned by Clair
 2019-02-13 16:04:34 -05:00
Sida Chen
d67bbd4917
Merge pull request #3352 from kleesc/elastic_logs_model_implementation 
Implement logs model using Elasticsearch
 2019-02-13 15:51:13 -05:00
Kenny Lee Sin Cheong
035541c6f2 Implement logs model using Elasticsearch 
- Implement logs model using Elasticsearch with tests
- Implement transition model using both elasticsearch and database model
- Add LOGS_MODEL configuration to choose which to use.

Co-authored-by: Sida Chen <sidchen@redhat.com>
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
 2019-02-12 16:33:42 -05:00
Joseph Schorr
0b706bc7c4 Make sure images in the process of being uploaded are not scanned by Clair 2019-02-08 16:20:44 -05:00
Joseph Schorr
bbaee0d3f2 Add additional logging for missing parent images 2019-02-06 17:34:18 -05:00
Joseph Schorr
82897a2bba Add a log for an integrity error in writing a manifest 2019-02-06 11:41:50 -05:00
Joseph Schorr
d4c74bc1d3 Optimize listing of logs by changing to bucket by datetime, over which we have an index 2019-01-28 17:45:45 -05:00
Joseph Schorr
9f09d68ad8
Merge pull request #3323 from quay/joseph.schorr/QUAY-1282/log-interfacing 
Interface out all action log data model operations
 2019-01-28 15:09:25 -05:00
Joseph Schorr
b773a18ed8 Interface out all action log data model operations 
This will allow us to reimplement the logs data model against a non-database system in the near future
 2019-01-25 15:52:22 -05:00
Joseph Schorr
0035a1ec6d Fix a bug in manifest backfill that made it only work on active tags 2019-01-21 16:59:37 -05:00
Joseph Schorr
fe45797490 Optimize the list active tags query 
Some customers are hitting this endpoint rapidly for repositories with many, many tags. This change drops the unnecessary joins, which should reduce database load somewhat.
 2019-01-14 13:34:25 -05:00
Joseph Schorr
171c7e5238 Further fixes for unicode handling in manifests 
We were occasionally trying to compute schema 2 version 1 signatures on the *unicode* representation, which was failing the signature check. This PR adds a new wrapper type called `Bytes`, which all manifests must take in, and which handles the unicodes vs encoded utf-8 stuff in a central location. This PR also adds a test for the manifest that was breaking in production.
 2019-01-09 15:14:41 -05:00
Joseph Schorr
e309508776 Change lookup of blobs to not use a subquery, which is murder on MySQL 2019-01-03 15:41:21 -05:00
Joseph Schorr
cdb49dbfd3 Add LogEntry3 table without the extra indexes and switch to writing to it 2019-01-03 13:50:43 -05:00
Joseph Schorr
b6db002729 Fix references to LogEntry model used and add support for a maximum page of results on the logs API 2019-01-02 16:04:15 -05:00
Joseph Schorr
4ba4d9141b Put user last accessed updating behind a feature flag 2019-01-02 16:04:15 -05:00
Joseph Schorr
a6b5b4d6e3 Switch to writing logs to LogEntry2 table 2019-01-02 16:04:15 -05:00
Joseph Schorr
58d7dd07b9 Revert the changes to drop LogEntry2 table definition 2019-01-02 13:29:35 -05:00
Joseph Schorr
442312402f
Merge pull request #3299 from quay/joseph.schorr/QUAY-1233/batch-logs-download 
Batch download of logs
 2018-12-19 15:38:22 -05:00
Joseph Schorr
9190c046c7 Remove unusable sitemap endpoint 
It calls a data model method which is extremely slow and heavy, and it isn't even used anyway
 2018-12-18 16:37:09 -05:00
Joseph Schorr
8a212728a3 Implement a worker for batch exporting of usage logs 
This will allow customers to request their usage logs for a repository or an entire namespace, and we can export the logs in a manner that doesn't absolutely destroy the database, with every step along the way timed.
 2018-12-18 15:33:03 -05:00
Joseph Schorr
6a94eba1a2 Remove all references to LogEntry2 from the code 
The migration to actually drop the table will go in after this code has been pushed to production
 2018-12-11 15:54:16 -05:00
Joseph Schorr
c3710a6a5e Add ability for specific geographic regions to be blocked from pulling images within a namespace 2018-12-06 17:14:45 -05:00
Joseph Schorr
492934de3c Remove redundant call 2018-11-27 18:44:34 +02:00
Joseph Schorr
4e1ff90cb2 Unify the get_layers calls across all implements of manifest schemas to ensure we have a common type returned 
Also renames some methods to make it more clear what kind of information they return
 2018-11-26 17:58:48 +02:00
Joseph Schorr
adccdd30ca Read blobs from new manifest blob table where relevant 2018-11-25 17:31:09 +02:00
Joseph Schorr
4985040d31 Properly handle the empty layer when pushing schema 2 manifests 
Docker doesn't send us the contents of this layer, so we are forced to synthesize it ourselves
 2018-11-25 16:16:59 +02:00
Joseph Schorr
ffd567e2c2 Rename accessed items in the data model 2018-11-21 18:06:16 +02:00
Joseph Schorr
e972e4088b Fix lookup of manifests referenced solely by a manifest list 
We need to ensure we can find them if there is an active tag pointing to the parent list
 2018-11-19 23:31:41 +02:00
Joseph Schorr
54904cfd6e Fix bug around pushing manifest lists that refer to the same manifest twice as children 2018-11-19 18:26:22 +02:00
Joseph Schorr
e6c2ddfa93 Add support for direct pushing of schema 2 manifests without tags 
This is required for manifest lists
 2018-11-19 14:01:41 +02:00
Joseph Schorr
82ee21bfbd Change OCI data model to use a content retriever for the various manifest operations 2018-11-19 12:24:04 +02:00
Joseph Schorr
1eaf5b18dd Adjustments based on code review feedback 2018-11-15 13:51:48 +02:00
Joseph Schorr
37b20010aa Add support for pushing and pulling schema 2 manifests with remote layers 
This is required for windows image support
 2018-11-14 13:21:50 +02:00
Joseph Schorr
d97055e2ba Code review fixes 2018-11-14 09:15:58 +02:00
Joseph Schorr
7a794e29c0 Add tests for manifest lists and fix some issues encountered while testing 2018-11-13 21:03:20 +02:00
Joseph Schorr
9994f0ae61 Disable GC for repositories with manifest lists 
This is a temporary measure so we can roll V2_2 out for internal customer testing without having to worry about GC interacting oddly with it
 2018-11-13 21:03:20 +02:00
Joseph Schorr
849e613386 Implement support for schema 2 manifests 2018-11-13 11:49:12 +02:00
Joseph Schorr
30f072aeff Add support for creating schema 2 manifests and manifest lists via the OCI model 2018-11-12 23:27:49 +02:00