vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
9228 commits 2 branches 62 tags 205 MiB
Commit graph

1698 commits

Author SHA1 Message Date
Sida Chen
ac51954e1f
Merge pull request #3402 from KeyboardNerd/QUAY-1358 
Change pg_trgm to be a precondition of pgsql database
 2019-03-12 16:24:04 -04:00
Joseph Schorr
ac566c7054 Fix NPE when setting tag to not expire 2019-03-11 12:23:51 -04:00
Joseph Schorr
2027d13d2b Change tag history call in OCI to not load the contents of the manifest 
Just in case that load is slow on the DB
 2019-03-08 14:59:33 -05:00
Joseph Schorr
0d6343871e Change the tags query used for OCI to list tags to be shallow 
This removes the join on Manifest which was (possibly) causing the prod issue. We also simplify the lookup for pre-OCI as well, a bit.
 2019-03-08 14:07:30 -05:00
Sida Chen
9533dc2981 Remove create extension query from migration 2019-03-08 10:28:05 -05:00
Sida Chen
7985167411 Add precondition check for database validation 2019-03-08 10:28:05 -05:00
Joseph Schorr
96072a44c0 Fix ordering of tags in the OCI model 
Before this change, we were neglecting to sort the tags by ID, which meant that pagination was broken
 2019-03-07 12:53:02 -05:00
Joseph Schorr
bdae32630e Fix V1 push for layers already uploaded 2019-03-05 12:13:01 -05:00
Joseph Schorr
9d3543a353
Merge pull request #3315 from quay/joseph.schorr/QUAY-1266/manifestlist-validation 
Add architecture validation to manifest lists that contain schema 1 manifests
 2019-02-27 18:32:21 -05:00
Joseph Schorr
91e9fe8050 Optimize the logs archiving worker to not issue extremely long queries 
Fixes https://jira.coreos.com/browse/QUAY-1371
 2019-02-26 14:37:42 -05:00
Joseph Schorr
f0f2d9cdf4 Add architecture validation to manifest lists that contain schema 1 manifests 
Fixes https://jira.coreos.com/browse/QUAY-1266
 2019-02-22 13:39:54 -05:00
Joseph Schorr
b5a5ce7c43
Merge pull request #3380 from quay/joseph.schorr/QUAY-1353/always-new-tag-manifest 
Always create a new tag manifest row when backfilling in the old data model
 2019-02-20 17:20:32 -05:00
Joseph Schorr
77fa28a749 Always create a new tag manifest row when backfilling in the old data model 
This is temporary anyway, but it will allow us to finish the backfill without having to worry about newer tags being displayed with just their V1IDs

Fixes https://jira.coreos.com/browse/QUAY-1353
 2019-02-20 16:04:31 -05:00
Joseph Schorr
6e36138251
Merge pull request #3359 from quay/logs-interface-tests 
Add tests for the logs model interface
 2019-02-20 15:06:12 -05:00
Joseph Schorr
d763361069
Merge pull request #3378 from quay/joseph.schorr/QUAY-1351/fix-super-large-manifest-backfill 
Fix backfills of super large manifests by stripping metadata from all but the final layer
 2019-02-20 15:00:13 -05:00
Joseph Schorr
693e47e063 Add additional assertions and further checks in storage code 2019-02-20 12:12:56 -05:00
Joseph Schorr
bacf074219 Fix backfills of super large manifests by stripping metadata from all but the final layer 
This is semantically valid because Docker only uses the leaf layer as the image config when reading a V2_1 manifest

Fixes https://jira.coreos.com/browse/QUAY-1351
 2019-02-20 12:08:20 -05:00
Joseph Schorr
77d7600b64 Add ability for proportional rollout of the OCI data model 2019-02-18 16:34:43 -05:00
Joseph Schorr
2cebf828a2
Merge pull request #3368 from quay/security-scan-set-opt 
Change the security scan update query to only change the indexed row
 2019-02-18 11:59:45 -05:00
Joseph Schorr
59e4896dbf Make absolutely sure we don't overwrite an existing storage checksum 
Old V1 code would occasionally do so, which was never correct, but we never hit it because we didn't (practically) share storage rows before. Now that we explicitly do, we were occasionally de-checksum-ing storage rows on V1 pushes (which are extremely rare as-is). This change makes sure we don't do that, and makes sure we always set a proper digest and checksum on storage rows.
 2019-02-16 12:40:53 -05:00
Joseph Schorr
4a22a52aee Make sure to delete the image placements in the manifest builder cleanup code 2019-02-16 00:28:27 -05:00
Joseph Schorr
d878232070 Make sure the manifest builder only deletes temporary storage rows it created 
We were accidentally deleting other image storage rows in the repository, which is really, really bad
 2019-02-15 21:18:02 -05:00
Joseph Schorr
3c526c957a Catch exceptions if trying to build an invalid manifest in the manifest backfill code 2019-02-15 21:17:33 -05:00
Joseph Schorr
10ee6be442 Change the security scan update query to only change the indexed row 
This will mean we try to re-index other similar rows, but that's a no-op on the Clair side, and this update query is extremely heavy and easily interleaving locks
 2019-02-15 18:13:42 -05:00
Joseph Schorr
f4a1646a8b Add an index on the logs_archived boolean on a RepositoryBuild 
Addresses the slow query:

```
SELECT `candidates` . `id` FROM ( SELECT `t1` . `id` FROM `repositorybuild` AS `t1` WHERE ( ( ( `t1` . `phase` IN (...) ) OR ( `t1` . `started` < ? ) ) AND ( `t1` . `logs_archived` = ? ) ) LIMIT ? ) AS `candidates` ORDER BY `Rand` ( ) LIMIT ? OFFSET ?
```

While the cardinality on `logs_archived` will be low, it should also only be in the `false` state for a very small number of records, so it should make this query very, very fast.
 2019-02-15 16:11:25 -05:00
Joseph Schorr
c4f7b28dc6
Merge pull request #3364 from quay/appr-search-fix 
Extend appr search to return private repositories
 2019-02-15 11:24:08 -05:00
Joseph Schorr
f8b19e798f Change appr search to return private repos when logged in 2019-02-14 17:39:40 -05:00
Joseph Schorr
331a7a640b Add some missing indexes that should make queries significantly faster 2019-02-14 17:23:30 -05:00
Sida Chen
ea9707b969
Merge pull request #3361 from KeyboardNerd/fix_es_test 
Fix elasticsearch test
 2019-02-14 15:17:49 -05:00
Joseph Schorr
7688a3e348 Make sure _lookup_repo_storages_by_content_checksum operates over a set 2019-02-14 14:43:23 -05:00
Sida Chen
2e7725cfa4 Fix elasticsearch test 
dictionary object was serialized in random key order, fix that by
comparing the deserialized dict directly.
 2019-02-14 14:19:23 -05:00
Joseph Schorr
f75f315037 Optimize lookup of shared global blobs 
Currently, we only have one (the shared empty layer), but this should make the blob lookups for repositories significantly faster, as we won't need to do the massive join.
 2019-02-14 12:46:42 -05:00
Joseph Schorr
7beac643ec Add an additional test for lookup repo blobs by digest 2019-02-14 11:22:06 -05:00
Joseph Schorr
ec35750f2c Add a lookup test for get_or_create_shared_blob 2019-02-14 10:27:34 -05:00
Joseph Schorr
20b583fd01 Skip flaky tests on MySQL 2019-02-13 17:08:31 -05:00
Joseph Schorr
f917de8c70
Merge pull request #3357 from quay/joseph.schorr/QUAY-1327/dont-scan-uploading 
Make sure images in the process of being uploaded are not scanned by Clair
 2019-02-13 16:04:34 -05:00
Sida Chen
d67bbd4917
Merge pull request #3352 from kleesc/elastic_logs_model_implementation 
Implement logs model using Elasticsearch
 2019-02-13 15:51:13 -05:00
Joseph Schorr
ba69b9c002 Add tests for the logs model interface 2019-02-13 14:45:39 -05:00
Kenny Lee Sin Cheong
035541c6f2 Implement logs model using Elasticsearch 
- Implement logs model using Elasticsearch with tests
- Implement transition model using both elasticsearch and database model
- Add LOGS_MODEL configuration to choose which to use.

Co-authored-by: Sida Chen <sidchen@redhat.com>
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
 2019-02-12 16:33:42 -05:00
Joseph Schorr
0b706bc7c4 Make sure images in the process of being uploaded are not scanned by Clair 2019-02-08 16:20:44 -05:00
Joseph Schorr
0bfbccdd44 Add a handler for broken tags in the tag backfill system 
This will generate a tag pointing to an empty manifest; the tag will be broken, but as it is *already* broken, at least the backfill can complete
 2019-02-07 13:36:47 -05:00
Joseph Schorr
bbaee0d3f2 Add additional logging for missing parent images 2019-02-06 17:34:18 -05:00
Joseph Schorr
82897a2bba Add a log for an integrity error in writing a manifest 2019-02-06 11:41:50 -05:00
Joseph Schorr
0d1ac7a4bb Small optimization in manifest loading in the OCI data model 2019-01-29 13:11:48 -05:00
Joseph Schorr
ec9913326a Change bulk logs downloader to use datetime ranges as well 2019-01-28 18:15:55 -05:00
Joseph Schorr
d4c74bc1d3 Optimize listing of logs by changing to bucket by datetime, over which we have an index 2019-01-28 17:45:45 -05:00
Joseph Schorr
9f09d68ad8
Merge pull request #3323 from quay/joseph.schorr/QUAY-1282/log-interfacing 
Interface out all action log data model operations
 2019-01-28 15:09:25 -05:00
Joseph Schorr
b773a18ed8 Interface out all action log data model operations 
This will allow us to reimplement the logs data model against a non-database system in the near future
 2019-01-25 15:52:22 -05:00
Joseph Schorr
bbfb0211dc Fix bug where we weren't properly caching enum values and we were always looking up the media type when constructing manifests 2019-01-22 14:46:38 -05:00
Joseph Schorr
0035a1ec6d Fix a bug in manifest backfill that made it only work on active tags 2019-01-21 16:59:37 -05:00