Commit graph

38 commits

Author SHA1 Message Date
Joseph Schorr
91e9fe8050 Optimize the logs archiving worker to not issue extremely long queries
Fixes https://jira.coreos.com/browse/QUAY-1371
2019-02-26 14:37:42 -05:00
Kenny Lee Sin Cheong
035541c6f2 Implement logs model using Elasticsearch
- Implement logs model using Elasticsearch with tests
- Implement transition model using both elasticsearch and database model
- Add LOGS_MODEL configuration to choose which to use.

Co-authored-by: Sida Chen <sidchen@redhat.com>
Co-authored-by: Kenny Lee Sin Cheong <kenny.lee@redhat.com>
2019-02-12 16:33:42 -05:00
Joseph Schorr
cdb49dbfd3 Add LogEntry3 table without the extra indexes and switch to writing to it 2019-01-03 13:50:43 -05:00
Joseph Schorr
b6db002729 Fix references to LogEntry model used and add support for a maximum page of results on the logs API 2019-01-02 16:04:15 -05:00
Joseph Schorr
a6b5b4d6e3 Switch to writing logs to LogEntry2 table 2019-01-02 16:04:15 -05:00
Joseph Schorr
58d7dd07b9 Revert the changes to drop LogEntry2 table definition 2019-01-02 13:29:35 -05:00
Joseph Schorr
8a212728a3 Implement a worker for batch exporting of usage logs
This will allow customers to request their usage logs for a repository or an entire namespace, and we can export the logs in a manner that doesn't absolutely destroy the database, with every step along the way timed.
2018-12-18 15:33:03 -05:00
Joseph Schorr
6a94eba1a2 Remove all references to LogEntry2 from the code
The migration to actually drop the table will go in after this code has been pushed to production
2018-12-11 15:54:16 -05:00
Joseph Schorr
7325b22c90 Change id column in LogEntry to a BigInt and migrate back to using that table 2018-07-30 12:42:56 -04:00
Brad Ison
d3d9cca182 Upgrade Peewee to latest 3.x
This requires a number of small changes in the data model code, as well as additional testing.
2018-07-23 12:43:23 -04:00
Joseph Schorr
a007332d4c Temporarily change to storing logs in a new LogEntry2 table
This will prevent us from running out of auto-incrementing ID values until such time as we can upgrade to peewee 3 and change the field type to a BigInt

Fixes https://jira.coreos.com/browse/QUAY-943
2018-05-18 20:15:16 -04:00
Joseph Schorr
e3248bde47 Small fixes to make loading of logs faster
Removes filtering of log types where not necessary, removes filtering based on namespace when filtering based on repository (superfluous check that was causing issues in MySQL preventing the use of the correct index) and fix some other small issues around the API

Fixes https://jira.coreos.com/browse/QUAY-931
2018-05-14 11:41:49 -04:00
Charlton Austin
131acde317 refactor(data+endpoints): code review changes
this puts the view logic on the object and adds a parameter for logging

[TESTING->locally with docker compose]

Issue: https://coreosdev.atlassian.net/browse/QUAY-632

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-07-12 14:10:21 -04:00
Charlton Austin
897a091692 style(data+endpoints): ran yapf
### Description of Changes

ran yapf for the branch

[TESTING->locally using docker compose]

Issue: https://coreosdev.atlassian.net/browse/QUAY-632

## Reviewer Checklist

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-07-10 09:46:02 -04:00
Charlton Austin
fc4b3642d3 refactor(endpoints/api/tag): refactor code for v22
this decouples the database models from the api

[TESTING->locally with docker compose]

Issue: https://coreosdev.atlassian.net/browse/QUAY-632

- [ ] It works!
- [ ] Comments provide sufficient explanations for the next contributor
- [ ] Tests cover changes and corner cases
- [ ] Follows Quay syntax patterns and format
2017-07-10 09:40:33 -04:00
EvB
20c4d971c4 refactor(model/log): pull allowed action types into constant 2017-04-07 11:39:54 -04:00
EvB
d0aaaaa1ef test(data/model/log): add more log_action tests
temp
2017-04-05 11:26:10 -04:00
EvB
ea740d15ba fix(model/log): log exception on swallowed db errors 2017-04-05 11:26:10 -04:00
EvB
625bd66b42 test(data/model): test action logging 2017-04-05 11:26:10 -04:00
EvB
6916d82e0d feat(endpoints/trackhelper): wrap log op for silent fails 2017-04-05 11:26:10 -04:00
Joseph Schorr
11c931f781 Log more information to the action logs and display the namespaces for superusers
This helps superusers understand better what, exactly, is going on in the registry
2017-02-14 14:55:24 -05:00
Joseph Schorr
0f46230493 Add an index for lookup by account to log entries
Also fixes the query to require one less join
2016-08-12 17:39:31 -04:00
Jimmy Zelinskie
ce14b9dddf modify log_action to internally resolve IDs 2016-08-08 12:38:15 -04:00
Joseph Schorr
853cca35f3 Change repo stats to use the RAC table and a nice UI 2016-06-22 15:06:53 -04:00
Joseph Schorr
043699cfb3 Always use log entry kind cache
Fixes #1445
2016-05-13 15:20:55 -04:00
Jimmy Zelinskie
972e4be811 log: cutoff at the max id past the cutoff_date
Previously we were using the min, which is always going to be equivalant
to the min id in the table.
2016-05-10 20:13:10 -07:00
Jimmy Zelinskie
29e2d7c9d4 data.model.log: remove unused method 2016-04-29 14:22:53 -04:00
Joseph Schorr
4f63a50a17 Change account-less logs to use a user and not null
This allows us to skip the migration
2016-04-29 14:09:37 -04:00
Joseph Schorr
11ff3e9b59 keys ui WIP 2016-04-29 14:05:16 -04:00
Jimmy Zelinskie
7239c465bf improve stale cutoff id perf (#1392) 2016-04-20 15:03:06 -04:00
Jimmy Zelinskie
ee705fe7a9 vastly simplify log rotation 2016-02-09 18:20:14 -05:00
Joseph Schorr
bd0a098282 Add ID-based pagination to logs using new decorators and an encrypted token
Fixes #599
2016-01-26 12:50:48 -05:00
Jimmy Zelinskie
e1f955a3f6 add a log rotation worker
Fixes #609.
2015-12-16 17:22:28 -05:00
Jimmy Zelinskie
9818481b08 limit logs to a maximum number of pages 2015-10-06 14:13:23 -04:00
Joseph Schorr
ea25538646 MySQL and Postgres complain about the group by, so calculate dates ourselves 2015-08-06 12:52:55 -04:00
Joseph Schorr
d34afde954 Fix logs view and API
- We needed to use an engine-agnostic way to extract the days
- Joining with the LogEntryKind table has *horrible* performance in MySQL, so do it ourselves
- Limit to 50 logs per page
2015-08-05 17:47:03 -04:00
Joseph Schorr
3d6c92901c Switch to using an aggregated logs query and infinite scrolling
This should allow users to work with large logs set.

Fixes #294
2015-07-31 16:38:02 -04:00
Jake Moshenko
3efaa255e8 Accidental refactor, split out legacy.py into separate sumodules and update all call sites. 2015-07-17 11:56:15 -04:00