Add a chunk cleanup queue for async GC of empty chunks
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
This commit is contained in:
parent
59cb6bd216
commit
5f99448adc
12 changed files with 191 additions and 59 deletions
|
@ -137,6 +137,7 @@ class DefaultConfig(object):
|
|||
DOCKERFILE_BUILD_QUEUE_NAME = 'dockerfilebuild'
|
||||
REPLICATION_QUEUE_NAME = 'imagestoragereplication'
|
||||
SECSCAN_NOTIFICATION_QUEUE_NAME = 'security_notification'
|
||||
CHUNK_CLEANUP_QUEUE_NAME = 'chunk_cleanup'
|
||||
|
||||
# Super user config. Note: This MUST BE an empty list for the default config.
|
||||
SUPER_USERS = []
|
||||
|
|
Reference in a new issue