Add a chunk cleanup queue for async GC of empty chunks
Instead of having the Swift storage engine try to delete the empty chunk(s) synchronously, we simply queue them and have a worker come along after 30s to delete the empty chunks. This has a few key benefits: it is async (doesn't slow down the push code), helps deal with Swift's eventual consistency (less retries necessary) and is generic for other storage engines if/when they need this as well
This commit is contained in:
parent
59cb6bd216
commit
5f99448adc
12 changed files with 191 additions and 59 deletions
7
conf/init/service/chunkcleanupworker/log/run
Executable file
7
conf/init/service/chunkcleanupworker/log/run
Executable file
|
@ -0,0 +1,7 @@
|
|||
#!/bin/sh
|
||||
|
||||
# Ensure dependencies start before the logger
|
||||
sv check syslog-ng > /dev/null || exit 1
|
||||
|
||||
# Start the logger
|
||||
exec logger -i -t chunkcleanupworker
|
8
conf/init/service/chunkcleanupworker/run
Executable file
8
conf/init/service/chunkcleanupworker/run
Executable file
|
@ -0,0 +1,8 @@
|
|||
#! /bin/bash
|
||||
|
||||
echo 'Starting chunk cleanup worker'
|
||||
|
||||
cd /
|
||||
venv/bin/python -m workers.chunkcleanupworker 2>&1
|
||||
|
||||
echo 'Chunk cleanup worker exited'
|
Reference in a new issue