Allows us to have a new config provider for each setup, with no overlap of the directories used, and automatic cleanup of those directories.