To smooth initial implementation, uploads were spooled to local file storage,
validated, then pushed to remote storage. That approach was flawed in that it
present easy clustering of registry services that share a remote storage
backend. The original plan was to implement resumable hashes then implement
remote upload storage. After some thought, it was found to be better to get
remote spooling working, then optimize with resumable hashes.
Moving to this approach has tradeoffs: after storing the complete upload
remotely, the node must fetch the content and validate it before moving it to
the final location. This can double bandwidth usage to the remote backend.
Modifying the verification and upload code to store intermediate hashes should
be trivial once the layer digest format has settled.
The largest changes for users of the storage package (mostly the registry app)
are the LayerService interface and the LayerUpload interface. The LayerService
now takes qualified repository names to start and resume uploads. In corallry,
the concept of LayerUploadState has been complete removed, exposing all aspects
of that state as part of the LayerUpload object. The LayerUpload object has
been modified to work as an io.WriteSeeker and includes a StartedAt time, to
allow for upload timeout policies. Finish now only requires a digest, eliding
the requirement for a size parameter.
Resource cleanup has taken a turn for the better. Resources are cleaned up
after successful uploads and during a cancel call. Admittedly, this is probably
not completely where we want to be. It's recommend that we bolster this with a
periodic driver utility script that scans for partial uploads and deletes the
underlying data. As a small benefit, we can leave these around to better
understand how and why these uploads are failing, at the cost of some extra
disk space.
Many other changes follow from the changes above. The webapp needs to be
updated to meet the new interface requirements.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change updates the path mapper to be able to specify upload management
locations. This includes a startedat file, which contains the RFC3339 formatted
start time of the upload and the actual data file.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
While reading from the input in WriteStream, the inmemory driver can deadlock
if the reader is from the same instance. To fix this, the write lock is
released before reading into a local buffer. The lock is re-acquired to
finish the actual write.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset implements a fileWriter type that can be used to managed writes
to remote files in a StorageDriver. Basically, it manages a local seek position
for a remote path. An efficient use of this implementation will write data in
large blocks.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Passing RepositoryInfo to ResolveAuthConfig, pullRepository, and pushRepository
Moving --registry-mirror configuration to registry config
Created resolve_repository job
Repo names with 'index.docker.io' or 'docker.io' are now synonymous with omitting an index name.
Adding test for RepositoryInfo
Adding tests for opts.StringSetOpts and registry.ValidateMirror
Fixing search term use of repoInfo
Adding integration tests for registry mirror configuration
Normalizing LookupImage image name to match LocalName parsing rules
Normalizing repository LocalName to avoid multiple references to an official image
Removing errorOut use in tests
Removing TODO comment
gofmt changes
golint comments cleanup. renaming RegistryOptions => registry.Options, and RegistryServiceConfig => registry.ServiceConfig
Splitting out builtins.Registry and registry.NewService calls
Stray whitespace cleanup
Moving integration tests for Mirrors and InsecureRegistries into TestNewIndexInfo unit test
Factoring out ValidateRepositoryName from NewRepositoryInfo
Removing unused IndexServerURL
Allowing json marshaling of ServiceConfig. Exposing ServiceConfig in /info
Switching to CamelCase for json marshaling
PR cleanup; removing 'Is' prefix from boolean members. Removing unneeded json tags.
Removing non-cleanup related fix for 'localhost:[port]' in splitReposName
Merge fixes for gh9735
Fixing integration test
Reapplying #9754
Adding comment on config.IndexConfigs use from isSecureIndex
Remove unused error return value from isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Adding back comment in isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
We now also have a storagedriver error variable for identifying
api calls that are not implemented by drivers (the URLFor method
is not implemented by either the filesystem or inmemory drivers)
v4auth will default to true and if the frankfurt (eu-central-1) region
is selected with v4auth set to false explicitly, the driver will error
out upon initialization.
The secure flag will be true by default and will change the
s3 endpoint of the region to http instead of https when selected as false.
The main benefits of running with secure being false is that it apparently
has a roughly 33% performance boost (even on pure data transfer, not only
connection setup which is what I would have expected).
No longer requires that file paths match the repository naming scheme,
but instead allows path components as short as a single character, as to
accommodate for single-character tag names.
When adding parameters to a location header, the client must not destroy
parameters already present. This change ensures that parameters are added,
rather than replaced when assembling the url.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the repo is no longer just the registry, we are moving the registry web
application package out of the repo root into a sub-package. We may break down
the registry package further to separate webapp components and bring the client
package under it. This change accomplishes the task of freeing up the repo root
for a distribution-oriented package. A stub doc.go file is left in place to
declare intent.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the repo is no longer just the registry, we are moving the registry web
application package out of the repo root into a sub-package. We may break down
the registry package further to separate webapp components and bring the client
package under it. This change accomplishes the task of freeing up the repo root
for a distribution-oriented package. A stub doc.go file is left in place to
declare intent.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The exported StringSet type is not necessary for the current use case of
validating issues and audiences. The exported fields on VerifyOptions have been
changed to require string slices. The collections package has been removed and
the StringSet has been moved to the token package, where it is used.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the common package no longer exists, the testutil package is being moved
up to the root. Ideally, we don't have large omnibus packages, like testutil,
but we can fix that in another refactoring round.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Because the repository name definitions are part of the v2 specification, they
have been moved out of the common package. This is part of the effort to break
up the common package into more sensible components.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
In preparation for removing the common package, the tarsum utilities are being
moved to the more relevant digest package. This functionality will probably go
away in the future, but it's maintained here for the time being.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
As part of the efforts to break up the common package before disaster strikes,
a new collections package has been created. More may belong there but for now,
it only includes an implementation of StringSet.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Note that the README currently contains details about the secure parameter which is part of a
separate pull request. I feel confident adding it here since I am certain we will eventually add
the secure parameter. Also note that encrypt now defaults to true and rootdirectory defaults to
the empty string.
To support clustered registry, upload UUIDs must be recognizable by
registries that did not issue the UUID. By creating an HMAC verifiable
upload state token, registries can validate upload requests that other
instances authorized. The tokenProvider interface could also use a redis
store or other system for token handling in the future.
It was probably ill-advised to couple manifest signing and verification to
their respective types. This changeset simply changes them from methods to
functions. These might not even be in this package in the future.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
