Commit graph

34 commits

Author SHA1 Message Date
Stephen J Day
2ebc373d91 Refactor inmemory driver for Stat and WriteStream methods
This change started out as simply updating the existing inmemory driver to
implement the new Stat call. After struggling with the map based
implementation, it has been refactored to be a tree-based implementation.

This process has exposed a few missing error cases in the StorageDriver API
that should be addressed in the coming weeks.
2014-12-04 20:25:14 -08:00
Stephen J Day
ab9570f872 Migrate filesystem driver to new storagedriver calls
The filesystem driver has been migrated to impleemnt the storagedriver
interface changes. Most interetingly, this provides a filesystem-based
implementation of the Stat driver call. With this comes some refactoring of
Reads and Write to be much simpler and more robust.

The IPC tests have been disabled to stability problems that we'll have to
troubleshoot at a later date.
2014-12-03 16:47:31 -08:00
Stephen J Day
2037b1d6bf Update testsuite with storagedriver interface changes
This change updates the testsuite to migrate to the new driver interface. This
includes the new Stat call, changes to int64 over uint64 and the changes to the
WriteStream signature. Several test cases have been added to vet
implementations against various assumptions.
2014-12-03 16:37:46 -08:00
Stephen J Day
2e3ecdca37 Remove size argument and using io.Reader for StorageDriver.WriteStream
We are change the the rpc call for WriteStream to not require the size
argument, opting to drive the process with io.Reader. The main issue was that
io.Reader may return io.EOF before reaching size, making the error handling
around this condition for callers more complex. To complement this, WriteStream
now returns the number of successfully written bytes.

The method no longer requires an io.ReadCloser, opting to require just an
io.Reader. This keeps the reader under the control of the caller, which
provides more flexibility.

This also begins to address some of the problems described in #791.
2014-12-02 21:47:28 -08:00
Stephen J Day
ac660e72bf Replace StorageLayer.CurrentSize interface call with Stat
To support single-flight Size and ModTime queries against backend storage file,
we are replacing the CurrentSize call with a Stat call. A FileInfo interface is
provided for backends to provide a type, with a default implementation called
FileInfoInternal, for use by driver implementations.

More work needs to follow this change to update all the driver implementations.
2014-12-02 21:00:42 -08:00
Stephen J Day
b047c92e1c Use sync.WaitGroup to control concurrent tests 2014-12-02 20:43:31 -08:00
Stephen J Day
66107df1af Use int64 for ReadStream and WriteStream offsets
This change brings the storagedriver API in line with the Go standard library's
use of int64 for offsets. The main benefit is simplicity in interfacing with
the io library reducing the number of type conversions in simple code.
2014-12-02 19:01:00 -08:00
Ahmet Alp Balkan
4054cd3e73 Azure storage driver implementation
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2014-12-01 18:38:23 -08:00
Ahmet Alp Balkan
c5bb224bf9 Fix read offset check for inmemory driver
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2014-11-25 21:43:12 -08:00
Anton Tiurin
4635a1e903 [IPC] Tiny cleaning 2014-11-22 23:23:03 +03:00
Brian Bland
18eac89506 Adds a test for concurrent storagedriver Write/Read Stream operations
This test is currently failing and Skipped for IPC drivers
2014-11-20 16:15:55 -08:00
Brian Bland
68fd15b688 Miscellaneous storagedriver+ipc fixes
Fixes/tests listing for keys beginning with "/"
No longer extraneously wraps Closers in ioutil.NopClosers
Uses omitempty for all ipc struct type fields
2014-11-20 14:14:55 -08:00
Brian Bland
a3481c5f1c Adds ability to unwrap ipc errors into their original type
This only works for a specific whitelist of error types, which is
currently all errors in the storagedriver package.

Also improves storagedriver tests to enforce proper error types are
returned
2014-11-18 17:41:48 -08:00
Brian Bland
88795e0a14 Lots of various golint fixes
Changes some names to match go conventions
Comments all exported methods
Removes dot imports
2014-11-17 15:46:06 -08:00
Olivier Gambier
378256de47 Merge pull request #690 from BrianBland/storagedriver-process-management
Adds logic for tracking ipc storage driver process status
2014-11-13 11:24:46 -08:00
Brian Bland
c8ea224f9c Miscellaneous go vet fixes
Fixes some format strings and uses keyed fields for struct construction
2014-11-12 17:19:19 -08:00
Brian Bland
31df62064d Adds logic for tracking ipc storage driver process status
This allows requests to not hang if the child process exits
2014-11-11 13:54:12 -08:00
Olivier Gambier
da205085f3 Merge pull request #686 from BrianBland/storagedriver-versioning
Adds versioning for out-of-process storage driver
2014-11-07 14:55:13 -08:00
Brian Bland
cb1bdacbe3 Renames ResumeWritePosition to CurrentSize in storage driver api 2014-11-07 12:58:48 -08:00
Brian Bland
150677f1f5 Expects storage driver executables to be of the form registry-storage-name
Moves main packages to sub-packages of the individual storage drivers
2014-11-06 18:32:06 -08:00
Brian Bland
1ae5485998 Adds versioning for out-of-process storage driver
The registry currently only accepts storage driver versions with the
same major version and an equal or lower minor version as its own
current storage driver api version, but this may be changed in the
future if we decide to implement specific version cross-compatibility.
2014-11-06 12:16:14 -08:00
Brian Bland
7daa850d44 Fixes documentation to show that StorageDriver.List is non-recursive 2014-11-04 09:52:24 -08:00
Brian Bland
43716a2850 Uses IsTruncated and NextMarker for S3 list internal pagination 2014-11-03 16:27:19 -08:00
Brian Bland
b522fbd675 Removes WrapReader boilerplate for updates to libchan
libchan now supports io.ReadCloser and io.WriteCloser, so we don't need
io.ReadWriteCloser wrapping
2014-10-31 11:50:02 -07:00
Brian Bland
3e4738587f Adds README for the storagedriver package 2014-10-30 16:31:43 -07:00
Brian Bland
0e5d41ff9b Updates documentation to follow godoc conventions 2014-10-29 16:44:26 -07:00
Brian Bland
ca0084fad1 Adds StorageDriverFactory, unifying creation of StorageDrivers
Custom storage drivers can register a factory to create the driver by
name, similar to the database/sql package's Register and Open
factory.Create returns an in-process driver if registered or an IPC
driver if one can be found, erroring otherwise
This standardizes parameter passing for creation of storage drivers

Also adds documentation for storagedriver package and children
2014-10-28 18:21:06 -07:00
Brian Bland
ff81f3a719 Adds conditional SkipCheck for storage driver tests 2014-10-27 13:24:07 -07:00
Brian Bland
bac40b2b35 Merge pull request #3 from AndreyKostov/next-generation
Preliminary s3 driver implementation
2014-10-27 12:08:43 -07:00
Andrey Kostov
e3a5955cd2 Unify permissions settings 2014-10-26 10:00:53 -07:00
Brian Bland
7c892deb1c Uses streams internally for ipc Get/Put Content
This is done because libchan/spdystream does not currently support sending
serialzied objects of size larger than 16MB
See https://github.com/docker/libchan/issues/65
2014-10-24 18:44:36 -07:00
Andrey Kostov
1342873367 Add s3 driver for the new Storage Layer API 2014-10-24 16:37:25 -07:00
Andrey Kostov
47ca8be42f Slight additions/modifications to the test suite 2014-10-24 16:36:17 -07:00
Brian Bland
3f95694180 Adds storage driver interface, tests, and two basic implementations 2014-10-22 10:50:43 -07:00