Commit graph

56 commits

Author SHA1 Message Date
Michael Crosby
71e8d765df set +x on overlay fs dirs
We need to set +x on the overlay dirs or after dropping from root to a
non-root user an eperm will happen on exec or other file access

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-03-24 12:55:35 -07:00
Derek McGowan
6997c5c32d Update btrfs undo logic
Ensure undo failures are logged in a consistent way.
Remove created subvolume on commit failure on create.
Ensure that removed value get logged on delete failure.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-22 10:58:45 -07:00
Derek McGowan
55ef482c5a Add storage testsuite
Added error variables and error check methods to snapshots

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:16:39 -07:00
Derek McGowan
8383519d96
Move parent bucket to same level as snapshot bucket
Adds a zero byte separator on composite key.
Stores child key in back reference to allow following.
Add parent bucket to get/create bucket functions.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:15:36 -07:00
Derek McGowan
397499c288
Add root flag to boltdb tests
Needed for CI to run root coverage

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:15:36 -07:00
Derek McGowan
73f467a32b
Use enumerated kind in boltdb proto message
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:15:36 -07:00
Derek McGowan
912746b016
Update btrfs driver to use snapshot storage
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:15:36 -07:00
Derek McGowan
61b524aff2
Update overlay snapshot driver to use metastore
Update tests to use boltdb.
Update test suite to pass context.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:15:35 -07:00
Derek McGowan
63ea9908c0
Add bolt db metastore implementation
Add metastore benchmark suite to test metastore performance.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:15:35 -07:00
Derek McGowan
17e7e70fe2
Add snapshot metadata store interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-21 23:15:35 -07:00
Kunal Kushwaha
8f13b160b1 Testcase for multiple Prepare/View on same key.
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2017-03-08 11:16:12 +09:00
Michael Crosby
27a99400e8 Merge pull request #610 from dmcgowan/snapshot-move-testsuite
snapshot: move testsuite to sub-package
2017-03-07 16:45:04 -08:00
Derek McGowan
a4247e2aa9 Add snapshot plugin type
Update existing snapshot drivers to register as plugins.
Load snapshot driver at containerd startup.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-07 14:55:36 -08:00
Derek McGowan
66bbf5c449 Move snapshot testsuite to sub package
Prevents imports of snapshot to require importing test packages.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-07 09:16:45 -08:00
Akihiro Suda
750cdc951f snapshotter: add more assertion
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-03-06 08:34:43 +00:00
Derek McGowan
2831e9b469 Merge pull request #581 from kunalkushwaha/testsuite-transitive
Testcase to test Transitivity of snapshot layers.
2017-03-02 11:38:49 -08:00
Kunal Kushwaha
8bbce4b80f Testcase for Transitivity of snapshot layers.
Given snapshots A <- B <- C,
B is the parent of C and A is a transitive parent of C

Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2017-02-28 18:35:03 +09:00
Stephen Day
9743a1531d Merge pull request #575 from dmcgowan/snapshot-test-checks
Update basic snapshotter test
2017-02-27 20:09:22 -08:00
Kunal Kushwaha
d9dd68ef5d Testcases added for to test Stat() Interface.
Two test cases added
- Prepare, Stat on new layer, should return Active layer.
- Prepare & Commit , Stat on new layer, should return Committed layer

Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2017-02-28 11:19:32 +09:00
Derek McGowan
b8855a8fa6 Update basic snapshotter test
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-27 16:49:02 -08:00
Stephen Day
be20bb1eb8 Merge pull request #566 from samuelkarp/snapshot-design
design: Update snapshots.md with current design
2017-02-27 14:55:28 -08:00
Derek McGowan
c48e9a763f Allow ro mounts without a parent
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-27 13:39:52 -08:00
Derek McGowan
eeb88554ac snapshot/overlay: use readonly bindmount for single parent view
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-27 13:26:03 -08:00
Derek McGowan
1723444ba2 snapshot/overlay: prevent committing view
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-27 13:26:03 -08:00
Derek McGowan
ca56448263 snapshot/overlay: implement view
Allow creating actives without an upper directory for
capturing changes. Actives without the upper directory
will always be mounted read only. Read only actives
must have a parent.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-27 13:26:03 -08:00
Samuel Karp
4382d553ea snapshot: Add context to Snapshotter interface
Signed-off-by: Samuel Karp <skarp@amazon.com>
2017-02-24 15:23:23 -08:00
Samuel Karp
92b9262f63 design: Update snapshots.md with current design
Signed-off-by: Samuel Karp <skarp@amazon.com>
2017-02-24 13:43:16 -08:00
Stephen J Day
5922756d8b
snapshot/btrfs: simplify test setup
After reviewing this closer, we removed a few buffers in favor
`(*exec.Command).Output` and simplified the loop file creation method to
just use `(*os.File).Truncate`.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-17 16:53:11 -08:00
Stephen J Day
9c4b235954
snapshot/btrfs: update btrfs to snappshotter
Updates the btrfs snapshotter to meet the interface and current tests.
Mostly, we merge the keyspace into a common index. Like with the overlay
driver, we will still need to do more verification work to ensure
idempotence of key collisions.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-07 17:21:19 -08:00
Stephen J Day
e6c1bb0ff2
snapshot/overlay: port overlay driver to Snapshotter
With the change to the snapshotter interface, we've now updated the
overlay driver to follow the conventions of the current test suite. To
support key unification, an hashed index was added to active and
committed directories. We still need to do some testing around
collisions, but we'll leave that for a future PR.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-07 17:20:21 -08:00
Stephen J Day
ab08944aa7
snapshot: clarify active and committed snapshots
After receiving feedback on the `snapshot.Driver` interface, it was
found that the behavior of active and committed snapshots was confusing.
We attempt to clean this up by doing the following:

1. Define the concept of "active" and "committed" snapshots and their
lifecycle relationship. Active snapshots can be created from a parent.
Committed snapshots can only be created from active snapshots.
2. Only committed snapshots can be a parent.
3. Unify the keyspace of snapshots. For common operations, such as
removal and stat, we only have a single method that works for both
active and committed snapshots. For methods that take one or the other,
the restriction is called out. `Remove` and `Delete` are consolidated
for this purpose.
4. Define the `Info` data type to include name, parent, kind and
readonly state. This allows us to collect `Exists` and `Parent` into a
single method `Stat` and simplifies the `Walk` method, eliding `Active`.
5. The `Driver` has been renamed to `Snapshotter` due to the overuse of
the term `Driver`.

Effectively, we now have snapshots that are either active or committed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-07 17:18:14 -08:00
Stephen J Day
74949be676
btrfs: calculate hash key only once
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-06 17:09:11 -08:00
Stephen J Day
aeffd4f92c
btrfs: test btrfs snapshots with driver suite
We now include btrfs in the snapshot driver test suite. This includes
the addition of parent links and name hashing into the btrfs driver.
We'll probably endup replacing this with a common metadata store, as
these relationships are generally identical between implementations.

A small bug was discovered in the delete implementation in the course
testing, so the btrfs package has been updated with a fix.

The overlay driver was modified accordingly with the btrfs driver to use
`Driver` as the exported type of each driver packge.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-03 15:54:11 -08:00
Derek McGowan
aaf18b5962 Rename CopyDirectory to CopyDir
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 12:08:53 -08:00
Derek McGowan
200cd6e877 Update snapshots to use fs package
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-02-03 11:28:06 -08:00
Stephen J Day
d0b4ce8d17
snapshot/overlay: add snapshot test suite to overlay
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-02 18:55:53 -08:00
Stephen J Day
68fd25221d
snapshot: define the snapshot driver interface
We now define the `snapshot.Driver` interface based on earlier work.
Many details of the model are worked out, such as snapshot lifecycle and
parentage of commits against "Active" snapshots.

The impetus of this change is to provide a snapshot POC that does a
complete push/pull workflow. The beginnings of a test suite for snapshot
drivers is included that we can use to verify the assumptions of
drivers. The intent is to port the existing tests over to this test
suite and start scaling contributions and test to the snapshot driver
subsystem.

There are still some details that need to be worked out, such as listing
and metadata access. We can do this activity as we further integrate
with tooling.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-02-02 18:06:02 -08:00
Samuel Karp
3c37c9e6b1 snapshot: Remove mistakenly-added binary file
Signed-off-by: Samuel Karp <skarp@amazon.com>
2017-01-30 13:57:58 -08:00
Samuel Karp
cacda40317 snapshot: Separate tests using root from non-root
Signed-off-by: Samuel Karp <skarp@amazon.com>
2017-01-25 17:13:29 -08:00
Akihiro Suda
1f763301a6 snapshot: fix terminology inconsistency
LayerManipulator, SnapshotManipulator -> SnapshotManager

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-01-20 02:02:29 +00:00
Aaron.L.Xu
08bcbddb32 fix typo I found in this repo
Signed-off-by: Aaron.L.Xu <likexu@harmonycloud.cn>
2017-01-20 01:18:26 +08:00
Michael Crosby
a7a6270f2a Remove shelling out to mount
Also remove the target from the Mount struct because it should not be
used at all.  The target can be variable and set by a caller, not by the
snapshot drivers.

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2017-01-18 11:10:31 -08:00
Stephen J Day
e3f83fd53d snapshots: separate implementations into packages
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-13 15:32:41 -08:00
Samuel Karp
2c36c387de snapshot: Fix leaked mounts and temp directories
Signed-off-by: Samuel Karp <skarp@amazon.com>
2017-01-12 01:12:37 +00:00
Samuel Karp
dbe117cb7e snapshot: automate btrfs test setup
Signed-off-by: Samuel Karp <skarp@amazon.com>
2017-01-10 00:50:33 +00:00
Gábor Lipták
102fb10fde Correct vet warnings
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2016-12-20 17:10:01 -05:00
Qiang Huang
3b8eee7522 Fix some typos
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2016-12-16 09:31:19 +08:00
Stephen J Day
030d5dde1e btrfs: add initial implementation of btrfs snapshot driver
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-12 21:53:48 -08:00
Stephen J Day
bdc065a1df snapshot: follow dst-first order convention in commit
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-12 21:10:09 -08:00
Michael Crosby
934940a96c Add parent cache to overlayfs storage driver
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
2016-12-12 13:35:02 -08:00