File systems verification utility and library, in likeness of mtree(8)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
Vincent Batts 9e437eee80
go: update modules
5 months ago
.github/workflows github: fix the branch 8 months ago
.vscode vscode: adding IDE tasks 5 years ago
cmd/gomtree main: compare two files with two -f flags 8 months ago
pkg/govis govis: comment for lint 5 years ago
test walk: directory is expected to be walked. A file is not. 7 months ago
testdata casync: adding a casync style mtree 5 years ago
vendor go: update modules 5 months ago
xattr *.go: updated go:build added via gofmt 7 months ago
.drone.yml trying out drone 1 year ago
.gitignore Makefile: staticcheck validation 8 months ago
LICENSE LICENSE: adding a license 7 years ago
Makefile Makefile: only make a single SUMS file, including sha1 and sha512 7 months ago
README.md README: github workflow badge 8 months ago
check.go check: remove unused TarCheck(), for Compare() 4 years ago
check_test.go check_test: check the error 8 months ago
cksum.go cksum: comment 7 years ago
cksum_test.go cksum: test is fine. commenting. 7 years ago
compare.go compare: gofmt 8 months ago
compare_test.go *_test.go: ignore cyclomatic complexity for test files 8 months ago
creator.go walk: implement FsEval hooks 6 years ago
entry.go entry: rework e.Path() handling for casync-mtree 4 years ago
fseval.go walk: implement FsEval hooks 6 years ago
fseval_test.go *_test.go: ignore cyclomatic complexity for test files 8 months ago
go.mod go: update modules 5 months ago
go.sum go: update modules 5 months ago
hierarchy.go compare: correctly use .Prefix() during comparisons 5 years ago
hierarchy_test.go compare: correctly use .Prefix() during comparisons 5 years ago
keywordfunc.go keywordfunc: ignore ripemd160 for compat sake 8 months ago
keywordfuncs_bsd.go *.go: updated go:build added via gofmt 7 months ago
keywordfuncs_linux.go keywords: tar.Xattrs is deprecated by PAXRecords 8 months ago
keywordfuncs_unsupported.go *.go: updated go:build added via gofmt 7 months ago
keywords.go *.go: go-staticcheck linting 8 months ago
keywords_linux_test.go *.go: go-staticcheck linting 8 months ago
keywords_test.go keyword_test: remove unused line 8 months ago
lchtimes_unix.go *.go: updated go:build added via gofmt 7 months ago
lchtimes_unsupported.go *.go: updated go:build added via gofmt 7 months ago
lookup.go lookup: remove legacy lookup code 8 months ago
mtree_test.go *.go: clean up variable names 4 years ago
parse.go *: make Keyword and KeyVal pervasive 6 years ago
releases.md release: add some steps to remember 6 years ago
stat_unix.go *.go: updated go:build added via gofmt 7 months ago
stat_windows.go *.go: updated go:build added via gofmt 7 months ago
tar.go *.go: go-staticcheck linting 8 months ago
tar_test.go *.go: go-staticcheck linting 8 months ago
update.go vendor: Replace Sirupsen/logrus with sirupsen/logrus 5 years ago
update_linux_test.go comment: typo 7 months ago
update_test.go *_test.go: ignore cyclomatic complexity for test files 8 months ago
updatefuncs.go vendor: Replace Sirupsen/logrus with sirupsen/logrus 5 years ago
updatefuncs_linux.go *.go: updated go:build added via gofmt 7 months ago
updatefuncs_unsupported.go *.go: updated go:build added via gofmt 7 months ago
version.go version: bump master back to -dev 3 years ago
walk.go walk: directory is expected to be walked. A file is not. 7 months ago
walk_test.go casync: adding a casync style mtree 5 years ago

README.md

go-mtree

Go Go Report Card

mtree is a filesystem hierarchy validation tooling and format. This is a library and simple cli tool for mtree(8) support.

While the traditional mtree cli utility is primarily on BSDs (FreeBSD, openBSD, etc), even broader support for the mtree specification format is provided with libarchive (libarchive-formats(5)).

There is also an mtree port for Linux though it is not widely packaged for Linux distributions.

Format

The format of hierarchy specification is consistent with the # mtree v2.0 format. Both the BSD mtree and libarchive ought to be interoperable with it with only one definite caveat. On Linux, extended attributes (xattr) on files are often a critical aspect of the file, holding ACLs, capabilities, etc. While FreeBSD filesystem do support extattr, this feature has not made its way into their mtree.

This implementation of mtree supports a few non-upstream "keyword"s, such as: xattr and tar_time. If you include these keywords, the FreeBSD mtree will fail, as they are unknown keywords to that implementation.

To have go-mtree produce specifications that will be strictly compatible with the BSD mtree, use the -bsd-keywords flag when creating a manifest. This will make sure that only the keywords supported by BSD mtree are used in the program.

Typical form

With the standard keywords, plus say sha256digest, the hierarchy specification looks like:

# .
/set type=file nlink=1 mode=0664 uid=1000 gid=100
. size=4096 type=dir mode=0755 nlink=6 time=1459370393.273231538
    LICENSE size=1502 mode=0644 time=1458851690.0 sha256digest=ef4e53d83096be56dc38dbf9bc8ba9e3068bec1ec37c179033d1e8f99a1c2a95
    README.md size=2820 mode=0644 time=1459370256.316148361 sha256digest=d9b955134d99f84b17c0a711ce507515cc93cd7080a9dcd50400e3d993d876ac

[...]

See the directory presently in, and the files present. Along with each path, is provided the keywords and the unique values for each path. Any common keyword and values are established in the /set command.

Extended attributes form

# .
/set type=file nlink=1 mode=0664 uid=1000 gid=1000
. size=4096 type=dir mode=0775 nlink=6 time=1459370191.11179595 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==
    LICENSE size=1502 time=1458851690.583562292 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==
    README.md size=2366 mode=0644 time=1459369604.0 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==

[...]

See the keyword prefixed with xattr. followed by the extended attribute's namespace and keyword. This setup is consistent for use with Linux extended attributes as well as FreeBSD extended attributes.

Since extended attributes are an unordered hashmap, this approach allows for checking each <namespace>.<key> individually.

The value is the base64 encoded of the value of the particular extended attribute. Since the values themselves could be raw bytes, this approach avoids issues with encoding.

Tar form

# .
/set type=file mode=0664 uid=1000 gid=1000
. type=dir mode=0775 tar_time=1468430408.000000000

# samedir
samedir type=dir mode=0775 tar_time=1468000972.000000000
    file2 size=0 tar_time=1467999782.000000000
    file1 size=0 tar_time=1467999781.000000000
    
[...]

While go-mtree serves mainly as a library for upstream mtree support, go-mtree is also compatible with tar archives (which is not an upstream feature). This means that we can now create and validate a manifest by specifying a tar file. More interestingly, this also means that we can create a manifest from an archive, and then validate this manifest against a filesystem hierarchy that's on disk, and vice versa.

Notice that for the output of creating a validation manifest from a tar file, the default behavior for evaluating a notion of time is to use the tar_time keyword. In the "filesystem hierarchy" format of mtree, time is being evaluated with nanosecond precision. However, GNU tar truncates a file's modification time to 1-second precision. That is, if a file's full modification time is 123456789.123456789, the "tar time" equivalent would be 123456789.000000000. This way, if you validate a manifest created using a tar file against an actual root directory, there will be no complaints from go-mtree so long as the 1-second precision time of a file in the root directory is the same.

Usage

To use the Go programming language library, see the docs.

To use the command line tool, first build it, then the following.

Create a manifest

This will also include the sha512 digest of the files.

gomtree -c -K sha512digest -p . > /tmp/root.mtree

With a tar file:

gomtree -c -K sha512digest -T sometarfile.tar > /tmp/tar.mtree

Validate a manifest

gomtree -p . -f /tmp/root.mtree

With a tar file:

gomtree -T sometarfile.tar -f /tmp/root.mtree

See the supported keywords

gomtree -list-keywords
Available keywords:
 uname
 sha1
 sha1digest
 sha256digest
 xattrs (not upstream)
 link (default)
 nlink (default)
 md5digest
 rmd160digest
 mode (default)
 cksum
 md5
 rmd160
 type (default)
 time (default)
 uid (default)
 gid (default)
 sha256
 sha384
 sha512
 xattr (not upstream)
 tar_time (not upstream)
 size (default)
 ripemd160digest
 sha384digest
 sha512digest

Building

Either:

go get github.com/vbatts/go-mtree/cmd/gomtree

or

git clone git://github.com/vbatts/go-mtree.git $GOPATH/src/github.com/vbatts/go-mtree
cd $GOPATH/src/github.com/vbatts/go-mtree
go build ./cmd/gomtree

Testing

On Linux:

cd $GOPATH/src/github.com/vbatts/go-mtree
make

On FreeBSD:

cd $GOPATH/src/github.com/vbatts/go-mtree
gmake