tarsum: updates for jamtur01 comments
Signed-off-by: Vincent Batts <vbatts@redhat.com>
This commit is contained in:
parent
9a45c4235a
commit
bd9c676bb7
1 changed files with 36 additions and 46 deletions
|
|
@ -1,5 +1,5 @@
|
|||
page_title: TarSum checksum specification
|
||||
page_description: Documentation for algorithm used in the TarSum checksum calculation
|
||||
page_description: Documentation for algorithms used in the TarSum checksum calculation
|
||||
page_keywords: docker, checksum, validation, tarsum
|
||||
|
||||
# TarSum Checksum Specification
|
||||
|
|
@ -7,58 +7,54 @@ page_keywords: docker, checksum, validation, tarsum
|
|||
## Abstract
|
||||
|
||||
This document describes the algorithms used in performing the TarSum checksum
|
||||
calculation on file system layers, the need for this method over existing
|
||||
calculation on filesystem layers, the need for this method over existing
|
||||
methods, and the versioning of this calculation.
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
The transportation of file systems, regarding docker, is done with tar(1)
|
||||
The transportation of filesystems, regarding Docker, is done with tar(1)
|
||||
archives. There are a variety of tar serialization formats [2], and a key
|
||||
concern here is ensuring a repeatable checksum given a set of inputs from a
|
||||
generic tar archive. Types of transportation include distribution to and from a
|
||||
registry endpoint, saving and loading through commands or docker daemon APIs,
|
||||
transferring the build context from client to docker daemon, and committing the
|
||||
file system of a container to become an image.
|
||||
registry endpoint, saving and loading through commands or Docker daemon APIs,
|
||||
transferring the build context from client to Docker daemon, and committing the
|
||||
filesystem of a container to become an image.
|
||||
|
||||
As tar archives are used for transit, but not preserved in many situations, the
|
||||
focus of the algorithm is to ensure the integrity of the preserved file system,
|
||||
focus of the algorithm is to ensure the integrity of the preserved filesystem,
|
||||
while maintaining a deterministic accountability. This includes neither
|
||||
constrain the ordering or manipulation of the files during the creation or
|
||||
constraining the ordering or manipulation of the files during the creation or
|
||||
unpacking of the archive, nor include additional metadata state about the file
|
||||
system attributes.
|
||||
|
||||
|
||||
## Intended Audience
|
||||
|
||||
This document is outlining the methods used for consistent checksum calculation
|
||||
for file systems transported via tar archives.
|
||||
for filesystems transported via tar archives.
|
||||
|
||||
Auditing these methodologies is an open and iterative process. This document
|
||||
should accommodate the review of source code. Ultimately, this document should
|
||||
be the starting point of further refinements to the algorithm and its future
|
||||
versions.
|
||||
|
||||
|
||||
## Concept
|
||||
|
||||
The checksum mechanism must ensure the integrity and assurance of the
|
||||
file system payload.
|
||||
|
||||
filesystem payload.
|
||||
|
||||
## Checksum Algorithm Profile
|
||||
|
||||
A checksum mechanism must define the following operations and attributes:
|
||||
|
||||
* associated hashing cipher - used to checksum each file payload and attribute
|
||||
* Associated hashing cipher - used to checksum each file payload and attribute
|
||||
information.
|
||||
* checksum list - each file of the file system archive has its checksum
|
||||
* Checksum list - each file of the filesystem archive has its checksum
|
||||
calculated from the payload and attributes of the file. The final checksum is
|
||||
calculated from this list, with specific ordering.
|
||||
* version - as the algorithm adapts to requirements, there are behaviors of the
|
||||
* Version - as the algorithm adapts to requirements, there are behaviors of the
|
||||
algorithm to manage by versioning.
|
||||
* archive being calculated - the tar archive having its checksum calculated
|
||||
|
||||
* Archive being calculated - the tar archive having its checksum calculated
|
||||
|
||||
## Elements of TarSum checksum
|
||||
|
||||
|
|
@ -73,13 +69,14 @@ There are two delimiters used:
|
|||
|
||||
Example:
|
||||
|
||||
```
|
||||
"tarsum.v1+sha256:220a60ecd4a3c32c282622a625a54db9ba0ff55b5ba9c29c7064a2bc358b6a3e"
|
||||
| | \ |
|
||||
| | \ |
|
||||
|_version_|_cipher__|__ |
|
||||
| \ |
|
||||
|_calculation_mechanics_|______________________expected_sum_______________________|
|
||||
|
||||
```
|
||||
|
||||
## Versioning
|
||||
|
||||
|
|
@ -92,51 +89,50 @@ The general algorithm will be describe further in the 'Calculation'.
|
|||
|
||||
This is the initial version of TarSum.
|
||||
|
||||
Its element in the checksum "tarsum"
|
||||
|
||||
Its element in the TarSum checksum string is `tarsum`.
|
||||
|
||||
### Version1
|
||||
|
||||
Its element in the checksum "tarsum.v1"
|
||||
Its element in the TarSum checksum is `tarsum.v1`.
|
||||
|
||||
The notable changes in this version:
|
||||
* exclusion of file mtime from the file information headers, in each file
|
||||
* Exclusion of file `mtime` from the file information headers, in each file
|
||||
checksum calculation
|
||||
* inclusion of extended attributes (xattrs. Also seen as "SCHILY.xattr." prefixed Pax
|
||||
* Inclusion of extended attributes (`xattrs`. Also seen as `SCHILY.xattr.` prefixed Pax
|
||||
tar file info headers) keys and values in each file checksum calculation
|
||||
|
||||
### VersionDev
|
||||
|
||||
*Do not use unless validating refinements to the checksum algorithm*
|
||||
|
||||
Its element in the checksum "tarsum.dev"
|
||||
Its element in the TarSum checksum is `tarsum.dev`.
|
||||
|
||||
This is a floating place holder for a next version. The methods used for
|
||||
calculation are subject to change without notice.
|
||||
This is a floating place holder for a next version and grounds for testing
|
||||
changes. The methods used for calculation are subject to change without notice,
|
||||
and this version is for testing and not for production use.
|
||||
|
||||
## Ciphers
|
||||
|
||||
The official default and standard hashing cipher used in the calculation mechanic
|
||||
is "sha256". This refers to SHA256 hash algorithm as defined in FIPS 180-4.
|
||||
is `sha256`. This refers to SHA256 hash algorithm as defined in FIPS 180-4.
|
||||
|
||||
Though the algorithm itself is not exclusively bound to this single hashing
|
||||
cipher, and support for alternate hashing ciphers was later added [1]. Presently
|
||||
use of this is for isolated use-cases and future-proofing the TarSum checksum
|
||||
format.
|
||||
Though the TarSum algorithm itself is not exclusively bound to the single
|
||||
hashing cipher `sha256`, support for alternate hashing ciphers was later added
|
||||
[1]. Use cases for alternate cipher could include future-proofing TarSum
|
||||
checksum format and using faster cipher hashes for tar filesystem checksums.
|
||||
|
||||
## Calculation
|
||||
|
||||
### Requirement
|
||||
|
||||
As mentioned earlier, the calculation is such that it takes into consideration
|
||||
the life and cycle of the tar archive. In that the tar archive is not an
|
||||
immutable, permanent artifact. Otherwise options like relying on a known hashing
|
||||
cipher checksum of the archive itself would be reliable enough. Since the tar
|
||||
archive is used as a transportation medium, and is thrown away after its
|
||||
contents are extracted. Therefore, for consistent validation items such as
|
||||
order of files in the tar archive and time stamps are subject to change once an
|
||||
image is received.
|
||||
|
||||
the lifecycle of the tar archive. In that the tar archive is not an immutable,
|
||||
permanent artifact. Otherwise options like relying on a known hashing cipher
|
||||
checksum of the archive itself would be reliable enough. The tar archive of the
|
||||
filesystem is used as a transportation medium for Docker images, and the
|
||||
archive is discarded once its contents are extracted. Therefore, for consistent
|
||||
validation items such as order of files in the tar archive and time stamps are
|
||||
subject to change once an image is received.
|
||||
|
||||
### Process
|
||||
|
||||
|
|
@ -175,7 +171,6 @@ For >= Version1, the extented attribute headers ("SCHILY.xattr." prefixed pax
|
|||
headers) included after the above list. These xattrs key/values are first
|
||||
sorted by the keys.
|
||||
|
||||
|
||||
#### Header Format
|
||||
|
||||
The ordered headers are written to the hash in the format of
|
||||
|
|
@ -184,13 +179,11 @@ The ordered headers are written to the hash in the format of
|
|||
|
||||
with no newline.
|
||||
|
||||
|
||||
#### Body
|
||||
|
||||
After the order headers of the file have been added to the checksum for the
|
||||
file, the body of the file is written to the hash.
|
||||
|
||||
|
||||
#### List of file sums
|
||||
|
||||
The list of file sums is sorted by the string of the hexadecimal digest.
|
||||
|
|
@ -199,7 +192,6 @@ If there are two files in the tar with matching paths, the order of occurrence
|
|||
for that path is reflected for the sums of the corresponding file header and
|
||||
body.
|
||||
|
||||
|
||||
#### Final Checksum
|
||||
|
||||
Begin with a fresh or initial state of the associated hash cipher. If there is
|
||||
|
|
@ -211,7 +203,6 @@ The resulting digest is formatted per the Elements of TarSum checksum,
|
|||
including the TarSum version, the associated hash cipher and the hexadecimal
|
||||
encoded checksum digest.
|
||||
|
||||
|
||||
## Security Considerations
|
||||
|
||||
The initial version of TarSum has undergone one update that could invalidate
|
||||
|
|
@ -220,7 +211,6 @@ with same names as prior files in the archive. The latter file will clobber the
|
|||
prior file of the same path. Due to this the algorithm now accounts for files
|
||||
with matching paths, and orders the list of file sums accordingly [3].
|
||||
|
||||
|
||||
## Footnotes
|
||||
|
||||
* [0] Versioning https://github.com/docker/docker/commit/747f89cd327db9d50251b17797c4d825162226d0
|
||||
|
|
|
|||
Loading…
Reference in a new issue