This commit adds a transfer manager which deduplicates and schedules
transfers, and also an upload manager and download manager that build on
top of the transfer manager to provide high-level interfaces for uploads
and downloads. The push and pull code is modified to use these building
blocks.
Some benefits of the changes:
- Simplification of push/pull code
- Pushes can upload layers concurrently
- Failed downloads and uploads are retried after backoff delays
- Cancellation is supported, but individual transfers will only be
cancelled if all pushes or pulls using them are cancelled.
- The distribution code is decoupled from Docker Engine packages and API
conventions (i.e. streamformatter), which will make it easier to split
out.
This commit also includes unit tests for the new distribution/xfer
package. The tests cover 87.8% of the statements in the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
aufs kernel module creates whiteout files on upper layer delete (and
other situations) and those files already are 'translated' regarding
ownership in host terms (e.g. they are already "0:0" owned), so when
these layers are copied around with pkg/archive we don't want to try and
translate these files regarding ownership.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
In the existing code, "diff" has function scope and the value from the
previous iteration may be used if it is not reset. This appears to be an
oversight. This commit changes its scope to the for loop body.
One confusing point is that the cursor movement escape sequences appear
to be necessary even if the requested movement is 0. I haven't been able
to figure out why this makes a difference.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
When we handle a message that isn't tracked in the "line" map (for
example, one with no ID), clear the line map. This means we won't update
lines that were part of a previous, completed set of operations when
doing something like pull -a. It also has the beneficial side effect
of avoiding terminal glitching in these types of situations, since
messages that don't get tracked in the "line" map cause the count of the
number of lines to get out of sync.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
When user call the `Call()` method, they don't always want to sent
some args or get the return value, so they use `nil` when call `Call()`
method and this will casue an error. It's better to not trying to
encode or decode if it's nil.
Signed-off-by: Lei Jitang <leijitang@huawei.com>
This function was starting a goroutine that modifies one of its return
values. The intent is for the goroutine to only influence the return
value when it's causing the function to return, but it's racy and can
also modify the return value when the function is returning due to the
timeout. Fix the goroutine to not modify return values directly.
Also, give the channel a buffer so that the goroutine doesn't block
forever after a timeout.
Fixes#18305
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
I saw a failure of TestDockerCmdWithTimeout. This test starts a command
that produces output after 10 ms, but uses a 5 ms timeout, so normally
the command will be killed before the output. The time intervals are so
small that the timeout may not reliably trigger before the output, which
can cause the test to fail.
This commit changes the test to only fail if the process is still alive
after 10 seconds. This means the test will confirm that the timeouts are
happening, but not attempt to gauge that the timeouts are happening
within milliseconds of when they are expected (which can't be done
reliably).
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
A TopicFunc is an interface to let the pubisher decide whether it needs
to send a message to a subscriber or not. It returns true if the
publisher must send the message and false otherwise.
Users of the pubsub package can create a subscriber with a topic
function by calling `pubsub.SubscribeTopic`.
Message delivery has also been modified to use concurrent channels per
subscriber. That way, topic verification and message delivery is not
o(N+M) anymore, based on the number of subscribers and topic verification
complexity.
Using pubsub topics, the API stops controlling the message delivery,
delegating that function to a topic generated with the filtering
provided by the user. The publisher sends every message to the
subscriber if there is no filter, but the api doesn't have to select
messages to return anymore.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Improves the current filtering implementation complixity.
Currently, the best case is O(N) and worst case O(N^2) for key-value filtering.
In the new implementation, the best case is O(1) and worst case O(N), again for key-value filtering.
Signed-off-by: David Calavera <david.calavera@gmail.com>
It will Tar up contents of child directory onto tmpfs if mounted over
This patch will use the new PreMount and PostMount hooks to "tar"
up the contents of the base image on top of tmpfs mount points.
Signed-off-by: Dan Walsh <dwalsh@redhat.com>
Moved a defer up to a better spot.
Fixed TestUntarPathWithInvalidDest to actually fail for the right reason
Closes#18170
Signed-off-by: Doug Davis <dug@us.ibm.com>
Add distribution package for managing pulls and pushes. This is based on
the old code in the graph package, with major changes to work with the
new image/layer model.
Add v1 migration code.
Update registry, api/*, and daemon packages to use the reference
package's types where applicable.
Update daemon package to use image/layer/tag stores instead of the graph
package
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Can't safely use uint32 for locker since we need to decrement the count,
which requires loading the unit and doing some math, which is inherintly
racey.
Instead use Int32 which we can safely use with atomic and AddInt32 with
`-1`
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Fixes#16555
Original docker `cp` always copy symbol link itself instead of target,
now we provide '-L' option to allow docker to follow symbol link to real
target.
Signed-off-by: Zhang Wei <zhangwei555@huawei.com>
- isZeroValue function from upstream go
- covers booleans, strings and numbers
- change integration to reflect new behavior
- resolves#9406
Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>
