Compare commits

..

1645 commits

Author SHA1 Message Date
Arko Dasgupta
f5cdc24dd3
Merge pull request #3245 from olegburov/reopen-pr-2973
Reopen PR #2973 (Support ECS TaskRole in S3 storage driver).
2020-10-28 17:30:56 -07:00
olegburov
34f1322664 Fix hardcoded credential provides.
Signed-off-by: olegburov <oleg.burov@outlook.com>
2020-09-02 17:32:34 -07:00
Manish Tomar
2800ab0224
Merge pull request #2973 from redmatter/support-ecs-instance-profile-in-s3-driver
Support ECS TaskRole in S3 storage driver
2020-08-26 15:24:20 -07:00
Manish Tomar
5dc1f65acc
Merge pull request #3224 from AndreasHassing/fix/bad-pointer-windows-EnableVirtualTerminalProcessing
Update logrus to v1.6.0, fixes #3223
2020-08-24 15:04:52 -07:00
Manish Tomar
fe1ffa86bc
Merge pull request #3226 from thaJeztah/bump_go_digest
vendor: opencontainers/go-digest v1.0.0
2020-08-24 10:50:21 -07:00
Sebastiaan van Stijn
8a8d91529d
vendor: opencontainers/go-digest v1.0.0
full diff: https://github.com/opencontainers/go-digest/compare/v1.0.0-rc1...v1.0.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-08-24 18:46:53 +02:00
Andreas Hassing
9466dd4e5a Update logrus to v1.6.0
Fixes #3223 by bumping logrus to v1.6.0, which in turn bumps
github.com/konsorten/go-windows-terminal-sequences to v1.0.3
wherein the fix to bad pointer is found.

Signed-off-by: Andreas Hassing <andreas@famhassing.dk>
2020-08-24 13:14:04 +02:00
Manish Tomar
0316f34bf2
Merge pull request #3118 from lawliet89/bump-aws-sdk
Bump AWS SDK to v1.34.9
2020-08-21 15:57:08 -07:00
Yong Wen Chua
e1464fd317
Bump AWS SDK
Fixes https://github.com/docker/distribution/issues/3097

Signed-off-by: Yong Wen Chua <lawliet89@users.noreply.github.com>
2020-08-21 17:35:24 +08:00
Derek McGowan
35b26def43
Merge pull request #3173 from adamwg/catalog-no-layers-upstream
catalog: List repositories with no unique layers
2020-08-07 23:12:18 -07:00
Adam Wolfe Gordon
a784441b62 catalog: List repositories with no unique layers
A repository need not contain any unique layers, if its images use only layers
mounted from other repositories. But, the catalog endpoint was looking for the
_layers directory to indicate that a directory was a repository.

Use the _manifests directory as the marker instead, since any repository with
revisions will contain a _manifests directory.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
2020-08-04 11:38:49 -06:00
Manish Tomar
1d0ea8ed7b
Merge pull request #3140 from linux-on-ibm-z/travis-s390x
Add s390x support for travis
2020-07-30 16:15:55 -07:00
Derek McGowan
208d68bd5c
Merge pull request #3209 from zhipengzuo/master
clean up code because err is always nil
2020-07-22 09:18:06 -07:00
zhipengzuo
f361d443b7 clean up code because err is always nil
Signed-off-by: zhipengzuo <zuozhipeng@baidu.com>
2020-07-22 10:54:46 +08:00
Derek McGowan
53e18a9d9b
Merge pull request #3196 from thaJeztah/add_redirect
docs: add redirect for old URL
2020-07-08 16:08:24 -07:00
Sebastiaan van Stijn
7728c5e445
docs: add redirect for old URL
Looks like there's some projects refering to this old URL:
https://grep.app/search?q=https%3A//docs.docker.com/reference/api/registry_api/

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-07-08 12:18:31 +02:00
Andrew Bulford
9690d843fa Support ECS TaskRole in S3 storage driver
Instead of constructing the list of credential providers manually, if we
use the default list we can take advantage of the AWS SDK checking the
environment and returning either the EC2RoleProvider or the generic HTTP
credentials provider, configured to use the ECS credentials endpoint.

Also, use the `defaults.Config()` function instead of `aws.NewConfig()`,
as this results in an initialised HTTP client which prevents a fatal
error when retrieving credentials from the ECS credentials endpoint.

Fixes #2960

Signed-off-by: Andrew Bulford <andrew.bulford@redmatter.com>
2020-07-01 08:42:56 +01:00
Sebastiaan van Stijn
62d0fd45e7
Merge pull request #3187 from bloodorangeio/fix-ci
Fix CI failures, upgrade to Go 1.14+
2020-06-29 19:14:34 +02:00
jdolitsky
81ba770eff Fix CI failures, upgrade to Go 1.14+
- Modify Travis manifest to use Go 1.14.x
- Hardcode version of golangci-lint in setup script
- Hardcode version of go-md2man in setup script

Signed-off-by: jdolitsky <393494+jdolitsky@users.noreply.github.com>
2020-06-26 10:59:50 -05:00
srajmane
a2ed1b5e80 Added dist: bionic, updated go version to 1.14.x and set GO111MODULE=on
Signed-off-by: Snehal Rajmane <srajmane@us.ibm.com>
2020-04-29 03:25:24 -07:00
srajmane
55f88c35b9 Adding s390x support.
Signed-off-by: Snehal Rajmane <srajmane@us.ibm.com>
2020-04-07 07:59:19 -07:00
Derek McGowan
742aab907b
Merge pull request #3127 from dmage/err-shadow
Fix err shadowing in gcs driver
2020-03-19 10:36:57 -07:00
Derek McGowan
17a394f9af
Merge pull request #3128 from dmcgowan/gosimple-fix
Fix gosimple checks
2020-03-18 16:06:17 -07:00
Derek McGowan
78c2ab6646
Fix gosimple checks
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-18 15:52:34 -07:00
Oleg Bulatov
cdb4ba947a
Fix err shadowing in gcs driver
Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2020-03-18 11:41:34 +01:00
Derek McGowan
303f1899bb
Merge pull request #3121 from wy65701436/upgrade-oci-library
Update oci library
2020-03-10 14:58:43 -07:00
wang yan
bf56f348be Update oci library
1, update go-digest to v1.0.0-rc1
2, update image-spec to v1.0.1

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-10 23:19:29 +08:00
Derek McGowan
581be91482
Merge pull request #3113 from dmcgowan/upstream-redis-fixes
Redis cache fixes and metrics
2020-03-09 13:36:53 -07:00
Derek McGowan
be29c05a1e
Remove deprecated cache metrics code
The metrics tracker in cached blob statter was replaced with prometheus
metrics and no longer needed.
Remove unused log wrapping.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-09 13:11:54 -07:00
Derek McGowan
495a4af7cf
Fix goimports
Separate fix for cherry-picked code

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Grant Watters
74d442a058
Consider redis.ErrNil as distribution.ErrBlobUnknown for Stat HGET
* Update redis.go

If the dgst key does not exist in the cache when calling HGET, `redis.String` will return an `ErrNil` which we need to translate into `distribution.ErrBlobUnknown` so that the error being returned can be properly handled. This will ensure that `SetDescriptor` is properly called from `cachedBlobStatter::Stat` for `repositoryScopedRedisBlobDescriptorService` which will update the redis cache and be considered as a Miss rather than an Error.

cc @manishtomar

* Update suite.go

Add unit test to ensure missing blobs for scoped repo properly return ErrBlobUnknown when HGET returns redis.ErrNil.

(cherry picked from commit dca6b9526a1d30dd218a9f321c4f84ecc4b5e62e)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Manish Tomar
795892662b
redis metrics
* redis metrics

it is working but metrics are not very useful since default buckets
start from 5ms and almost all of them are in that range.

* remove extra comment

(cherry picked from commit ba1a1d74e7eb047dd1056548ccf0695e8846782c)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Manish Tomar
ce101280fe
fix redis caching issue
* fix redis caching issue

earlier redis cache was updated when there was any error including any
temporary connectivity issue. This would trigger set calls which would
further increase load and possibly connectivity errors from redis
leaving the system with continuous errors and high latency. Now the
cache is updated only when it is genuine cache miss. Other errors do not
trigger a cache update.

* add back tracker Hit() and Miss() calls

*squashed commits*
(cherry picked from commit 6f3e1c10260ef59ba4e9c42e939329fad9fdd8c3)
(cherry picked from commit 6738ff3320cf82cc2df919a95a1bde2f7789a501)

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Derek McGowan
4c7c63b557
Add cache unit test
Test base functionality of the cache statter

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Derek McGowan
492a10376c
Merge pull request #3111 from dmcgowan/update-report-issues
Update reporting issues guidelines
2020-03-04 14:39:19 -08:00
Derek McGowan
c79aa81772
Merge pull request #2550 from halcyonCorsair/halcyonCorsair/use-go-events-1
Use go-events package
2020-03-04 14:38:15 -08:00
Elliot Pahl
800cb95821
Use go-events package
TBD: Queue not converted yet

Signed-off-by: Elliot Pahl <elliot.pahl@gmail.com>
2020-03-04 12:49:32 -08:00
Derek McGowan
2d1126ecc1
Merge pull request #3080 from thaJeztah/bump_go_metrics
vendor: update docker/go-metrics v0.0.1
2020-03-04 10:21:48 -08:00
Derek McGowan
3b4d1e3ddf
Merge pull request #3110 from dmcgowan/update-maintainers
Update governance and maintainers
2020-03-04 10:19:46 -08:00
Derek McGowan
1e25ecefe4
Update governance and maintainers
Update format of governance and bring in language from containerd.
Update maintainers to reflect active maintainers.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 10:04:38 -08:00
Derek McGowan
db3c418ada
Update reporting issues guidelines
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-03 16:56:46 -08:00
Sebastiaan van Stijn
98dcc5195e
vendor: update docker/go-metrics v0.0.1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-03-02 20:14:15 +01:00
Derek McGowan
6b972e50fe
Merge pull request #2272 from naveedjamil/fips
Increase Unit Test Code Coverage
2020-02-22 17:40:41 -08:00
Derek McGowan
e65b3f1316
Fix CI for test updates
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-02-22 17:27:55 -08:00
Naveed Jamil
efdba4f210
Increase Unit Test Code Coverage
Unit test coverge was increased to cover the usages of crypto. This helps to ensure that everything is working fine with fips mode enabled.
Also updated sha1 to sha256 in registry/storage/driver/testsuites/testsuites.go because sha1 is not supported in fips mode.

Signed-off-by: Naveed Jamil <naveed.jamil@tenpearl.com>
2020-02-22 17:09:00 -08:00
Derek McGowan
e9a023180f
Merge pull request #3071 from dmcgowan/update-readme-docs
Update readme and contributing docs
2020-02-22 16:58:53 -08:00
Derek McGowan
f1cb8a56c8
Merge pull request #3105 from thaJeztah/bump_golang_1.13.8
Update Golang 1.13.8
2020-02-22 16:54:49 -08:00
Derek McGowan
bfa6b923e7
Merge pull request #3020 from kd7lxl/updatefrequency
Fix typo cloudfront updatefrenquency
2020-02-22 16:54:03 -08:00
Derek McGowan
f593b8d975
Merge pull request #2446 from legionus/docker-configuration-ptr
Fix the pointer initialization
2020-02-22 16:42:17 -08:00
Sebastiaan van Stijn
1f77c9a57f
Update Golang 1.13.8
full diff: https://github.com/golang/go/compare/go1.13.7...go1.13.8

go1.13.8 (released 2020/02/12) includes fixes to the runtime, the crypto/x509,
and net/http packages. See the Go 1.13.8 milestone on the issue tracker for details.

https://github.com/golang/go/issues?q=milestone%3AGo1.13.8+label%3ACherryPickApproved

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-22 17:59:03 +01:00
Derek McGowan
861aa2a621
Merge pull request #3089 from thaJeztah/bump_golang_1.13.7
Update Golang 1.13.7, golang.org/x/crypto (CVE-2020-0601, CVE-2020-7919)
2020-02-21 18:08:27 -08:00
Sebastiaan van Stijn
9b6a019081
prevent dev-tools from updating go.mod
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-21 23:47:14 +01:00
Sebastiaan van Stijn
7e290869e7
vendor: update golang.org/x/crypto v0.0.0-20200128174031-69ecbb4d6d5d (CVE-2020-7919)
Includes 69ecbb4d6d
(forward-port of 8b5121be2f),
which fixes CVE-2020-7919:

- Panic in crypto/x509 certificate parsing and golang.org/x/crypto/cryptobyte
  On 32-bit architectures, a malformed input to crypto/x509 or the ASN.1 parsing
  functions of golang.org/x/crypto/cryptobyte can lead to a panic.
  The malformed certificate can be delivered via a crypto/tls connection to a
  client, or to a server that accepts client certificates. net/http clients can
  be made to crash by an HTTPS server, while net/http servers that accept client
  certificates will recover the panic and are unaffected.
  Thanks to Project Wycheproof for providing the test cases that led to the
  discovery of this issue. The issue is CVE-2020-7919 and Go issue golang.org/issue/36837.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-21 23:45:56 +01:00
Sebastiaan van Stijn
016549532f
Dockerfile: use alpine 3.11
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-21 23:45:54 +01:00
Sebastiaan van Stijn
974375f66c
re-vendor with go 1.13
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-21 23:45:52 +01:00
Sebastiaan van Stijn
4ae059c714
Update Golang 1.13.7 (CVE-2020-0601, CVE-2020-7919)
full diff: https://github.com/golang/go/compare/go1.13.4...go1.13.7

go1.13.7 (released 2020/01/28) includes two security fixes. One mitigates
the CVE-2020-0601 certificate verification bypass on Windows. The other affects
only 32-bit architectures.

https://github.com/golang/go/issues?q=milestone%3AGo1.13.7+label%3ACherryPickApproved

- X.509 certificate validation bypass on Windows 10
  A Windows vulnerability allows attackers to spoof valid certificate chains when
  the system root store is in use. These releases include a mitigation for Go
  applications, but it’s strongly recommended that affected users install the
  Windows security update to protect their system.
  This issue is CVE-2020-0601 and Go issue golang.org/issue/36834.
- Panic in crypto/x509 certificate parsing and golang.org/x/crypto/cryptobyte
  On 32-bit architectures, a malformed input to crypto/x509 or the ASN.1 parsing
  functions of golang.org/x/crypto/cryptobyte can lead to a panic.
  The malformed certificate can be delivered via a crypto/tls connection to a
  client, or to a server that accepts client certificates. net/http clients can
  be made to crash by an HTTPS server, while net/http servers that accept client
  certificates will recover the panic and are unaffected.
  Thanks to Project Wycheproof for providing the test cases that led to the
  discovery of this issue. The issue is CVE-2020-7919 and Go issue golang.org/issue/36837.
  This is also fixed in version v0.0.0-20200124225646-8b5121be2f68 of golang.org/x/crypto/cryptobyte.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-02-21 23:45:49 +01:00
Derek McGowan
d054b13dc3
Merge pull request #3086 from thaJeztah/forward_port_ISS-2819
[master] Use same env var in Dockerfile and Makefile
2020-02-21 10:24:25 -08:00
Derek McGowan
244d5246c2
Merge pull request #3023 from sayboras/feature/golangci-lint
Migrate to golangci-lint
2020-02-21 10:21:15 -08:00
sayboras
66809646d9 Migrate to golangci-lint
Signed-off-by: Tam Mach <sayboras@yahoo.com>
2020-02-14 08:11:16 +11:00
Ryan Abrams
23f6bdd743
Use same env var in Dockerfile and Makefile
Ensures that build tags get set in the Dockerfile so that OSS and GCS drivers
are built into the official registry binary.

Closes #2819

Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
(cherry picked from commit bf74e4f91d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-01-22 10:54:02 +01:00
Derek McGowan
a837179414
Merge pull request #3072 from fermayo/fix-TestRegistryAsCacheMutationAPIs
Fix TestRegistryAsCacheMutationAPIs
2019-12-15 20:48:56 -08:00
Fernando Mayo Fernandez
6ca7b9e9fa
Fix TestRegistryAsCacheMutationAPIs
Use a synthetic upstream registry when creating the testing mirror configuration
to avoid the test fail when trying to reach http://example.com

Signed-off-by: Fernando Mayo Fernandez <fernando@undefinedlabs.com>
2019-12-15 13:51:25 +01:00
Derek McGowan
f5e84a4939
Update readme and contributing docs
Update project details to align more closely with community
updates in OCI and containerd.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2019-12-13 17:20:45 -08:00
Tom Hayward
0f5e2753a6 Fix typo cloudfront updatefrenquency
Signed-off-by: Tom Hayward <thayward@infoblox.com>
2019-12-13 12:11:42 -08:00
Derek McGowan
bdf3438b52
Merge pull request #2985 from novas0x2a/default-transport
make it possible to wrap the client transport in another one
2019-12-13 11:36:12 -08:00
Derek McGowan
29c300c106
Merge pull request #3050 from ducksecops/patch-1
bump golang to 1.13.4
2019-12-13 11:09:05 -08:00
Derek McGowan
4798651387
Merge pull request #3053 from adrian-plata/table-fix
Fixing broken table
2019-12-13 11:08:00 -08:00
Derek McGowan
f18781257e
Merge pull request #3061 from guillaumerose/reconciliate
Add pathspec for repo _layers directory and allow Repository.BlobStore to enumerate over blobs
2019-12-06 11:27:13 -08:00
Guillaume Rose
c9c3324300 Add unit tests for BlobEnumerator
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-06 09:38:13 +01:00
Manish Tomar
5538da4923 fixes to make layersPathSpec work
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Manish Tomar
fa7d949408 allow Repository.BlobStore to enumerate over blobs
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Manish Tomar
cf77113795 add pathspec for repo _layers directory
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Adrian Plata
b4694b0d2d Fixing broken table
Signed-off-by: Adrian Plata <adrian.plata@docker.com>
2019-11-26 17:38:40 -08:00
ducksecops
070cc010f7 bump golang to 1.13.4
Signed-off-by: Daniel Sutton <daniel@ducksecops.uk>
2019-11-13 23:15:11 +00:00
Derek McGowan
dee21c0394
Merge pull request #2991 from ad-m/patch-1
use latest version of alpine when building the Docker container
2019-10-24 15:54:08 -07:00
Ryan Abrams
ae2e973db9
Merge pull request #2748 from manishtomar/tag-digests
API to retrive tag's digests
2019-10-08 12:14:56 -07:00
Ryan Abrams
14b96e55d8
Merge pull request #3003 from laverya/fix-build-badges
use travis, not circle, build badge
2019-09-05 08:29:32 -07:00
Ryan Abrams
f656e60de5
Merge pull request #2984 from bouk/rempo
registry: Fix typo in RepositoryRemover warning
2019-09-05 08:23:18 -07:00
Ryan Abrams
740d4d1211
Merge pull request #2918 from dmathieu/test-blob-writer-write
Test httpBlobUpload.Write method
2019-09-05 08:16:36 -07:00
Andrew Lavery
cc97b94f5d
use travis, not circle, build badge
Signed-off-by: Andrew Lavery <laverya@umich.edu>
2019-09-04 15:38:24 -07:00
Derek McGowan
aeaeb84407
Merge pull request #3000 from adrian-plata/master
Adding deprecated schema instructions
2019-09-03 11:54:37 -07:00
Adrian Plata
07a50201c9 Adding deprecated schema instructions
Signed-off-by: Adrian Plata <adrian.plata@docker.com>
2019-09-03 11:44:28 -07:00
Bouke van der Bijl
1c481d34d9 registry: Fix typo in RepositoryRemover warning
Signed-off-by: Bouke van der Bijl <me@bou.ke>
2019-09-02 16:07:34 +00:00
Adam Dobrawy
a994bb839d use latest version of alpine when building the Docker container
Signed-off-by: Adam Dobrawy <naczelnik@jawnosc.tk>
2019-08-30 00:58:36 +02:00
Mike Lundy
c486db2d71
make it possible to wrap the client transport in another one
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
2019-08-22 17:37:47 -04:00
Ryan Abrams
1fb7fffdb2
Merge pull request #2950 from terinjokes/patches/swift-segment-hash
swift: correct segment path generation
2019-07-11 15:35:31 -07:00
Ryan Abrams
10f726344d
Merge pull request #2955 from alex-laties/master
allow for VERSION and REVISION to be passed in during docker builds
2019-07-11 15:30:23 -07:00
Ryan Abrams
8063102951
Merge pull request #2917 from dmathieu/repository-blob-resume
Implement Repository Blobs upload resuming
2019-07-11 15:28:05 -07:00
Derek McGowan
438b67feef
Merge pull request #2947 from tariq1890/update_versions
Update the versions of several dependencies
2019-07-08 13:33:36 -07:00
Ryan Abrams
be07be9904
Merge pull request #2522 from tifayuki/notification_metrics
Add notification metrics
2019-06-28 11:10:51 -07:00
Alex Laties
92d213d2c1 allow for VERSION and REVISION to be passed in during docker builds
Signed-off-by: Alex Laties <agl@tumblr.com>
2019-06-26 18:06:51 -04:00
Damien Mathieu
dd3bdee21c implement Repository Blobs upload resuming
Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-06-26 09:46:49 +02:00
Ryan Abrams
90dfea7952
Merge pull request #2921 from dmathieu/repository-serve-blob
Implement Repository ServeBlob
2019-06-25 19:07:38 -07:00
Ryan Abrams
6c72ec2e85
Merge pull request #2927 from dmathieu/blob-create-uuid
Handle Blob Create when the underlying registry doesn't provide 'Docker-Upload-UUID'
2019-06-25 19:06:46 -07:00
Ryan Abrams
ec84b86013
Merge pull request #2930 from dmathieu/extract-blob-resume
Extract blob upload resume into its own method
2019-06-25 19:06:16 -07:00
Damien Mathieu
898b1f2a53 test httpBlobUpload.Write method
Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-06-25 09:32:58 +02:00
Damien Mathieu
c5d5f938e3 fast-stop ServeBlob if we're doing a HEAD request
A registry pointing to ECR is having issues if we try loading the blob

Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-06-25 09:30:22 +02:00
Damien Mathieu
3800c47fd2 Implement Repository ServeBlob
Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-06-25 09:30:22 +02:00
Damien Mathieu
a45e5cb13f handle create blob if the uuid couldn't be retrieved from headers or URL
Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-06-25 09:29:38 +02:00
Damien Mathieu
8b31a894bd deduce blob UUID from location if it wasn't provided in the headers
Some registries (ECR) don't provide a `Docker-Upload-UUID` when creating
a blob. So we can't rely on that header. Fallback to reading it from the
URL.

Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-06-25 09:29:38 +02:00
Damien Mathieu
94097512db extract blob upload resume into its own method
I've found this logic being in a single method to be quite hard to get.
I believe extracting it makes it easier to read, as we can then more
easily see what the main method does and possibly ignore the intricacies
of `ResumeBlobUpload`.

Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-06-25 09:25:56 +02:00
Terin Stock
b23dd1ef37 swift: correct segment path generation
When uploading segments to Swift, the registry generates a random file,
by taking the hash of the container path and 32-bytes of random data.
The registry attempts to shard across multiple directory paths, by
taking the first three hex characters as leader.

The implementation in registry, unfortunately, takes the hash of
nothing, and appends it to the path and random data. This results in all
segments being created in one directory.

Fixes: #2407
Fixes: #2311
Signed-off-by: Terin Stock <terinjokes@gmail.com>
2019-06-22 23:44:43 -07:00
Ryan Abrams
79f6bcbe16
Merge pull request #2946 from tariq1890/upd_alpine
use latest version of alpine when building the Docker container
2019-06-20 10:38:41 -07:00
Tariq Ibrahim
afe29bb697
update the golang compiler version and the versions of several dependencies
Signed-off-by: Tariq Ibrahim <tariq181290@gmail.com>
2019-06-19 22:43:52 -07:00
Tariq Ibrahim
45b2d0498d
use latest version of alpine when building the Docker container
Signed-off-by: Tariq Ibrahim <tariq181290@gmail.com>
2019-06-19 22:21:07 -07:00
Ryan Abrams
84f47e7bb3
Merge pull request #2900 from sevki/no-token-err-fix
fix no error returned in fetchTokenWithOAuth
2019-06-19 19:16:18 -07:00
Ryan Abrams
6c329e56a2
Merge pull request #2920 from dmathieu/blob_writer_offset
Append the written bytes to the blob writer's size
2019-06-19 19:14:49 -07:00
Ryan Abrams
71309e96e4
Merge pull request #2809 from andyzhangx/ssl
change default Dockerfile to install ssl utils
2019-06-19 19:14:14 -07:00
Ryan Abrams
0c394fdd84
Merge pull request #2861 from yuwaMSFT2/master
Fixes #2835 Process Accept header MIME types in case-insensitive way
2019-06-19 19:09:29 -07:00
Ryan Abrams
d80a17d8e0
Merge pull request #2879 from tbe/fix-s3-ceph
Fix s3 driver for supporting ceph radosgw
2019-06-19 19:09:07 -07:00
Ryan Abrams
55287010ce
Merge pull request #2894 from jabrown85/fix-offset-typos
Fix typo: offest -> offset
2019-06-19 19:08:22 -07:00
Ryan Abrams
c2254191d4
Merge pull request #2941 from tariq1890/gomod
migrate to go modules from vndr
2019-06-19 18:41:26 -07:00
Tariq Ibrahim
5223c27422
migrate to go modules from vndr
Signed-off-by: Tariq Ibrahim <tariq181290@gmail.com>
2019-06-19 12:24:07 -07:00
Ryan Abrams
dcfe05ce6c
Merge pull request #2926 from tariq1890/no_rsc
replace rsc.io/letsencrypt in favour of golang.org/x/crypto
2019-06-05 15:34:28 -07:00
Tariq Ibrahim
8f9c8094fb
replace rsc.io/letsencrypt in favour of golang.org/x/crypto
Signed-off-by: Tariq Ibrahim <tariq181290@gmail.com>
2019-06-04 12:04:18 -07:00
Damien Mathieu
0e2d080a8a append the written bytes to the blob writer's size
Any byte written should append to the size. Otherwise, the full Size is
always zero.

Signed-off-by: Damien Mathieu <dmathieu@salesforce.com>
2019-05-09 14:17:58 +02:00
Ryan Abrams
3226863cbc
Merge pull request #2849 from Shawnpku/master
support Alibaba Cloud CDN storage middleware
2019-04-16 18:43:06 -07:00
Sevki Hasirci
5afbf32400 fix no error returned in fetchTokenWithOAuth
fetchTokenWithBasicAuth checks if a token is in the token response
but fetchTokenWithOAuth does not

these changes implements the same behaviour for the latter
returning a `ErrNoToken` if a token is not found in the resposne

Signed-off-by: Sevki Hasirci <sevki@cloudflare.com>
2019-04-14 11:05:59 +01:00
Jesse Brown
74f429a5ad Fix typo: offest -> offset
Signed-off-by: Jesse Brown <jabrown85@gmail.com>
2019-04-05 14:20:20 -05:00
Shawn Chen
51bb5cee5b import alicdn package
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-19 15:06:02 +08:00
Shawn Chen
fd77cf43a6 change package name & format document
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-18 11:35:46 +08:00
Thomas Berger
c18c6c33b2 S3 Driver: added comment for missing KeyCount workaround
Signed-off-by: Thomas Berger <loki@lokis-chaos.de>
2019-03-15 21:05:21 +01:00
Eohyung Lee
f877726503 Fix s3 driver for supporting ceph radosgw
Radosgw does not support S3 `GET Bucket` API v2 API but v1.
This API has backward compatibility, so most of this API is working
correctly but we can not get `KeyCount` in v1 API and which is only
for v2 API.

Signed-off-by: Eohyung Lee <liquidnuker@gmail.com>
2019-03-08 12:45:20 +01:00
Ryan Abrams
6d62eb1d4a
Merge pull request #2837 from vishesh92/fix-cloudfront-middleware
Fix cloudfront middleware
2019-03-04 16:42:08 -08:00
Shawn Chen
3390f32aec fix Context issue
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-04 17:48:32 +08:00
Shawn Chen
ae91d1f429 fix ci issue
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-04 17:17:57 +08:00
Shawnpku
6e10631d9c fix default cdn auth duration
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-03-04 14:53:48 +08:00
Vishesh Jindal
e1e72e9563
Fix cloudfront documentation formatting
Signed-off-by: Vishesh Jindal <vishesh92@gmail.com>
2019-03-02 08:58:52 +05:30
Vishesh Jindal
f9a0506191
Bugfix: Make ipfilteredby not required
Signed-off-by: Vishesh Jindal <vishesh92@gmail.com>
2019-03-02 08:58:52 +05:30
Ryan Abrams
d3ddc3572c
Merge pull request #2854 from manishtomar/log-authed-name
Log authorized username
2019-03-01 14:45:43 -08:00
Ryan Abrams
c192a281f8
Merge pull request #2813 from lucab/ups/spec-json-binary
registry: fix binary JSON content-type
2019-03-01 14:26:54 -08:00
Yu Wang
a683c7c235 Fixes #2835 Process Accept header MIME types in case-insensitive way
Use mime.ParseMediaType to parse the media types in Accept header in manifest request. Ignore the failed ones.

Signed-off-by: Yu Wang <yuwa@microsoft.com>
2019-02-21 15:11:41 -08:00
Shawnpku
bbc9885aa2 fix func name
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-02-20 15:54:21 +08:00
Honglin Feng
92a6436714 rename the metrics label
Signed-off-by: Honglin Feng <tifayuki@gmail.com>
2019-02-17 12:46:10 +08:00
Honglin Feng
d5a615b8c9 update the event number
Signed-off-by: Honglin Feng <tifayuki@gmail.com>
2019-02-15 21:55:17 +08:00
Honglin Feng
09a63caa37 run go fmt and goimports
Signed-off-by: Honglin Feng <tifayuki@gmail.com>
2019-02-15 21:14:57 +08:00
Honglin Feng
228bafca0b run go fmt
Signed-off-by: Honglin Feng <tifayuki@gmail.com>
2019-02-15 21:14:57 +08:00
Honglin Feng
76da6290b0 add label to the metrics
Signed-off-by: Honglin Feng <tifayuki@gmail.com>
2019-02-15 21:14:57 +08:00
tifayuki
8b70616846 Add notification metrics
It adds notification related prometheus metrics, including:
  - total count for events/success/failure/error
  - total count for notification per each status code
  - gauge of the pending notification queue

Signed-off-by: tifayuki <tifayuki@gmail.com>
2019-02-15 21:14:57 +08:00
Manish Tomar
ec6566c02b Log authorized username
This is useful to know which user pulled/pushed which repo.

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-13 08:49:37 -08:00
Shawnpku
3aa2a282f7 support alicdn middleware
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-02-11 15:11:26 +08:00
Derek McGowan
0d3efadf01
Merge pull request #2840 from manishtomar/fix-lint
Fix gometalint errors
2019-02-04 16:58:09 -08:00
Manish Tomar
48818fdea7 Remove err nil check
since type checking nil will not panic and return appropriately

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:42:44 -08:00
Ryan Abrams
411d6bcfd2
Merge pull request #2786 from thaJeztah/add_normalize_util
Add reference. ParseDockerRef utility function
2019-02-04 16:19:21 -08:00
Manish Tomar
da8db4666b Fix gometalint errors
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:01:04 -08:00
Derek McGowan
b75069ef13
Merge pull request #2791 from AliyunContainerService/support-oss-byok
Support BYOK for OSS storage driver
2019-01-16 16:37:37 -08:00
Ryan Abrams
b1fd12d0c1
Merge pull request #2808 from gregrebholz/feat/tls1.2only
Registry - make minimum TLS version user configurable
2019-01-14 09:31:36 -08:00
Luca Bruno
15b0204758
registry: fix binary JSON content-type
This fixes registry endpoints to return the proper `application/json`
content-type for JSON content, also updating spec examples for that.

As per IETF specification and IANA registry [0], the `application/json`
type is a binary media, so the content-type label does not need any
text-charset selector. Additionally, the media type definition
explicitly states that it has no required nor optional parameters,
which makes the current registry headers non-compliant.

[0]: https://www.iana.org/assignments/media-types/application/json

Signed-off-by: Luca Bruno <lucab@debian.org>
2019-01-14 09:04:42 +00:00
Greg Rebholz
cdb62b2b77 Registry - make minimum TLS version user configurable
Signed-off-by: J. Gregory Rebholz <gregrebholz@gmail.com>
2019-01-11 18:11:03 -05:00
andyzhangx
7df881dcbe change default Dockerfile to install ssl utils
Signed-off-by: andyzhangx <xiazhang@microsoft.com>
2019-01-10 03:56:42 +00:00
Derek McGowan
91b0f0559e
Merge pull request #2801 from caervs/ISS-2793
Add docs for autoredirect config parameter
2019-01-07 10:58:55 -08:00
Derek McGowan
980d1a696b
Merge pull request #2800 from davidswu/autoredirect
default autoredirect to false
2019-01-07 10:26:29 -08:00
Ryan Abrams
d1abdeb623 Add docs for autoredirect config parameter
Config parameter is user facing so should be documented.

Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
2019-01-04 13:11:20 -08:00
Sebastiaan van Stijn
0ac367fd6b
Add reference.ParseDockerRef utility function
ParseDockerRef normalizes the image reference following the docker
convention. This is added mainly for backward compatibility. The reference
returned can only be either tagged or digested. For reference contains both tag
and digest, the function returns digested reference, e.g.

    docker.io/library/busybox:latest@sha256:7cc4b5aefd1d0cadf8d97d4350462ba51c694ebca145b08d7d41b41acc8db5aa

will be returned as

    docker.io/library/busybox@sha256:7cc4b5aefd1d0cadf8d97d4350462ba51c694ebca145b08d7d41b41acc8db5aa.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-01-04 21:26:06 +01:00
David Wu
eb1a2cd911 default autoredirect to false
Signed-off-by: David Wu <david.wu@docker.com>
2019-01-04 11:05:12 -08:00
Li Yi
b7839211af Update doc for BYOK support in OSS storage driver
Change-Id: I195f29195f999e3e14ba921fd3435a6dc4788417
Signed-off-by: Li Yi <denverdino@gmail.com>
2018-12-25 09:32:16 +08:00
Li Yi
90bed67126 Support BYOK for OSS storage driver
Change-Id: I423ad03e63bd38aded3abfcba49079ff2fbb3b74
Signed-off-by: Li Yi <denverdino@gmail.com>
2018-12-25 08:30:40 +08:00
Derek McGowan
40b7b5830a
Merge pull request #2775 from caervs/release_notes_2.7
Release notes for 2.7
2018-12-03 16:05:39 -08:00
Ryan Abrams
08c6bbed05 Release notes for 2.7
Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
2018-11-29 16:35:06 -08:00
Derek McGowan
d9e1218235
Merge pull request #2772 from dmcgowan/add-arm-flag
Add GOARM flag to dockerfile
2018-11-29 15:15:00 -08:00
Derek McGowan
63f6c1205d
Add GOARM flag to dockerfile
When building with arm on alpine, GOARM should
be set to 6 by default.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-11-28 10:40:29 -08:00
Derek McGowan
aa985ba889
Merge pull request #2711 from davidswu/autoredirect
add autoredirect auth config
2018-11-27 15:48:25 -08:00
Derek McGowan
dd36fd3622
Merge pull request #2742 from tescherm/configure-bugsnag-logging
add bugsnag logrus hook
2018-11-27 15:08:17 -08:00
Manish Tomar
1251e51ad0 better name and updated tests
- use ManifestDigests name instead of Indexes
- update tests to validate against multiple tags

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-11-01 10:31:08 -07:00
Manish Tomar
9ebf151ac2 API to retrive tag's digests
Add an interface alongside TagStore that provides API to retreive
digests of all manifests that a tag historically pointed to. It also
includes currently linked tag.

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-10-29 18:19:05 -07:00
Matt Tescher
7c4d584e58 add bugsnag logrus hook
Signed-off-by: Matt Tescher <matthew.tescher@docker.com>
2018-10-25 14:52:10 -07:00
Olivier Gambier
93e082742a
Merge pull request #2734 from mgrachev/patch-1
Fix typo
2018-10-24 10:01:56 -07:00
Olivier Gambier
f7046a6d68
Merge pull request #2735 from tonistiigi/update-dockerfile
update Dockerfile to multi-stage
2018-10-24 10:01:01 -07:00
Grachev Mikhail
cd1648d62c Fix typo
Signed-off-by: Mikhail Grachev <work@mgrachev.com>
2018-10-18 17:54:19 +03:00
Tonis Tiigi
8a800e1292 update Dockerfile to multi-stage
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-15 15:48:05 -07:00
Olivier Gambier
1cb4180b1a
Merge pull request #2729 from liyongxin/master
typo fix about overridden
2018-10-02 15:04:33 -07:00
Olivier Gambier
451cd548a0
Merge pull request #2731 from mirake/fix-typos
Fix typo: commmand -> command
2018-10-02 15:03:57 -07:00
Rui Cao
6335cc258f Fix typo: commmand -> command
Signed-off-by: Rui Cao <ruicao@alauda.io>
2018-09-29 15:12:10 +08:00
Derek McGowan
17b3ff188d
Merge pull request #2730 from dmcgowan/version-update-2.7.0-rc.0
Update version to 2.7.0-rc.0
2018-09-28 16:17:04 -07:00
Derek McGowan
f08b3486c8
Update version to 2.7.0-rc.0
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-28 16:07:56 -07:00
Derek McGowan
f3adfea35b
Merge pull request #2721 from dmcgowan/release-notes-2.7.0-rc
Add release notes for 2.7.0 release candidates
2018-09-27 14:41:04 -07:00
Derek McGowan
e1817db884
Merge pull request #2720 from dmcgowan/update-release-process
Update release documents
2018-09-27 14:32:28 -07:00
Yongxin Li
de8636b78c typo fix about overridden
Signed-off-by: Yongxin Li <yxli@alauda.io>
2018-09-27 20:27:09 +08:00
Derek McGowan
97cb7f35b0
Update release documents
Remove redundant AUTHORS file in favor of relying
on .mailmap for keeping track of author names.
Move older changelogs to release directory.
Move release checklist to releases README.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-26 18:30:44 -07:00
Derek McGowan
2eb7a17225
Add 2.7.0-rc release notes
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-26 11:15:01 -07:00
Derek McGowan
06a4c2f61b
Update mailmap file
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-26 11:15:01 -07:00
Olivier Gambier
d37f816427
Merge pull request #2723 from mirake/fix-typos
Fix some typos
2018-09-25 15:34:16 -07:00
Rui Cao
569d18aef9 Fix some typos
Signed-off-by: Rui Cao <ruicao@alauda.io>
2018-09-24 09:05:44 +08:00
David Wu
2e1e6307dd add autoredirect to option
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-20 19:33:06 -07:00
David Wu
b2bd465760 fix checks
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-20 15:24:27 -07:00
Viktor Stanchev
f730f3ab77 add autoredirect auth config
It redirects the user to to the Host header's domain whenever they try to use
token auth.

Signed-off-by: David Wu <david.wu@docker.com>
2018-09-20 14:47:43 -07:00
Olivier Gambier
16128bbac4
Merge pull request #2707 from davidswu/go-1.11
remove dependencies on resumable
2018-09-20 12:47:44 -07:00
Olivier Gambier
b089e91688
Merge pull request #2712 from liyongxin/master
typo fix from existant to existent
2018-09-13 15:03:39 -07:00
liyongxin
6133840f49 typo fix from existant to existent
Signed-off-by: liyongxin <yxli@alauda.io>
Signed-off-by: Yongxin Li <yxli@alauda.io>
2018-09-13 19:37:13 +08:00
David Wu
a927fbdb9b track digest offset in blobwriter
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-12 14:53:27 -07:00
David Wu
d8bde9b97e remove go 1.9 and 1.10 checks from travis
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 16:32:27 -07:00
David Wu
bd41413d57 remove closenotifier
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 16:14:10 -07:00
David Wu
166874ade9 fix gofmt and goimports
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 16:14:10 -07:00
Derek McGowan
a5c2fdc5b8
Merge pull request #2705 from mirake/fix-typo
Fix some typos
2018-09-11 16:12:22 -07:00
David Wu
9da0f07c92 update travis with go 1.11
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 08:22:04 -07:00
David Wu
877d706b38 remove dependencies on resumable
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 08:21:27 -07:00
ruicao
d1f36d46c9 Fix some typos
Signed-off-by: ruicao <ruicao@alauda.io>
2018-09-07 18:13:53 +08:00
Derek McGowan
642075f42c
Merge pull request #2631 from whoshuu/feature/improve-gcs-driver
Improve gcs driver
2018-09-05 17:48:03 -07:00
Derek McGowan
15de837aa8
Merge pull request #2704 from dmcgowan/fix-2703
Fix registry stripping newlines from manifests
2018-09-05 17:46:56 -07:00
Huu Nguyen
7a195dd5ca Add back include_gcs build constraint
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Huu Nguyen
69299d93d9 Use existing jwtConf instead of creating a scoped one
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Tony Holdstock-Brown
f9187b2572 Add regulator to GCS
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Andrey Kostov
b424c3d870 Better error handling for GCS credential argument addition
Signed-off-by: Andrey Kostov <kostov.andrey@gmail.com>
2018-09-05 15:48:29 -07:00
Andrey Kostov
3f9f073cef Edit configuration.md to add gcs credentials option
Signed-off-by: Andrey Kostov <kostov.andrey@gmail.com>
2018-09-05 15:48:29 -07:00
Andrey Kostov
78238ef1a0 Add credentials argument for GCS driver
Signed-off-by: Andrey Kostov <kostov.andrey@gmail.com>
2018-09-05 15:48:29 -07:00
Olivier Gambier
efa4c3bb5f
Merge pull request #2702 from caervs/fix_path_enumeration
Ignore missing paths during enumeration
2018-09-05 15:44:52 -07:00
Olivier Gambier
6d66d0367e
Merge pull request #2698 from cquon/swift_vendor
Update ncw/swift Vendor Package
2018-09-05 15:26:25 -07:00
Derek McGowan
c88728f217
Fix registry stripping newlines from manifests
Content must be preserved exactly

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-05 13:40:42 -07:00
Ryan Abrams
6b73a9ab89 Ignore missing paths during enumeration
It's possible to run into a race condition in which the enumerator lists
lots of repositories and then starts the long process of enumerating through
them. In that time if someone deletes a repo, the enumerator may error out.

Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
2018-09-05 10:17:08 -07:00
Corey Quon
fd32d5f962
update github.com/ncw/swift package in vendor to v1.0.40
Signed-off-by: Corey Quon <corey.quon@gmail.com>
2018-09-05 00:28:40 -07:00
mlmhl
5a74b806f0
update github.com/ncw/swift package in vendor to avoid potential memory leaks
Signed-off-by: mlmhl <409107750@qq.com>
2018-09-05 00:28:40 -07:00
Olivier Gambier
9930542dc5
Merge pull request #2701 from davidswu/metalinter
fix gometalinter
2018-09-04 19:34:41 -07:00
David Wu
8d7e4cd388 fix goimports and gofmt
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-04 17:45:39 -07:00
Olivier Gambier
90705d2fb8
Merge pull request #2362 from twistlock/populate_htpasswd
Create and populate htpasswd file if missing
2018-08-31 00:25:37 -07:00
Derek McGowan
b12bd4004a
Merge pull request #2639 from andrew-leung/manifesteventlayers
Add configurable layers in manifest events
2018-08-28 16:03:05 -07:00
Derek McGowan
059f301d54
Merge pull request #2685 from manishtomar/mani-graceful-shutdown
Graceful shutdown
2018-08-27 14:24:53 -07:00
Manish Tomar
f95ac7db95 fix doc - thanks @dmp42
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-24 14:57:26 -07:00
Olivier Gambier
3354cf98e3
Merge pull request #2680 from manishtomar/mani-fix-mem-leak
fix memory leak introduced in PR #2648
2018-08-24 14:35:12 -07:00
Derek McGowan
ef859e1b21
Merge pull request #2474 from vikstrous/disable-v1-master
disable schema1 by default, add a config flag to enable it
2018-08-24 10:58:39 -07:00
Olivier Gambier
90070b3367
Merge pull request #2694 from caervs/fix_nginx_spacing
Replace tab with space in nginx config
2018-08-23 16:08:33 -07:00
Ryan Abrams
0101db11ef Replace tab with space in nginx config
Quick follow-up to #9223. This gives better consistency and readability to the
file.

Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
2018-08-23 16:06:46 -07:00
Olivier Gambier
1300e580d1
Merge pull request #2328 from stepyu/patch-1
Fix the sentence
2018-08-23 16:02:07 -07:00
Olivier Gambier
16eb3b658d
Merge pull request #2368 from monikakatiyar16/patch-1
Added nginx config for uploading large sized blobs
2018-08-23 15:59:47 -07:00
Derek McGowan
02bf4a2887
Merge pull request #2689 from dmp42/reviewers
Add new reviewers
2018-08-20 14:24:02 -07:00
Olivier
610440eb19 Add new reviewers
Signed-off-by: Olivier <o+github@gambier.email>
2018-08-20 12:24:15 -07:00
Olivier
ee94aa48d7
Merge pull request #2690 from dmcgowan/remove-aaron-maintainers
Remove Aaron Lehmann from maintainers file
2018-08-20 12:22:46 -07:00
Olivier
53bd46af5c
Merge pull request #2651 from mikebrow/manifest-version-test-on-put
adds validation testing for schema version values
2018-08-20 12:19:40 -07:00
Olivier
6411087274
Merge pull request #2681 from dmcgowan/update-yaml
Update yaml parser
2018-08-20 12:18:21 -07:00
Derek McGowan
0a302c7fa9
Remove Aaron Lehmann from maintainers file
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-20 11:47:54 -07:00
Derek McGowan
633401c9e8
Merge pull request #2687 from davidswu/remove-region-validation
use aws sdk to validate regions
2018-08-20 11:11:49 -07:00
David Wu
0b0d470281 use aws sdk to validate regions
Signed-off-by: David Wu <david.wu@docker.com>
2018-08-20 11:02:14 -07:00
Andrew Leung
5e4b81a578 Use references terminology instead of layers.
Signed-off-by: Andrew Leung <anwleung@gmail.com>
2018-08-20 10:01:40 -07:00
Manish Tomar
40efb602d6
Add support to gracefully shutdown the server
This is done by draining the connections for configured time after registry receives a SIGTERM signal.
This adds a `draintimeout` setting under `HTTP`. Registry doesn't drain
if draintimeout is not provided.

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-20 10:01:26 -07:00
Derek McGowan
9bf62ca7b3
Merge pull request #2678 from caervs/bump_aws
Bump aws sdk to v1.15.11
2018-08-14 19:07:50 -07:00
Derek McGowan
f0ee5720a5
Update yaml parser
Mark the top level Loglevel field as deprecated

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-14 16:42:36 -07:00
Manish Tomar
13f8189f2a fix memory leak introduced in PR #2648
context.App.repoRemover is single registry instance stored throughout
app run. It was wrapped in another remover when processing each request.
This remover happened to be remover got from previous request. This way
every remover created was stored in infinite linked list causing memory
leak. Fixing it by storing the wrapped remover inside the request context
which will get gced when request context is gced. This was introduced in
PR #2648.

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-14 16:41:45 -07:00
Derek McGowan
94deea2951
Merge pull request #2679 from dmcgowan/revert-bad-vendor-update
Revert "Fixed customer facing grammar and spelling"
2018-08-14 14:59:24 -07:00
Derek McGowan
de21eb6c96
Revert "Fixed customer facing grammar and spelling"
This reverts commit a64394ece5.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-14 14:24:01 -07:00
Ryan Abrams
57212c909b Bump aws sdk to v1.15.11
This is the latest official release for this dependency

Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
2018-08-14 12:56:19 -07:00
Olivier
c0995537be
Merge pull request #2674 from Parth/patch-1
Fixed customer facing grammar and spelling
2018-08-14 12:46:29 -07:00
Mike Brown
2fdb2ac270 adds validation testing for schema version values
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-08-14 08:53:54 -05:00
Parth Mehrotra
a64394ece5 Fixed customer facing grammar and spelling
Indentation is spelled wrong as intendation everywhere the word occurs

Can either be:
> "found a tab character that violates indentation"
or
> "found tab characters that violate indentation"

Signed-off-by: Parth Mehrotra <parth@mehrotra.me>
2018-08-11 12:38:35 -04:00
Derek McGowan
5f37adaa41
Merge pull request #2673 from dmp42/TLS
Remove ciphers that do not support perfect forward secrecy
2018-08-10 16:03:59 -07:00
Derek McGowan
88530ef7a0
Merge pull request #2671 from dmp42/goamz
Remove goamz
2018-08-10 15:54:04 -07:00
Olivier
7d9f067716
Merge pull request #2632 from whoshuu/feature/improve-s3-driver
Improve s3 driver
2018-08-10 15:01:11 -07:00
Olivier
d260b18f2f Remove ciphers that do not support perfect forward secrecy
Signed-off-by: Olivier <o+github@gambier.email>
2018-08-10 14:58:51 -07:00
dmp
9caa7a81bc Remove goamz
Signed-off-by: Olivier <o+github@gambier.email>
2018-08-10 11:11:53 -07:00
Olivier
b7446e89bf
Merge pull request #2599 from legionus/storage-inmemory-deadlock
Fix deadlock in the inmemory storage driver
2018-08-09 15:26:12 -07:00
Olivier
0cbe144826
Merge pull request #2602 from dmage/errmessage
Use e.Message field instead of e.Code.Message()
2018-08-09 15:21:32 -07:00
Olivier
4a75b72fd3
Merge pull request #2662 from dmcgowan/enable-structcheck
Enable static checks
2018-08-07 19:01:38 -07:00
Olivier
dc35ae6421
Merge pull request #2598 from dmcgowan/update-maintainers
Update project maintainers file
2018-08-07 18:59:36 -07:00
Olivier
eefe9670bd
Merge pull request #2661 from dmcgowan/enable-goimports
Enable goimports check
2018-08-07 14:59:49 -07:00
Derek McGowan
003aa051b4
Merge pull request #2648 from manishtomar/tag-deleted-event
add repo and tag deletion event
2018-08-07 11:50:56 -07:00
Derek McGowan
db0a4ec1c8
Enable static checks
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 14:49:11 -07:00
Derek McGowan
795e11d5fb
Enable goimports check
Validates that goimports has been run on all files

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 14:37:44 -07:00
Derek McGowan
32e2260be2
Enable struct check
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 13:55:43 -07:00
Manish Tomar
8c05756141 lint fix
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-06 09:46:42 -07:00
Olivier
dc53e37d98
Merge pull request #2657 from dmcgowan/add-travis-ci
Update build to use travis
2018-08-05 21:14:54 -07:00
Manish Tomar
e4d5a0a17c Add documentation
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-03 14:08:00 -07:00
Manish Tomar
8f6758278d take handler update
forgot to commit this earlier

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-02 23:13:37 -07:00
Manish Tomar
328069bb4d add support for repo deleted event also
by having another interface RepositoryRemover that is implemented by
registry instance and is injected in app context for event tracking

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-02 23:05:05 -07:00
Derek McGowan
dc49f84dcc
Update build to use travis
Update Makefile targets

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-01 16:54:54 -07:00
Andrew Leung
54aef6251e Fix typo in manifestLayers variable name
Signed-off-by: Andrew Leung <anwleung@gmail.com>
2018-07-29 08:01:21 -07:00
Derek McGowan
0dae0957e5
Merge pull request #2076 from mikebrow/ocitype
Adds support for oci manifests and manifestlists
2018-07-20 10:21:23 -07:00
Mike Brown
20aecf1d7b added test for initial oci schema version
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-07-19 19:41:31 -05:00
Mike Brown
5f588fbf9b address review comment regarding panic use
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-07-19 16:07:26 -05:00
Manish Tomar
0d8f4ac7b8 add tag deletion event
whenever a tag is deleted an event is sent out via the regular
notification channels

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-07-18 11:34:09 -07:00
Andrew Leung
276fdce3d9 Add configurable layers in manifest events
Signed-off-by: Andrew Leung <anwleung@gmail.com>
2018-06-27 09:27:24 -07:00
Mike Brown
e8d7941ca6 address lint and gofmt issues
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-25 15:45:15 -05:00
Huu Nguyen
f5c6357c6d Update s3_test to account for SkipVerify parameter
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-06-22 12:00:28 -07:00
Huu Nguyen
7655a3d91f Add option to skip certificate verification for the s3 driver
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-06-22 12:00:21 -07:00
Mike Brown
321d636e76 Merge pull request #1 from owtaylor/oci-media-types
Handle OCI manifests and image indexes without a media type

Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-19 11:46:49 -05:00
Owen W. Taylor
132abc6de5 Test storing OCI image manifests and indexes with/without a media type
OCI Image manifests and indexes are supported both with and without
an embeded MediaType (the field is reserved according to the spec).
Test storing and retrieving both types from the manifest store.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2018-06-19 11:24:25 -04:00
Owen W. Taylor
60d9c5dfad Handle OCI manifests and image indexes without a media type
In the OCI image specification, the MediaType field is reserved
and otherwise undefined; assume that manifests without a media
in storage are OCI images or image indexes, and determine which
by looking at what fields are in the JSON. We do keep a check
that when unmarshalling an OCI image or image index, if it has
a MediaType field, it must match that media type of the upload.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2018-06-19 11:24:25 -04:00
Owen W. Taylor
1d47ef7b80 Check media types when unmarshalling manifests
When unmarshalling manifests from JSON, check that the MediaType field
corresponds to the type that we are unmarshalling as. This makes sure
that when we retrieve a manifest from the manifest store, it will have
the same type as it was handled as before storing it in the manifest
store.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2018-06-19 11:24:25 -04:00
Mike Brown
f6224f78ba register oci image index
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 15:18:59 -05:00
Mike Brown
ad7ab0853c folow commit 9c88801a12
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
7b47fb13cf update url policy support; testing for annoations in index
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
f186e1da1c adding some support for annotations at the manifest level
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
ec2aa05cdf addressing comments from stevvooe
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
c1532332ad update to image spec v1.0.0
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
6bae7ca597 refactor adding enum for storage types
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
9e3f78b8c8 addresses minor debug comments
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
b0cef05626 removes oci. namespace feature
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
426afb3a4c address get manifest issue with oci. namespace; and comment descriptions
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
fcaffa38bc vendor oci image-spec
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
c94f28805e OCI media types; annotation support; oci index
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Mike Brown
6fcea22b0a add an ocischema manifest handler for the registry
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Mike Brown
9986e8ca7c adds support for oci manifests and manifestlists
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Derek McGowan
749f6afb45
Merge pull request #2603 from yuwaMSFT2/newazuresdk
closes #2496 and #2552
2018-06-11 11:39:26 -07:00
Yu Wang
62797237b9 closes #2496 and #2552
Update Azure SDK with release v16.2.1
Update Azure autorest SDK with release v10.8.1

Signed-off-by: Yu Wang <yuwa@microsoft.com>
2018-06-08 16:53:02 -07:00
Oleg Bulatov
7bc438a534
Use e.Message field instead of e.Code.Message()
Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2018-06-04 17:29:46 +02:00
Gladkov Alexey
fc7e8f42d7 Fix deadlock in the inmemory storage driver
According golang documentation [1]: no goroutine should expect to be
able to acquire a read lock until the initial read lock is released.

[1] https://golang.org/pkg/sync/#RWMutex

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>
2018-05-30 18:01:08 +02:00
Derek McGowan
86b030c1ca
Update project maintainers file
Include rules and roles in file.
Remove inactive maintainers who have already acknowledged
leaving the maintainer role.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-05-29 16:02:45 -07:00
Derek McGowan
f0cc927784
Merge pull request #2582 from thaJeztah/bump-golang-1.10
Update Golang to 1.10
2018-05-22 10:56:53 -07:00
Sebastiaan van Stijn
7f02f9e450
Update Golang to 1.10
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-05-22 19:42:22 +02:00
Derek McGowan
ed42a4909a
Merge pull request #2594 from dmcgowan/update-integration-test-certs
Update integration test certificates
2018-05-21 17:01:25 -07:00
Derek McGowan
dabcfdc5a6
Update for vndr update
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-05-21 16:33:38 -07:00
Derek McGowan
b4dd9b4376
Update certificates
Set expiration to 10 years

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-05-21 16:00:52 -07:00
Derek McGowan
83389a1480
Merge pull request #2556 from justincormack/fewer-gorillas
Remove gorilla/context from vendor.conf
2018-03-27 13:24:08 -07:00
Justin Cormack
fb7b0ddfc3
Remove gorilla/context from vendor.conf
This has been unused since we switched to Go context.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2018-03-26 15:46:07 -07:00
Olivier Gambier
23bef416bd
Merge pull request #2538 from tianon/challanges
Fix minor "Challanges" typo
2018-03-21 09:58:14 -07:00
Derek McGowan
607ae5d128
Merge pull request #2501 from xiaonancc77/master
Added ignore event types into notifications
2018-03-14 10:37:02 -07:00
Stephen Day
34c706e759
Merge pull request #2549 from miminar/redis-delete-key
redis: delete right size attribute
2018-03-13 16:12:15 -07:00
Michal Minář
492844e09c redis: delete right size attribute
Signed-off-by: Michal Minář <miminar@redhat.com>
2018-03-13 15:25:32 +01:00
elsanli(李楠)
fc1d3647c6 Added ignore event typs into notifications
Signed-off-by: elsanli(李楠) <elsanli@tencent.com>
2018-03-13 16:00:44 +08:00
Tianon Gravi
f2805894c8 Fix minor "Challanges" typo
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2018-03-01 11:25:05 -08:00
Stephen Day
6fca8d6e67
Merge pull request #2535 from xiaods/master
fix out of date cli arg for registry version
2018-02-27 15:34:29 -08:00
Deshi Xiao
8b93700d99 fix out of date cli arg for registry version
Signed-off-by: Deshi Xiao <xiaods@gmail.com>
2018-02-28 06:46:06 +08:00
Stephen Day
6664ec7039
Merge pull request #2466 from tifayuki/prometheus_go_metrics
add prometheus metrics
2018-02-09 15:49:04 -08:00
tifayuki
e3c37a46e2 Add Prometheus Metrics
at the first iteration, only the following metrics are collected:

  - HTTP metrics of each API endpoint
  - cache counter for request/hit/miss
  - histogram of storage actions, including:
    GetContent, PutContent, Stat, List, Move, and Delete

Signed-off-by: tifayuki <tifayuki@gmail.com>
2018-02-09 14:27:51 -08:00
Derek McGowan
13076371a6
Merge pull request #2520 from ywk253100/ignore_path_not_found_error
Ignore path not found error when look up tags
2018-02-09 11:19:52 -08:00
Wenkai Yin
005c6e0236 ignore path not found error when look up tags
Signed-off-by: Wenkai Yin <yinw@vmware.com>
2018-02-08 17:32:25 +08:00
Stephen Day
d707ea2428
Merge pull request #2469 from felixbuenemann/letsencrypt-host-whitelist-support
Let's Encrypt Host Whitelist Support
2018-02-06 16:11:20 -08:00
Stephen J Day
1ba5b3b553
registry/storage: ignore missing tag on delete
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-02-06 15:25:51 -08:00
Stephen Day
fea8bd5114
Merge pull request #2471 from GarageDeveloper/master
fixes #2249: sanitize tenant and tenantid
2018-02-02 16:29:34 -08:00
Derek McGowan
85b4d46b4f
Merge pull request #2507 from mistyhacks/fix-storage-driver-link
Fixed broken storage driver link
2018-02-01 13:49:34 -08:00
Felix Bünemann
4ecb17cc4c registry: support whitelisting letsencrypt hosts
This adds a configuration setting `HTTP.TLS.LetsEncrypt.Hosts` which can
be set to a list of hosts that the registry will whitelist for retrieving
certificates from Let's Encrypt. HTTPS connections with SNI hostnames
that are not whitelisted will be closed with an "unknown host" error.
It is required to avoid lots of unsuccessful registrations attempts that
are triggered by malicious clients connecting with bogus SNI hostnames.

NOTE: Due to a bug in the deprecated vendored rsc.io/letsencrypt library
clearing the host list requires deleting or editing of the cachefile to
reset the hosts list to null.

Signed-off-by: Felix Buenemann <felix.buenemann@gmail.com>
2018-02-01 21:16:58 +01:00
Misty Stanley-Jones
a4c32bce50 Fixed broken storage driver link
Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2018-01-25 14:54:50 -08:00
Derek McGowan
5cb406d511
Merge pull request #2455 from sargun/make-walk-faster
s3: improve walk performance
2018-01-19 13:00:03 -08:00
Sargun Dhillon
cbcbcb02c5 Remove old walk function
This removes the old global walk function, and changes all
the code to use the per-driver walk functions.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-18 13:06:07 -08:00
Sargun Dhillon
c7b0da2622 Use the new walk method for catalog enumeration
This change is primarily to make GC faster.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-18 11:58:46 -08:00
Derek McGowan
b5db8eeeb8
Merge pull request #2500 from stevvooe/remove-net-context-vestiges
context, registry/handlers: remove net/context references
2018-01-18 10:26:24 -08:00
Stephen J Day
585cdeb571
context, registry/handlers: remove net/context references
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-01-17 16:01:41 -08:00
Sargun Dhillon
35b29a609e Use the New Driver Walk method for catalog enumeration
This changes the Walk Method used for catalog enumeration. Just to show
how much an effect this has on our s3 storage:
Original:
List calls: 6839

real    3m16.636s
user    0m0.000s
sys    0m0.016s

New:
ListObjectsV2 Calls: 1805

real    0m49.970s
user    0m0.008s
sys    0m0.000s

This is because it no longer performs a list and stat per item, and instead
is able to use the metadata gained from the list as a replacement to stat.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Sargun Dhillon
32ac467992 Introduce Walk Method Per Storage Driver
Move the Walk types into registry/storage/driver, and add a Walk method to each
storage driver. Although this is yet another API to implement, there is a fall
back implementation that relies on List and Stat. For some filesystems this is
very slow.

Also, this WalkDir Method conforms better do a traditional WalkDir (a la filepath).

This change is in preparation for refactoring.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Sargun Dhillon
9f664468ea Upgrade AWS library to v1.12.36
This upgrade, and vendors aws-sdk-go to version v1.12.36.
This is because it has new API calls accessible to the S3 client,
specifically S3.ListObjectsV2PagesWithContext

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Derek McGowan
277ed486c9
Merge pull request #2302 from zetaab/oma
add possibility to clean untagged manifests
2018-01-05 15:27:52 -08:00
Jesse Haka
ff87ad884c add possibility to clean untagged manifests
add tests

add possibility to clean untagged manifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add dry tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove underscores

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fixes

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

opts struct+use camelcase

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

doublecheck manifest in paths.go

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add gofmt

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix lint

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add log print

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move log to dryrun as well

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove counter

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove manifest tag references

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add tag to tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

manifestsWithoutTags -> removeUntagged

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove RemoveManifestTagReferences and use removemanifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove pathfor

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move removemanifest out of manifestenumerator, it does not work correctly if we delete stuff in it

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix context -> dcontext

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix gofmt
2018-01-05 12:20:01 +02:00
Stephen Day
db0bc08b85
Merge pull request #2484 from perlun/patch-1
Update CONTRIBUTING.md
2018-01-02 16:07:18 -08:00
Stephen Day
70c5e31509
Merge pull request #2485 from perlun/patch-2
api.md: Fixed incorrect grammar
2018-01-02 16:06:25 -08:00
Per Lundberg
aac2f6c8b7 api.md: Fixed incorrect grammar
I noted this while reading at https://docs.docker.com/registry/spec/api/

Signed-off-by: Per Lundberg <perlun@gmail.com>
2018-01-02 12:49:31 +02:00
Per Lundberg
eaa90a3c6d Update CONTRIBUTING.md
Fixed the headers. They were rendering incorrectly on GitHub for some weird reason; I copied the strings and rewrote the ## characters at the beginning and now they render as they should.

Signed-off-by: Per Lundberg <per.lundberg@ecraft.com>
2017-12-29 14:41:43 +02:00
Viktor Stanchev
e9864ce8b9 disable schema1 by default, add a config flag to enable it
port of #2473

Signed-off-by: Viktor Stanchev <me@viktorstanchev.com>
2017-12-19 10:23:25 -08:00
Raphaël Enrici
8777e97b72 fixes #2249: sanitize tenant and tenantid
If tenant or tenantid are passed as env variables, we systematically use Sprint to make sure they are string and not integer as it would make mapstructure fail.

Signed-off-by: Raphaël Enrici <raphael@root-42.com>
2017-12-17 18:33:34 +01:00
Derek McGowan
f411848591
Merge pull request #2447 from tifayuki/cloudfront-s3-filter
add s3 region filters for cloudfront
2017-12-07 10:04:35 -08:00
tifayuki
e8ecc6dc55 add s3 region filters for cloudfront
Signed-off-by: tifayuki <tifayuki@gmail.com>
2017-12-01 15:58:58 -08:00
Stephen Day
bc3c7b0525
Merge pull request #2435 from zhouhaibing089/digestset-ut
digestset: refine some words on unit test
2017-11-22 16:42:22 -08:00
Stephen Day
8cedd7b3a4
Merge pull request #2448 from kklin/follow-relative-links
Properly follow relative links when listing tags
2017-11-22 16:38:47 -08:00
Stephen Day
1503fa2109
Merge pull request #2431 from mistyhacks/fix-keywords
Fix keyword format for downstream docs
2017-11-22 16:34:27 -08:00
Monika Katiyar
118c8ee1f1 Added nginx configuration for uploading large sized blobs (layer size) to registry when using nginx as reverse proxy from monikakatiyar16
Signed-off-by: Monika Katiyar <monika@jeavio.com>
2017-11-20 15:17:47 +05:30
Kevin Lin
1bfbeca726 Properly follow relative links when listing tags
The previous code assumed that the link returned when listing tags was
always absolute. However, some registries, such as quay.io, return the
link as a relative link (e.g. the second page for the quay.io/coreos/etcd
image is /v2/coreos/etcd/tags/list?next_page=<truncated>&n=50). Because
the relative link was retrieved directly, the fetch failed (with the
error `unsupported protocol scheme ""`).

Signed-off-by: Kevin Lin <kevin@kelda.io>
2017-11-18 22:04:19 -08:00
Gladkov Alexey
e69837454a Add tests for configuration parser
Signed-off-by: Gladkov Alexey <agladkov@redhat.com>
2017-11-15 17:01:19 +01:00
Gladkov Alexey
c9eba1a5bb Fix the pointer initialization
If the overwriteStruct() finds an uninitialized pointer, it tries to initialize it,
but does it incorrectly. It tries to assign a pointer to pointer, instead of pointer.

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>
2017-11-14 15:32:05 +01:00
gbarr01
e5b5e44386 Revert "Add text about win os version ordering"
This reverts commit 1a860d8c19.
2017-11-09 14:49:04 -08:00
gbarr01
1a860d8c19
Add text about win os version ordering 2017-11-09 14:15:00 -08:00
Liron Levin
c785740af7 Create and populate htpasswd file if missing
If htpasswd authentication option is configured but the htpasswd file is
missing, populate it with a default user and automatically generated
password.
The password will be printed to stdout.

Signed-off-by: Liron Levin <liron@twistlock.com>
2017-11-01 12:30:25 +02:00
zhouhaibing089
f74613907d digestset: refine some words on unit test
1. when lookup an entry which is missing, it should say NotFound.
2. when add duplicated entry, the entries size should be increased.
3. when add entry which has different algorithm, it should be allowed.

Signed-off-by: zhouhaibing089 <zhouhaibing089@gmail.com>
2017-10-31 16:33:36 +08:00
Misty Stanley-Jones
fda42e5ef9 Fix keyword format for downstream docs
Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2017-10-24 15:26:28 -07:00
Stephen Day
3800056b88 Merge pull request #2428 from zhouhaibing089/algo-import
cmd/digest: import crypto algorithms
2017-10-23 10:07:44 +02:00
zhouhaibing089
d66208108d cmd/digest: import crypto algorithms
the digest cli does not work if we do not import this two packages,
tested in go1.9. basically, we have to make several algorithms to
be available by calling crypto.RegisterHash in init functions.

Signed-off-by: zhouhaibing089 <zhouhaibing089@gmail.com>
2017-10-14 12:36:55 +08:00
Derek McGowan
7484e51bf6 Merge pull request #2402 from smarterclayton/order_manifest
Allow clients to request specific manifest media types
2017-10-11 10:17:12 -07:00
Clayton Coleman
3c5f85abd1
Allow clients to request specific manifest media types
The current registry/client sends the registered manifest types in
random order. Allow clients to request a single specific manifest type
or a preferred order as per the HTTP spec.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-10-03 17:06:11 -04:00
Derek McGowan
bb49a1685d Merge pull request #2416 from stevvooe/feed-the-linter
registry: feed the linter by removing redundant err check
2017-09-29 13:51:02 -07:00
Stephen J Day
4abf680c76
registry: feed the linter by removing redundant err check
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-29 13:27:49 -07:00
Derek McGowan
8710fa58ae Merge pull request #2377 from stevvooe/healthcheck-storage-ignore-notfound
registry/handlers: ignore notfound on storage driver healthcheck
2017-09-29 13:14:15 -07:00
Derek McGowan
5db89f0ca6 Merge pull request #2388 from tiborvass/remove-misused-context
Remove context in NewRegistry and NewRepository
2017-09-05 13:44:47 -07:00
Tibor Vass
13e0608bc8 gofmt -w -s registry/client/repository.go
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2017-09-05 13:14:43 -07:00
Tibor Vass
2c58ce1a7f Remove context in NewRegistry and NewRepository
The context parameter was either not used or misused.

Signed-off-by: Tibor Vass <teabee89@gmail.com>
2017-09-05 10:34:56 -07:00
Derek McGowan
30578ca329 Merge pull request #2384 from smarterclayton/client_retry
Support HEAD requests without Docker-Content-Digest header
2017-08-25 15:06:52 -07:00
Clayton Coleman
a2015272c1
Support HEAD requests without Docker-Content-Digest header
A statically hosted registry that responds correctly to GET with a
manifest will load the right digest (by looking at the manifest body and
calculating the digest). If the registry returns a HEAD without
`Docker-Content-Digest`, then the client Tags().Get() call will return
an empty digest.

This commit changes the client to fallback to loading the tag via GET if
the `Docker-Content-Digest` header is not set.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-08-25 17:18:01 -04:00
Derek McGowan
d9e0121fef Merge pull request #2382 from smarterclayton/scope_check
If the request already has the scope, don't fetchToken again
2017-08-24 15:50:00 -07:00
Clayton Coleman
23f8ca88e1
If the request already has the scope, don't force token fetch
AuthorizeRequest() injects the 'pull' scope if `from` is set
unconditionally. If the current token already has that scope, it will
be inserted into the scope list twice and `addedScopes` will be set to
true, resulting in a new token being fetched that has no net new scopes.

Instead, check whether `additionalScopes` are actually new.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-08-23 19:27:37 -04:00
Stephen J Day
1618b49d5b
registry/handlers: ignore notfound on storage driver healthcheck
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-21 15:04:31 -07:00
Stephen Day
5f6282db7d Merge pull request #2371 from lazize/patch-1
Fix registry version argument
2017-08-17 10:56:59 -07:00
Leonardo Azize Martins
3f771adca6 Fix registry version argument
Signed-off-by: Leonardo Azize Martins <leonardo.azize@scania.com>
2017-08-17 08:35:39 +02:00
Derek McGowan
e0b4f55f2b Merge pull request #2366 from stevvooe/remove-logging-dependencies
registry: remove dependency on logrus for client
2017-08-15 11:03:52 -07:00
Stephen J Day
860b28c5b9
registry: remove dependency on logrus for client
To simplify the vendoring story for the client, we have now removed the
requirement for `logrus` and the forked `context` package (usually
imported as `dcontext`). We inject the logger via the metrics tracker
for the blob cache and via options on the token handler. We preserve
logs on the proxy cache for that case. Clients expecting these log
messages may need to be updated accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-15 10:28:38 -07:00
Stephen Day
06fa77aa11 Merge pull request #2360 from stevvooe/remove-context-type
context: remove definition of Context
2017-08-11 16:30:26 -07:00
Stephen J Day
9c88801a12
context: remove definition of Context
Back in the before time, the best practices surrounding usage of Context
weren't quite worked out. We defined our own type to make usage easier.
As this packaged was used elsewhere, it make it more and more
challenging to integrate with the forked `Context` type. Now that it is
available in the standard library, we can just use that one directly.

To make usage more consistent, we now use `dcontext` when referring to
the distribution context package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-11 15:53:31 -07:00
Stephen Day
7a8efe719e Merge pull request #2350 from jonjohnsonjr/spec-typo
Fix typo in api spec
2017-08-04 12:10:18 -07:00
Jon Johnson
3d7803ec8c Fix typo in api spec
Signed-off-by: Jon Johnson <jonjohnson@google.com>
2017-07-28 10:13:32 -07:00
Derek McGowan
edc3ab29cd Merge pull request #2316 from igmor/logrus_case_dep_update
Update logrus and logrus-logstash-hook libraries
2017-07-26 10:46:10 -07:00
Aaron Lehmann
e18fe7d3f4 Merge pull request #2346 from riyazdf/security-list
add CONTRIBUTING info for security issues
2017-07-24 17:40:39 -07:00
Stephen Day
cb851f6598 Merge pull request #2345 from mstanleyjones/put-back-image
Put back v2-registry-auth.png
2017-07-24 15:01:48 -07:00
Riyaz Faizullabhoy
f7fb45f59a add CONTRIBUTING info for security issues
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
2017-07-24 14:58:32 -07:00
Misty Stanley-Jones
1d95716792 Put back v2-registry-auth.png
Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2017-07-24 14:27:16 -07:00
Stephen Day
91c507a39a Merge pull request #2340 from stevvooe/limit-payload-size
registry/{storage,handlers}: limit content sizes
2017-07-20 13:57:55 -07:00
YuJie
20f225005a Fix the sentence
Signed-off-by: YuJie <390282283@qq.com>
2017-07-20 05:52:56 +08:00
Derek McGowan
5cfdfbdce5 Merge pull request #2338 from stevvooe/api-typo
api: url typo in specification
2017-07-18 15:49:00 -07:00
Stephen J Day
5e5156afa3
api: url typo in specification
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-18 14:48:55 -07:00
Aaron Lehmann
fb90a182a9 Merge pull request #2336 from fate-grand-order/fixed
fix misspelling "algorithm" for cache/redis/redis.go
2017-07-18 11:56:19 -07:00
fate-grand-order
a11fe173d5 fix misspelling "algorithm" for cache/redis/redis.go
Signed-off-by: Helen Chen <chenjg@harmonycloud.cn>
2017-07-18 16:02:42 +08:00
Stephen J Day
55ea440428
registry/{storage,handlers}: limit content sizes
Under certain circumstances, the use of `StorageDriver.GetContent` can
result in unbounded memory allocations. In particualr, this happens when
accessing a layer through the manifests endpoint.

This problem is mitigated by setting a 4MB limit when using to access
content that may have been accepted from a user. In practice, this means
setting the limit with the use of `BlobProvider.Get` by wrapping
`StorageDriver.GetContent` in a helper that uses `StorageDriver.Reader`
with a `limitReader` that returns an error.

When mitigating this security issue, we also noticed that the size of
manifests uploaded to the registry is also unlimited. We apply similar
logic to the request body of payloads that are full buffered.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-06 17:13:39 -07:00
Aaron Lehmann
f86db6b226 Merge pull request #2315 from ipanova/mediatype-typo
Fixing image manifest schema2 medaitype typo in docs.
2017-06-27 11:58:08 -07:00
Aaron Lehmann
caa175c710 Merge pull request #2299 from dmage/regulator
Fix signalling Wait in regulator.enter
2017-06-23 15:25:35 -07:00
Derek McGowan
b9f76758ae Merge pull request #2310 from dmage/digest
Update Docker-Content-Digest if manifest list is rewritten
2017-06-23 15:07:39 -07:00
Igor Morozov
a97d7c0c15 moved Sirupsen to sirupsen on a case sensitive system
Signed-off-by: Igor Morozov <igor@adhoc05-sjc1.prod.uber.internal>
2017-06-23 20:28:48 +00:00
Ina Panova
5ccd03d28a Fixing image manifest schema2 medaitype typo in docs.
Signed-off-by: Ina Panova <ipanova@redhat.com>
2017-06-23 12:25:52 +02:00
Oleg Bulatov
5b1b6afae2 Update Docker-Content-Digest if manifest list is rewritten
If the client doesn't support manifest lists, the registry will
rewrite a manifest list into the old format. The Docker-Content-Digest
header should be updated in this case.

Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2017-06-16 16:20:14 +02:00
Oleg Bulatov
258345ba0d Fix signalling Wait in regulator.enter
In some conditions, regulator.exit may not send a signal to blocked
regulator.enter.

Let's assume we are in the critical section of regulator.exit and r.available
is equal to 0. And there are three more gorotines. One goroutine also executes
regulator.exit and waits for the lock. Rest run regulator.enter and wait for
the signal.

We send the signal, and after releasing the lock, there will be lock
contention:

  1. Wait from regulator.enter
  2. Lock from regulator.exit

If the winner is Lock from regulator.exit, we will not send another signal to
unlock the second Wait.

Signed-off-by: Oleg Bulatov <obulatov@redhat.com>
2017-06-02 15:41:55 +02:00
Stephen Day
1e2f10eb65 Merge pull request #2281 from sixeyed/patch-1
Add ARGs for cross-compiling
2017-05-24 13:58:24 -07:00
Derek McGowan
a1576d6e21 Merge pull request #2165 from kevinetc123/patch-1
fix some misspells
2017-05-24 11:18:20 -07:00
Derek McGowan
49bb3242da Merge pull request #2282 from mstanleyjones/put-back-architecture-doc
Put architecture.md back into distribution repo
2017-05-23 13:10:21 -07:00
Misty Stanley-Jones
b0f98e9382 Put architecture.md back into distribution repo
Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2017-05-23 11:24:11 -07:00
Derek McGowan
079f5b179d Merge pull request #2276 from nycholas/patch-1
Update README.md
2017-05-23 10:30:50 -07:00
Elton Stoneman
646fc9702c Add ARGs for cross-compiling
Add build args. Defaults to Linux/x64 so no change to existing image, but can build for other platforms - e.g.
```
docker build --build-arg GOOS=windows -t distribution-builder:windows .
```

Signed-off-by: Elton Stoneman <elton@sixeyed.com>
2017-05-23 10:02:15 +01:00
Nycholas de Oliveira e Oliveira
5573a13f15 Update README.md
Signed-off-by: Nycholas de Oliveira e Oliveira <nycholas@gmail.com>
2017-05-18 10:11:41 -03:00
Derek McGowan
a528cc1fc3 Merge pull request #2206 from stevvooe/update-resumable-hash
vendor: update resumable dependency
2017-05-17 13:09:54 -07:00
Stephen J Day
f01bcc8f62
vendor: update resumable dependency
Updates resumable hash implementation to Go 1.8 equivalent. This should
be a major speedup, since it includes a number of optimizations from Go
1.7.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-05-16 15:12:58 -07:00
Stephen Day
83f857ca12 Merge pull request #2207 from docker/update-go-1.8
go 1.8: update dockerfile and circle
2017-05-16 13:16:41 -07:00
Derek McGowan
9098f843d6 Merge pull request #2275 from IamBusy/master
Remove unused function
2017-05-16 11:51:13 -07:00
Derek McGowan
df1e488526
Update registry build to use go 1.8
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-05-16 11:33:57 -07:00
william wei
37ca688dc0 Remove unused function
Signed-off-by: william wei <1342247033@qq.com>
2017-05-16 17:47:13 +08:00
Derek McGowan
ce936a8591 Merge pull request #2262 from m-masataka/add_description_of_timeout
DOC:configuration.md: Change description of interval.
2017-05-05 11:45:43 -07:00
Masataka Mizukoshi
7d8dab5fdc DOC:configuration.md: Change description of interval.
Signed-off-by: Masataka Mizukoshi <m.mizukoshi.wakuwaku@gmail.com>
2017-05-04 19:14:29 +00:00
Stephen Day
a40abc69f2 Merge pull request #2245 from aaronlehmann/fetch-origin
[CI] Fetch origin before diffing
2017-05-04 11:51:08 -07:00
Stephen Day
7cc8e701c9 Merge pull request #2264 from luislobo/patch-1
Fixed #htpasswd link
2017-05-04 11:24:27 -07:00
Luis Lobo Borobia
212f47c318
Fixed #htpasswd link
Fixed #htpasswd link

Signed-off-by: Luis Lobo Borobia <luislobo@gmail.com>
2017-05-03 23:19:44 -05:00
Masataka Mizukoshi
05ac637aec DOC:configuration.md: Change description of interval.
Signed-off-by: Masataka Mizukoshi <m.mizukoshi.wakuwaku@gmail.com>
2017-05-01 09:28:32 +00:00
Derek McGowan
1d7824702b Merge pull request #2247 from yuwaMSFT2/revendorazuresdk
closes #2224 re-vendor the latest Azure Storage SDK for better performance
2017-04-27 10:54:25 -07:00
Derek McGowan
b7d5d9bfed Merge pull request #2258 from tt/return-early-to-prevent-nil-pointer-dereference
Return early to prevent nil pointer dereference
2017-04-27 09:38:03 -07:00
Troels Thomsen
1935c8d50b Return early to prevent nil pointer dereference
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2017-04-27 14:59:37 +02:00
Derek McGowan
f4a1d3e0d5 Merge pull request #2230 from tt/support-session-token
Support session token
2017-04-26 12:53:50 -07:00
Troels Thomsen
7f510ae9c9 Support session token
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2017-04-25 19:31:50 +02:00
Derek McGowan
e85ef3c019 Merge pull request #2253 from stevvooe/allow-more-digest-alg-separators
reference: allow more than 1 digest algorithm separator
2017-04-25 09:30:17 -07:00
Stephen J Day
3da015f8aa
reference: allow more than 1 digest algorithm separator
This updates the grammar to allow more than one digest algorithm
separator, matching the regular expression and intended grammar.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-04-24 15:14:12 -07:00
Yu Wang (UC)
ac05d143d8 closes #2224: re-vendor the latest Azure Storage SDK for better performance
Signed-off-by: Yu Wang <yuwa@microsoft.com>
2017-04-14 14:20:18 -07:00
Aaron Lehmann
2f728896a0 [CI] Fetch origin before diffing
It turns out that origin/master may not be up to date in CircleCI
checkouts. Fetch origin so that diffing for the vendor check can be done
correctly.

Note that this doesn't help the case of PRs against branches other than
master, but the worst case is that those will waste a few seconds doing
unnecessary vendor validation. Since those PRs are relatively rare, that
seems fine.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2017-04-12 13:18:06 -07:00
Stephen Day
c3e06c6069 Merge pull request #2239 from stevvooe/use-request-context
register/handlers: remove context manager
2017-04-10 16:14:28 -07:00
Stephen J Day
818ba4babf
vendor: update gorilla/mux to be compatible with Go 1.7
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-04-10 15:27:42 -07:00
Stephen J Day
1f0a9dbca0
register/handlers: remove context manager
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-04-10 15:27:30 -07:00
Stephen Day
a73ed75f5a Merge pull request #2241 from dmcgowan/vendor-fix
Update vendor directory to match expectation of vndr tool
2017-04-10 14:08:28 -07:00
Derek McGowan
b22c6b7a4e Update vendor directory to match expectation of vndr tool
Adds READMEs and enforces vendor is done at repository root

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-04-10 12:07:32 -07:00
Stephen Day
7b9ebdc54f Merge pull request #2215 from AlvinFeng/expires
Remove expires tag from s3 upload
2017-04-07 12:14:10 -07:00
Derek McGowan
0dd6ca97af Merge pull request #2238 from SantiagoTorres/fix_readme_broken_link
DOC: README: Fix broken link in migrator
2017-04-06 10:25:25 -07:00
Santiago Torres
364d2e4a5b
DOC: README: Fix broken link in migrator
A stray space in the link for the migrator repository wasn't allowing it
to be rendered properly in markdown viewers. Remove such a space to
avoid this.

Signed-off-by: Santiago Torres-Arias <torresariass@gmail.com>
2017-04-05 19:22:18 -04:00
Derek McGowan
0700fa570d Merge pull request #2219 from dmcgowan/fix-forwarded-logic
Fix forwarded logic
2017-03-21 10:14:25 -07:00
Derek McGowan
81a47d9766 Remove support for X-Forwarded-Port
Partially reverts change adding support for X-Forwarded-Port.
Changes the logic to prefer the standard Forwarded header over
X-Forwarded headers. Prefer forwarded "host" over "for" since
"for" represents the client and not the client's request.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-20 16:13:33 -07:00
Troels Thomsen
d4c3e88426
Add test for precendence with standard port
Signed-off-by: Troels Thomsen <troels@thomsen.io>
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-20 16:10:44 -07:00
Derek McGowan
11c74e07e0 Merge pull request #2218 from aaronlehmann/restore-vendor-dir
Makefile: Restore vendor directory after dep-validate
2017-03-20 11:28:17 -07:00
Derek McGowan
d2da3173db Merge pull request #2217 from aaronlehmann/fix-vendor-validation
Fix vendor validation
2017-03-20 11:28:00 -07:00
Aaron Lehmann
299b90b0bd Makefile: Restore vendor directory after dep-validate
Currently, this target makes changes to the vendor directory, which may
overwrite local changes or cause build errors. It's better to restore
the original vendor directory after running the check.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2017-03-20 11:12:03 -07:00
Aaron Lehmann
f6c62456b8 Fix vendor validation
The dep-validate target appears to never get invoked. In the CircleCI
build environment, "master" points to the commit under test. The
circle.yml fragment needs to compare again "origin/master" instead.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2017-03-20 11:11:10 -07:00
Derek McGowan
29e5cd5304 Merge pull request #2216 from fate-grand-order/fixed
fix some typos in notifications/event.go and sinks.go
2017-03-20 10:23:09 -07:00
fate-grand-order
eaf60fffee fix some typos in notifications/event.go and sinks.go
Signed-off-by: fate-grand-order <chenjg@harmonycloud.cn>
2017-03-20 20:23:11 +08:00
Alvin Feng
45bb7c9cc9 Remove expires tag from s3 upload
Signed-off-by: Alvin Feng <alvin4feng@yahoo.com>
2017-03-17 23:41:15 +00:00
Derek McGowan
08b06dc023 Merge pull request #2203 from clnperez/manifest-url-err
Better error message for BuildManifestURL if not tagged or digested
2017-03-03 13:22:46 -08:00
Derek McGowan
4ac39769dc Merge pull request #2155 from dmcgowan/update-release-checklist
Update release checklist with further details
2017-03-03 10:17:23 -08:00
Derek McGowan
d7905dc725 Merge pull request #2200 from fate-grand-order/fixed
fix typos in comment
2017-03-03 10:04:35 -08:00
Christy Perez
0810eba2ad Better error message for BuildManifestURL if not tagged or digested
Since there's no default case, if there's not a tag or digest you get
back a confusing error from the router about it not matching the
expected pattern.

Also redoing the tests for URLs a bit so that they can handle checking
for failures.

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>
2017-03-02 11:27:23 -06:00
fate-grand-order
3161f9d1fd fix typos in comment
Signed-off-by: fate-grand-order <chenjg@harmonycloud.cn>
2017-02-27 14:06:18 +08:00
Derek McGowan
50133d6372 Merge pull request #2195 from fate-grand-order/fixed
Use errors.New() to output the error message and fix some typos
2017-02-21 11:08:14 -08:00
fate-grand-order
6a8e2ca84f Use errors.New() to output the error message and fix some typos
Signed-off-by: fate-grand-order <chenjg@harmonycloud.cn>
2017-02-20 10:39:58 +08:00
Derek McGowan
df5327f76f Merge pull request #2187 from nwt/notifications-fix-expvar
notifications: fix expvar for Go 1.7
2017-02-17 11:36:21 -08:00
Derek McGowan
4f87c80073 Merge pull request #2192 from uhayate/refactor-code-style
refactor the code style in distribution/registry/storage/driver/s3-goamz/s3.go
2017-02-15 17:12:16 -08:00
Noah Treuhaft
9a58c91051 notifications: fix expvar for Go 1.7
Remove EndpointConfig.Transport from the return value of the
registry.notifications.endpoints expvar.Func.  It results in an empty
value for that expvar variable under Go 1.7 because it is a non-nil
*http.Transport, which Go 1.7 can no longer encode as JSON.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2017-02-14 10:51:20 -08:00
Aaron Lehmann
62d8d910b5 Merge pull request #2184 from sakeven/master
Check whether must use v4 auth in specific aws region ( storage driver s3-goamz )
2017-02-13 22:29:02 -08:00
sakeven
72bdf0e320 check whether must use v4 auth in specific aws region
Signed-off-by: sakeven <jc5930@sina.cn>
2017-02-14 10:42:20 +08:00
Aaron Lehmann
b38e5838b7 Merge pull request #2185 from yongtang/29936-validate-insecure-registry
Expose `DomainRegexp` from reference
2017-02-13 11:48:09 -08:00
uhayate
75c2e524a1 refactor the code style in distribution/registry/storage/driver/s3-goamz/s3.go
Signed-off-by: uhayate <uhayate.gong@daocloud.io>
2017-02-13 17:29:08 +08:00
kaiwentan
20036597bf fix some misspells
Signed-off-by: kaiwentan <kaiwentan@harmonycloud.cn>
2017-02-08 15:53:01 +08:00
Yong Tang
95daa793b8 Expose DomainRegexp from reference
This fix is based on:
https://github.com/docker/docker/pull/30746#discussion_r99650885

The goal is to reuse the `DomainRegexp` in docker to check
for `<host>:<port>` pattern.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2017-02-07 17:43:28 -08:00
Derek McGowan
b1993c9530 Merge pull request #2179 from Windfarer/master
Fixing acronym's case
2017-02-07 14:24:14 -08:00
Eric Yang
2d20471cd7 fixing acronyms case
Signed-off-by: Eric Yang <qizhao.yang@daocloud.io>
2017-01-31 01:29:30 +08:00
Aaron Lehmann
fb0bebc4b6 Merge pull request #2176 from stevvooe/clean-empty-blob-comments
registry/storage: clean up zero-length blob comments
2017-01-27 15:42:54 -08:00
Stephen J Day
7f565ed65a
registry/storage: clean up zero-length blob comments
A previous inspection of the code surrounding zero-length blobs led to
some interesting question. After inspection, it was found that the hash
was indeed for the empty string (""), and not an empty tar, so the code
was correct. The variable naming and comments have been updated
accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 15:28:30 -08:00
Derek McGowan
545102ea07 Merge pull request #2173 from aaronlehmann/tagnameonly
reference: Replace EnsureTagged with TagNameOnly
2017-01-26 14:35:53 -08:00
Aaron Lehmann
245ca4659e reference: Replace EnsureTagged with TagNameOnly
The common use case for this function is to add a default tag if the
reference only has a name. The current behavior only adds the default
tag if there is no *tag*, which requires most callers to check for a
digest. Change the behavior to only add default tags to name-only
references, and change the name to reflect this. The documentation
already described the new behavior, so it does not need to be changed.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2017-01-26 11:46:00 -08:00
Derek McGowan
7a0972304e Merge pull request #2167 from dmcgowan/fix-digest-reference-recursion
Fix recursion with digest String
2017-01-20 12:22:10 -08:00
Derek McGowan
69bdac7eff
Fix recursion with digest String
Update check for equal references to call the String method

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-20 11:38:07 -08:00
Derek McGowan
56b712f454 Merge pull request #2161 from dmcgowan/fix-test-panic
Update handler test to use WithName
2017-01-19 16:47:40 -08:00
Derek McGowan
1bcc73251d
Update handler test to use WithName
Fix test panic in handler test.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-19 15:21:40 -08:00
Derek McGowan
96b02e885a Merge pull request #2158 from fate-grand-order/master
fix misspell "lexicographically" in registry.go
2017-01-18 22:24:00 -08:00
fate-grand-order
e37d23c8a9 fix misspell "lexicographically" in registry.go
Signed-off-by: fate-grand-order <chenjg@harmonycloud.cn>
2017-01-19 10:42:04 +08:00
Derek McGowan
11cb04e994 Merge pull request #2142 from dmcgowan/reference-enforce-canonical-parsing
reference: ParseNamed updated to enforce canonical format
2017-01-18 17:06:08 -08:00
Derek McGowan
ea4b89db4e Merge pull request #2154 from justinsb/update_aws_sdk_go_1_6_12
Update aws-sdk-go dependency to 1.6.12
2017-01-18 14:10:59 -08:00
Derek McGowan
c91563ff3a Merge pull request #2156 from dmcgowan/helenxie-correct-variable-name
[carry #2153] Correct the variable name
2017-01-18 14:07:53 -08:00
Derek McGowan
a4853520e8 Merge pull request #2157 from dmcgowan/reference-tab-vs-spaces
Fix inconsistent spacing in reference grammar
2017-01-18 14:07:16 -08:00
Derek McGowan
d8fcbeeb71
Update registry to use WithName for creating Named values
The registry uses partial Named values which the named parsers
no longer support. To allow the registry service to continue
to operate without canonicalization, switch to use WithName.
In the future, the registry should start using fully canonical
values on the backend and WithName should no longer support
creating partial values.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-18 13:56:56 -08:00
Derek McGowan
6170ac53da
Update ParseNamed to require canonical form
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-18 13:56:56 -08:00
Derek McGowan
c6f5b6bf15
Fix inconsistent spacing in reference grammar
The identifier fields were using spaces while the rest
of the lines were using tabs!

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-18 13:34:02 -08:00
Stephen Day
0111f1e3cf Merge pull request #2138 from yuwaMSFT2/master
closes issue#2135 image pull returns 404 on manifest request if there is storage error
2017-01-18 12:02:55 -08:00
Helen-xie
90856017e6
Correct the variable name.
Signed-off-by: Helen-Xie <xieyulin821@harmonycloud.cn>
2017-01-18 11:55:20 -08:00
Justin Santa Barbara
9b4f24fc66 Update aws-sdk-go dependency to 1.6.12
1.6.12 = c6fc52983ea2375810aa38ddb5370e9cdf611716

Signed-off-by: Justin Santa Barbara <justin@fathomdb.com>
2017-01-18 14:49:28 -05:00
yuwaMSFT2
0bb696c5bf add test for manifest handler where storage driver fails
Signed-off-by: Yu Wang (UC) <yuwa@microsoft.com>
2017-01-18 11:39:45 -08:00
Derek McGowan
959659c3fa
Update release checklist with further details
Add details for release checklist for where to put the update
the changelist and which branch to cherry-pick from. Also add
more details around creating the tag and verifying the tag is
correct.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-18 11:07:35 -08:00
Derek McGowan
954b4e8154 Merge pull request #2140 from mfojtik/add-region
Add more regions to registry S3 storage driver
2017-01-18 10:29:49 -08:00
Derek McGowan
de5aa9cecc Merge pull request #2149 from dmcgowan/release-notes-2.6
Release notes for 2.6
2017-01-17 18:02:06 -08:00
Derek McGowan
99861b87bf
Update release notes for 2.6
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-17 16:16:05 -08:00
Derek McGowan
3bba9ff92b
Release notes for v2.6.0-rc2
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-17 15:38:45 -08:00
Derek McGowan
87cde81148 Merge pull request #2148 from jvrplmlmn/health-api-doc-fix
Documentation fix for package 'health'. Use proper import path for 'health/api'
2017-01-17 15:05:47 -08:00
Javier Palomo Almena
8b3d826801 Documentation fix for package 'health'. Use proper import path for 'health/api'
Signed-off-by: Javier Palomo Almena <javier.palomo.almena@gmail.com>
2017-01-16 18:38:22 +01:00
Stephen Day
44eff0143e Merge pull request #2143 from dmcgowan/reference-familiar-match
reference: move match function to helpers
2017-01-13 18:32:33 -08:00
Derek McGowan
3b0497541a
Move match function to helpers
Update match function to operate on familiar names

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-13 17:08:46 -08:00
Derek McGowan
7cb0c15480 Merge pull request #2137 from dmcgowan/reference-ambiguous-library-namespace
Remove ambiguity for unsupported official repository input
2017-01-13 16:22:41 -08:00
Derek McGowan
c965e24c8c Merge pull request #2139 from dmcgowan/unexport-normalized-named
Remove NormalizedNamed from public interface
2017-01-12 14:45:56 -08:00
Michal Fojtik
44dfd1b766 bump aws-sdk-go dependency (1.6.3)
Signed-off-by: Michal Fojtik <mfojtik@redhat.com>
2017-01-12 11:18:50 +01:00
Michal Fojtik
9e510d67f5 Add more regions to registry S3 storage driver
Namely adding ca-central-1, ap-south-1 and eu-west-1.

Signed-off-by: Michal Fojtik <mfojtik@redhat.com>
2017-01-11 22:38:24 +01:00
Derek McGowan
63cb8cf23b
Remove NormalizedNamed from public interface
The NormalizedNamed interface has shown to not be necessary for
integrating the change downstream. The FamiliarName and FamiliarString
helpers are the only used interface and allow hiding the normalized
completely.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-11 12:58:47 -08:00
Yu Wang (UC)
2e94c22fe4 issue#2135 image pull returns 404 on manifest request if there is storage error
When get manifest, the handler will try to retrieve it from storage driver. When storage driver is cloud storage, it can fail due to various reasons even if the manifest exists
(like 500, 503, etc. from storage server). Currently manifest handler blindly return 404 which can be confusing to user.

This change will return 404 if the manifest blob doesn't exist, and return 500 UnknownError for all other errors (consistent with the behavior of other handlers).

Signed-off-by: Yu Wang (UC) <yuwa@microsoft.com>
2017-01-11 12:34:11 -08:00
Derek McGowan
24cbdc41ba
Remove ambiguity for unsupported official repository input
Officials repositories always have 2 part names with the first part
being library and second part being the offical repository name. Names
with more than 2 parts should not hit the special case for official
repositories since they are not valid official repositories.
Add tests for this ambiguity and to ensure that 3 part names are
supports for the default repository, as used by the docker store.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-11 10:46:48 -08:00
Derek McGowan
69c7f303d5 Merge pull request #2134 from stevvooe/images-are-manifests
registy/handlers: move images to manifests
2017-01-11 10:37:34 -08:00
Stephen J Day
c91a68ca05
registy/handlers: move images to manifests
Once upon a time, we referred to manifests and images interchangably.
That simple past is no more. As we grow, we update our nomenclature and
so follows our code.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-10 16:09:15 -08:00
Stephen Day
ff68ca391b Merge pull request #2133 from dmcgowan/fix-notification-unit-test
[Carry #1990]Fix notification unit test
2017-01-10 11:08:51 -08:00
Derek McGowan
31a522977e Merge pull request #2132 from dmcgowan/split-hostname-test
Add tests for splithostname on normalized values
2017-01-10 10:31:29 -08:00
Derek McGowan
e468480bc3 Merge pull request #2101 from ahmetalpbalkan/pr-azure-update
azure: revendor + remove hacky solution in is404
2017-01-09 17:44:08 -08:00
Derek McGowan
efc32091e7
Fix unit test
If running test behide a proxy, we may get the error code
403 Forbidden which will fail line 135 for the last testcase.

Detail:
```
metrics not as expected: notifications.EndpointMetrics{Pending:0,
Events:0, Successes:4, Failures:0, Errors:0,
Statuses:map[string]int{"307 Temporary Redirect":0, "400 Bad Request":0,
"403 Forbidden":0, "200 OK":4}} !=
notifications.EndpointMetrics{Pending:0, Events:0, Successes:4,
Failures:0, Errors:0, Statuses:map[string]int{"400 Bad Request":0, "200
OK":4, "307 Temporary Redirect":0}}
```

Immediate close will fix that

Signed-off-by: Hu Keping <hukeping@huawei.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-01-09 17:31:00 -08:00
Derek McGowan
cc68bdb38b
Remove unneeded subpackage
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 17:27:52 -08:00
Ahmet Alp Balkan
4549791522
Update azure sdk version in vendor.conf
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2017-01-09 17:23:22 -08:00
Ahmet Alp Balkan
0a1ce58e2c
azure: revendor + remove hacky solution in is404
Removing the temporary workaround in is404() method by re-vendoring
the azure-sdk-for-go.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2017-01-09 17:22:28 -08:00
Derek McGowan
11406050af Merge pull request #2085 from dmcgowan/plugin-integration-test
Plugin integration test
2017-01-09 17:14:09 -08:00
Derek McGowan
0b1bcfda71
Add tests for splithostname on normalized values
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 16:52:05 -08:00
Derek McGowan
fd13a2e0a6
Remove container after export
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 16:44:37 -08:00
Derek McGowan
37758029fb
Add test for 1.13
Use rc4 for support with plugins change

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 16:44:37 -08:00
Derek McGowan
effe01aeae
Add plugin push/pull tests
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 16:44:37 -08:00
Derek McGowan
cfff433744
Update login helpers to handle email removal
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 16:44:36 -08:00
Aaron Lehmann
129ad8ea0c Merge pull request #2131 from dmcgowan/fix-reference-package
Fix reference package
2017-01-09 16:06:30 -08:00
Derek McGowan
320f72d252
Unexport NamedRepository
NamedRepository only needs to be used internally. Additionally this
interface may go away in the future and be merged with Named.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 15:44:59 -08:00
Derek McGowan
e0286a7de0
Use go-digest package
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-09 15:44:13 -08:00
Aaron Lehmann
dbc336e1ff Merge pull request #1778 from dmcgowan/reference-with-split-hostname
Integrate docker reference changes
2017-01-09 15:17:10 -08:00
Derek McGowan
2bc4a9459c Merge pull request #2121 from nwt/delete-action
Change DELETE action from "*" to "delete"
2017-01-09 12:34:23 -08:00
Stephen Day
7dba427612 Merge pull request #2111 from stevvooe/use-digest-package
digest: use new go-digest package
2017-01-06 16:25:57 -08:00
Noah Treuhaft
a33af0587b Add test for auth token with "*" action
Test that an auth token with the "*" action is allowed any action on its
resource.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2017-01-06 16:08:32 -08:00
Stephen J Day
72150bef10 vendor: add opencontainers/go-digest to vendor.conf
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-06 15:42:12 -08:00
Stephen J Day
532ec9f036
digest: migrate to opencontainers/go-digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-06 15:42:03 -08:00
Derek McGowan
429c75faf0
Add NormalizedName interface
Add interface for for a normalized name and corresponding parser for that type.
New normalized versions of all interfaces are not added since all type information is preserved on calls to Familiar.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-05 16:57:11 -08:00
Derek McGowan
042fe9bf46
Move docker reference functionality to reference package
Add normalization functions and Docker specific domain splitting to
reference package.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-05 15:43:27 -08:00
Derek McGowan
21db8e8597
Add identifier grammar
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-05 15:41:56 -08:00
Derek McGowan
9a43b8f696
Split apart repository reference into domain and path
Allows having other parsers which are capable of unambiguously keeping domain and path separated in a Reference type.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-05 15:41:56 -08:00
Derek McGowan
534b155cc8 Merge pull request #2128 from dmcgowan/update-logrus-dep
Update logrus vendor
2017-01-05 14:28:12 -08:00
Derek McGowan
1d6c1a6468
Update logrus vendor
closes #2125

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-05 11:40:18 -08:00
Derek McGowan
76f514b618 Merge pull request #2115 from yixizhang/health
Use app.driver.Stat for registry health check
2017-01-03 12:50:18 -08:00
Noah Treuhaft
ccb839e0e3 Change DELETE action from "*" to "delete"
With token authentication, requiring the "*" action for DELETE requests
makes it impossible to administratively lock a repository against pushes
and pulls but still allow deletion.  This change adds a new "delete"
action for DELETE requests to make that possible.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2017-01-03 12:27:12 -08:00
Derek McGowan
8e065ad239 Merge pull request #2116 from mstanleyjones/fix_config_ref_formatting
Improve formatting of configuration.md
2017-01-03 10:10:11 -08:00
Misty Stanley-Jones
6ee03f5da7 Improve formatting of configuration.md
Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2016-12-29 11:32:31 -08:00
yixi zhang
8e915d69f4 Use app.driver.Stat for registry health check
`app.driver.List` on `"/"` is very expensive if registry contains significant amount of images. And the result isn't used anyways.
In most (if not all) storage drivers, `Stat` has a cheaper implementation, so use it instead to achieve the same goal.

Signed-off-by: yixi zhang <yixi@memsql.com>
2016-12-21 17:12:43 -08:00
Derek McGowan
2d500932f2 Merge pull request #2110 from dmcgowan/manifest-add-uncompressed-type
Add uncompressed layer media type
2016-12-16 11:45:55 -08:00
Derek McGowan
9ab7b0ed43
Add uncompressed layer media type
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-12-16 11:29:51 -08:00
Derek McGowan
6b998f2755 Merge pull request #2109 from dmcgowan/oh-circleci-why-you-get-us-here
Fix small lint and unit test issues
2016-12-15 23:05:37 -08:00
Derek McGowan
b0327914c5
Fix small lint and unit test issues
Pull request ci was not running and caused these
minor issues to not be caught.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-12-15 22:38:38 -08:00
Derek McGowan
44282379f6 Merge pull request #2108 from andyxning/refactor_doc_comment
refactor doc comment
2016-12-15 21:03:29 -08:00
Stephen Day
60cc8b5d3c Merge pull request #2107 from stevvooe/fix-broken-build
registry/handlers: fix incorrect use of Digest.Verifiers
2016-12-15 20:00:26 -08:00
ning xie
e7140e5361 refactor doc comment
Signed-off-by: andy xie <andy.xning@gmail.com>
2016-12-16 11:42:06 +08:00
Stephen J Day
1040c824ba
registry/handlers: fix incorrect use of Digest.Verifiers
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 18:33:26 -08:00
Derek McGowan
b09de5e105 Merge pull request #2106 from stevvooe/digest-test-coverage
digest: better test coverage
2016-12-15 17:46:42 -08:00
Stephen J Day
ecb2ee3824
digest: better test coverage
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 17:33:33 -08:00
Derek McGowan
6a74c94bf8 Merge pull request #2103 from joaofnfernandes/fix-configuration.md
Format configuration.md with code fences to avoid render issues
2016-12-15 16:56:36 -08:00
Derek McGowan
729b8c5b91 Merge pull request #2105 from stevvooe/algorithm-own-file
digest: cleanup digester and verifier creation
2016-12-15 16:55:37 -08:00
Stephen J Day
9159833265
digest: remove error return from Digest.Verifier
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 16:43:59 -08:00
Derek McGowan
898c26326d Merge pull request #2104 from stevvooe/digest-parse
digest: remove stuttering ParseDigest function
2016-12-15 15:44:27 -08:00
Stephen J Day
e37baed88e
digest: cleanup digester and verifier creation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:17:21 -08:00
Stephen J Day
01dfa0fcb9
digest: use digest.Parse over ParseDigest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:07:42 -08:00
Stephen J Day
d5cc235c48
digest: remove stuttering ParseDigest function
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 14:58:54 -08:00
Joao Fernandes
a24f2a6d78 Format configuration.md with code fences to avoid render issues
Signed-off-by: Joao Fernandes <joao.fernandes@docker.com>
2016-12-15 14:35:24 -08:00
Derek McGowan
ad80b3d492 Merge pull request #2102 from dmcgowan/schema2-deps
[carry #1972] Update schema2 builder to take media type
2016-12-15 13:28:18 -08:00
Derek McGowan
8867e8fac3
Update schema2 builder to take media type
Modify manifest builder so it can be used to build
manifests with different configuration media types.
Rename config media type const to image config.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2016-12-14 16:17:20 -08:00
Derek McGowan
923c7763b0 Merge pull request #2072 from andyxning/fix_filechecker_in_health
fix filechecker in health with precondition check
2016-12-09 09:51:46 -08:00
Derek McGowan
844b92879f Merge pull request #2095 from dmcgowan/enforce-class-backward-compatibility
Enforce image is treated as empty class
2016-12-07 14:28:13 -08:00
Stephen Day
84aa95ebd0 Merge pull request #2097 from stevvooe/digest-from-string
digest: add FromString helper to Digest
2016-12-07 14:24:04 -08:00
Stephen J Day
85994fcfc0
digest: add FromString helper to Digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-07 13:46:47 -08:00
Derek McGowan
2a3510004f Merge pull request #2080 from dmcgowan/use-vndr
Replace godep with vndr
2016-12-07 10:26:39 -08:00
Derek McGowan
15dc1296af Merge pull request #2088 from ahmetalpbalkan/pr-upstream-azure-race-fix
azure: fix race condition in PutContent()
2016-12-06 14:07:53 -08:00
Derek McGowan
67095fbce3 Merge pull request #2035 from runcom/fix-foreign-urls-check
registry/handles/app: always append default urls regexps
2016-12-06 10:21:14 -08:00
ning xie
658cda621f fix filechecker in health with precondition check
Signed-off-by: andy xie <andy.xning@gmail.com>
2016-12-06 10:13:55 +08:00
Derek McGowan
426d0734d4
Enforce image is treated as empty class
Enforces backwards compatibility with older authorization servers
without requiring the client to know about the compatibility
requirements.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-12-05 15:10:29 -08:00
Antonio Murdaca
0fb25dd094
registry/handles/app: always append default urls regexps
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-12-05 20:28:51 +01:00
Derek McGowan
314144ac0b Merge pull request #2070 from kdada/master
filter listResponse.Contents in OSS driver.List()
2016-12-05 10:23:46 -08:00
Ahmet Alp Balkan
78d0660319
azure: fix race condition in PutContent()
See #2077 for background.

The PR #1438 which was not reviewed by azure folks basically introduced
a race condition around uploads to the same blob by multiple clients
concurrently as it used the "writer" type for PutContent(), introduced in #1438.
This does chunked upload of blobs using "AppendBlob" type, which was not atomic.

Usage of "writer" type and thus AppendBlobs on metadata files is currently not
concurrency-safe and generally, they are not the right type of blob for the job.

This patch fixes PutContent() to use the atomic upload operation that works
for uploads smaller than 64 MB and creates blobs with "BlockBlob" type. To be
backwards compatible, we query the type of the blob first and if it is not
a "BlockBlob" we delete the blob first before doing an atomic PUT. This
creates a small inconsistency/race window "only once". Once the blob is made
"BlockBlob", it is overwritten with a single PUT atomicallly next time.

Therefore, going forward, PutContent() will be producing BlockBlobs and it
will silently migrate the AppendBlobs introduced in #1438 to BlockBlobs with
this patch.

Tested with existing code side by side, both registries with and without this
patch work fine without breaking each other. So this should be good from a
backwards/forward compatiblity perspective, with a cost of doing an extra
HEAD checking the blob type.

Fixes #2077.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-30 12:40:43 -08:00
Derek McGowan
c599955707 Merge pull request #2087 from aaronlehmann/missing-digest-panic
schema1: Validate descriptors in AppendReference
2016-11-30 10:26:55 -08:00
Aaron Lehmann
ec5fa1f9d6 schema1: Validate descriptors in AppendReference
If the digest is malformed, it will cause a panic when building the
manifest.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-11-29 17:22:07 -08:00
Derek McGowan
bf27f260cd Merge pull request #2083 from mstanleyjones/fix_doubled_words
Fix doubled words
2016-11-28 13:03:35 -08:00
Misty Stanley-Jones
8e703afdc6 Fix doubled words
Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2016-11-28 11:29:11 -08:00
Derek McGowan
acad7c43a8
Enable dependency validation
Re-enable dependency validation using vndr instead of godep

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-23 15:45:04 -08:00
Derek McGowan
a685e3fc98
Replace godep with vndr
Vndr has a simpler configuration and allows pointing to forked
packages. Additionally other docker projects are now using
vndr making vendoring in distribution more consistent.

Updates letsencrypt to use fork.
No longer uses sub-vendored packages.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-23 15:07:06 -08:00
Derek McGowan
38fbd03266 Merge pull request #1840 from dmcgowan/support-1.12-integration-tests
Integration tests support for docker 1.12
2016-11-23 14:19:09 -08:00
Derek McGowan
8d096a4f42 Merge pull request #1969 from TrumanLing/master
api.md:fix disordered placement of section "Deleting a Layer"
2016-11-23 13:59:17 -08:00
Derek McGowan
f2620ca7c1
Bump call to use 1.12.3
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-23 13:33:19 -08:00
Derek McGowan
00a505bc42
Support docker 1.12
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-23 13:33:19 -08:00
Derek McGowan
8f9abbd27f Merge pull request #2079 from dmcgowan/update-swift-vendor
Update swift vendor for upstreamed change
2016-11-23 13:19:53 -08:00
Derek McGowan
e9684b126e
Update swift vendor for upstreamed change
Gets rid of deviation between hash and upstream.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-23 11:45:56 -08:00
Derek McGowan
a6bf3dd064 Merge pull request #2067 from dmcgowan/add-repo-class
Add support for repository class
2016-11-21 20:23:04 -08:00
Derek McGowan
e02278f22a
Update registry server to support repository class
Use whitelist of allowed repository classes to enforce.
By default all repository classes are allowed.

Add authorized resources to context after authorization.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-21 16:36:36 -08:00
Derek McGowan
61e65ecd9d
Update contrib token server to support repository class
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-21 13:32:12 -08:00
Derek McGowan
01509db714
Add class to repository scope
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-21 13:32:12 -08:00
Derek McGowan
e10634f8c3
Update scope specification for resource class
Update grammar to support a resource class. Add
example for plugin repository class.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-21 13:32:12 -08:00
Kira
4accc8f2be filter listResponse.Contents in driver.List()
Signed-off-by: Kira <me@imkira.com>
2016-11-17 10:38:56 +08:00
Derek McGowan
7694c31658 Merge pull request #2063 from dmcgowan/fix-golint-error-newline
Remove newlines from end of error strings
2016-11-14 15:22:52 -08:00
Derek McGowan
8b71d39c34
Remove newlines from end of error strings
Golint now checks for new lines at the end of go error strings,
remove these unneeded new lines.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-14 14:03:01 -08:00
Derek McGowan
d22e09a668 Merge pull request #2062 from nishanttotla/with-default-tag
Adding utility functions - IsNameOnly and EnsureTagged
2016-11-11 15:34:52 -08:00
Nishant Totla
2caeb6192a
Adding utility functions - IsNameOnly and WithDefaultTag
Signed-off-by: Nishant Totla <nishanttotla@gmail.com>
2016-11-11 15:17:16 -08:00
Richard Scothern
e249b61e90 Merge pull request #2059 from RichardScothern/2.6-release-notes
Release notes for v2.6.0-rc1
2016-11-11 12:03:20 -08:00
Richard Scothern
ff9533dd6f Release notes for v2.6.0-rc1
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-11-11 11:54:35 -08:00
Richard Scothern
cb98aa744b Merge pull request #2060 from RichardScothern/release-prep
Release preperation: update AUTHORS and version file
2016-11-11 10:28:15 -08:00
Richard Scothern
cb2c78ce9b Release preperation: update AUTHORS and version file
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-11-11 09:23:05 -08:00
Derek McGowan
9cc7c94e4e Merge pull request #2052 from RichardScothern/fix-links
Fix storage driver links
2016-11-11 09:05:44 -08:00
Richard Scothern
6e62b39842 Merge pull request #2036 from pyr/fix/sort-v2-headers
v2 signer: correctly sort headers
2016-11-10 15:31:24 -08:00
Derek McGowan
8016d2d890 Merge pull request #2053 from dmcgowan/add-trim-name
Add trim name method helper to create name only references
2016-11-10 15:03:34 -08:00
Derek McGowan
31a91e419c
Add trim name method helper to create name only references
Add a simple helper to create name only references without having
to first get the name as a string and reparse. This method does
not fail so ignoring or checking error to simply trim reference
is not required.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-10 14:46:04 -08:00
Richard Scothern
4d65dd513e Merge pull request #2038 from spacexnice/master
fix: oss driver would get connection reset by peer when upload large image layer.
2016-11-10 14:44:32 -08:00
Richard Scothern
09b2d06d5d Fix storage driver links
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-11-10 14:28:28 -08:00
Stephen Day
4ad885a672 Merge pull request #2051 from vieux/update_plugin_media_typev1
switch to v1 and remove .image
2016-11-10 11:49:48 -08:00
Victor Vieux
b82a2efc33 switch to v1 and remove .image
Signed-off-by: Victor Vieux <vieux@docker.com>
2016-11-10 11:39:15 -08:00
Derek McGowan
ac0493db12 Merge pull request #2040 from RichardScothern/release-checklist
Document registry release process
2016-11-10 10:33:30 -08:00
Derek McGowan
717134d7c2 Merge pull request #2048 from dmcgowan/use-api-errors
Update oauth errors to use api errors
2016-11-09 20:27:40 -08:00
Derek McGowan
051801f1d0
Update oauth errors to use api errors
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-09 19:29:18 -08:00
Stephen Day
26c9a77535 Merge pull request #2039 from vdemeester/add-match-support-to-reference
Add a new Match method to the reference package
2016-11-09 18:49:21 -08:00
Stephen Day
e04e6ddd2c Merge pull request #2047 from dmcgowan/fix-authorization-error
Add OAuth error for client
2016-11-09 18:03:38 -08:00
Derek McGowan
16396a7a80
Add OAuth error for client
Allow clients to handle errors being set in the WWW-Authenticate
rather than in the body. The WWW-Authenticate errors give a
more precise error describing what is needed to authorize
with the server.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-09 15:27:40 -08:00
Derek McGowan
a1a73884f9
Refactor authorization challenges to its own package
Split challenges into its own package. Avoids possible
import cycle with challenges from client.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-09 15:27:40 -08:00
Vincent Demeester
353e3a4c9d
Add a new Match method to the reference package
The Match method allows to see if a reference matches a specified
patterns.

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2016-11-09 22:04:52 +01:00
Derek McGowan
02f4195788 Merge pull request #2044 from aaronlehmann/preserve-tag-and-digest
digest: Preserve tag and digest in With* functions
2016-11-08 15:10:22 -08:00
Aaron Lehmann
bc1e7aa603 digest: Preserve tag and digest in With* functions
When WithDigest is called on a reference that has a tag, it should
preserve the tag.

When WithTag is called on a reference that has digest, it should
preserve the digest.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-11-08 14:47:33 -08:00
yaoyao.xyy
a4a227e351 oss native large file copy consume too much time which will eventually lead to client timeout because of no data transmit throughout native copy. change maxCopySize to 128MB, ensure only sm all medium size file use oss native copy to avoid connection reset by peer. And fix Move function with CopyLargeFileInParallel to optimize oss upload copy
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-11-08 12:14:13 +08:00
Richard Scothern
1e501d0bd7 Document registry release process
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-11-07 12:48:55 -08:00
Derek McGowan
a2611c7520 Merge pull request #2027 from ahmetalpbalkan/pr-azure-memleak2
Update vendored azure-sdk-for-go
2016-11-04 10:08:40 -07:00
Ahmet Alp Balkan
2ab25288a2
Update vendored azure-sdk-for-go
Updating to a recent version of Azure Storage SDK to be
able to patch some memory leaks through configurable HTTP client
changes which were made possible by recent patches to it.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-03 13:24:57 -07:00
Derek McGowan
fbe6e8d212 Merge pull request #2008 from miminar/honor-x-forwarded-port
Honor X-Forwarded-Port header
2016-11-02 12:56:17 -07:00
Derek McGowan
6edf9c5070 Merge pull request #2032 from yuexiao-wang/cleanup-makefile
Add targets for Makefile
2016-11-02 10:40:47 -07:00
Pierre-Yves Ritschard
f1cf7de788 fixup! v2 signer: correctly sort headers
Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:07:02 +01:00
Pierre-Yves Ritschard
775cc6d632 v2 signer: correctly sort headers
The current code determines the header order for the
"string-to-sign" payload by sorting on the concatenation
of headers and values, whereas it should only happen on the
key.

During multipart uploads, since `x-amz-copy-source-range` and
`x-amz-copy-source` headers are present, V2 signatures fail to
validate since header order is swapped.

This patch reverts to the expected behavior.

Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:01:34 +01:00
Michal Minář
1b43e1e30d
Honor X-Forwarded-Port and Forwarded headers
Prefer non-standard headers like X-Forwarded-Proto, X-Forwarded-Host and
X-Forwarded-Port over the standard Forwarded header to maintain
backwards compatibility.

If a port is not specified neither in Host nor in forwarded headers but
it is specified just with X-Forwarded-Port, use its value in base urls
for redirects.

Forwarded header is defined in rfc7239.

X-Forwarded-Port is a non-standard header. Here's a description copied
from "HTTP Headers and Elastic Load Balancing" of AWS ELB docs:

> The X-Forwarded-Port request header helps you identify the port that
> an HTTP or HTTPS load balancer uses to connect to the client.

Signed-off-by: Michal Minář <miminar@redhat.com>
2016-11-02 16:49:42 +01:00
Richard Scothern
3d22ab0547 Merge pull request #2030 from ahmetalpbalkan/pr-go17
Bump to go1.7
2016-11-01 15:42:12 -07:00
yuexiao-wang
8530d65c9b Add targets for Makefile
Signed-off-by: yuexiao-wang <wang.yuexiao@zte.com.cn>
2016-11-01 17:48:25 +08:00
Ahmet Alp Balkan
af0d4f72fa
Bump to go1.7
Ref: #2027

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-31 13:05:03 -07:00
Richard Scothern
93a48e361c Merge pull request #2031 from ahmetalpbalkan/swift-fix
driver/swift: Fix go vet warning
2016-10-31 12:55:38 -07:00
Ahmet Alp Balkan
a994f35657
driver/swift: Fix go vet warning
Driver was passing connections by copying. Storing
`swift.Connection` as pointer to fix the warnings.

Ref: #2030.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-31 11:41:53 -07:00
Derek McGowan
994d1a56ff Merge pull request #2025 from ahmetalpbalkan/pr-azure-memleak1
storagedriver/azure: close leaking response body
2016-10-28 16:58:32 -07:00
Ahmet Alp Balkan
6d2a0bafcd
storagedriver/azure: close leaking response body
In GetContent() we read the bytes from a blob but do not close
the underlying response body.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-28 15:13:22 -07:00
Derek McGowan
62e88f0fe7 Merge pull request #2019 from joaofnfernandes/remove-jenkinsfile
Remove Jenkinsfile used for docs CI
2016-10-27 13:15:10 -07:00
Joao Fernandes
a80821295e Remove Jenkinsfile used for docs CI
Signed-off-by: Joao Fernandes <joao.fernandes@docker.com>
2016-10-26 11:55:20 -07:00
Derek McGowan
4fb6b181cd Merge pull request #2016 from spacexnice/master
optimize oss copy action when push upload finished.
2016-10-26 11:15:55 -07:00
yaoyao.xyy
f394e82d2b oss data copy optimize after upload finished
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-10-26 09:37:58 +08:00
Stephen Day
fbb70dc3a1 Merge pull request #2015 from runcom/accept-identity-get-layer
registry/client: set Accept: identity header when getting layers
2016-10-25 17:46:27 -07:00
Antonio Murdaca
16f086a0ec
registry/client: set Accept: identity header when getting layers
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-10-25 10:33:11 +02:00
LingFaKe
0c42a73b70 api.md/api.md.tmpl:format wrong style of section "Deleting a Layer"
Signed-off-by: Ling FaKe <lingfake@huawei.com>
2016-10-19 09:07:13 +08:00
Stephen Day
8234784a1a Merge pull request #1999 from stevvooe/fold-target-references
manifest: references should cover all children
2016-10-18 11:59:44 -07:00
Stephen J Day
c9aaff00f8
manifest: references should cover all children
To allow generic manifest walking, we define an interface method of
`References` that returns the referenced items in the manifest. The
current implementation does not return the config target from schema2,
making this useless for most applications.

The garbage collector has been modified to show the utility of this
correctly formed `References` method. We may be able to make more
generic traversal methods with this, as well.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-10-18 11:43:33 -07:00
Richard Scothern
e3aabfb47e Merge pull request #2009 from difro/fix-broken-links
Fix broken doc links
2016-10-18 11:34:24 -07:00
Richard Scothern
96f63cbe7f Merge pull request #2010 from mbentley/add-new-s3-region
Added new us-east-2 region for S3
2016-10-18 10:25:30 -07:00
Richard Scothern
5bd2cfaddd Merge pull request #2000 from jessfraz/1950-comment-on-non-dryrun-as-well
storage/garbagecollect: add emit comments for when dryRun is false
2016-10-18 10:00:04 -07:00
Matt Bentley
3857f50825
Added new us-east-2 region for S3
Signed-off-by: Matt Bentley <mbentley@mbentley.net>
2016-10-18 12:30:34 -04:00
Jihoon Chung
89e41acddf Fix broken doc links
Signed-off-by: Jihoon Chung <jihoon@gmail.com>
2016-10-18 21:31:08 +09:00
Richard Scothern
d0cdc4802b Merge pull request #2002 from lnr0626/1996-instance-roles-with-regionendpoint
Allow using ec2 roles when specifying region endpoint
2016-10-17 13:50:02 -07:00
Richard Scothern
b6e8a3f441 Merge pull request #2005 from RichardScothern/nwt-fix-s3-goamz-delete
Nwt fix s3 goamz & Aliyun OSS delete
2016-10-17 13:09:48 -07:00
Derek McGowan
4dd19db95b Merge pull request #2007 from qar/fix/docs
Fix documentation typo
2016-10-17 11:40:35 -07:00
Richard Scothern
a621a86cb4 Fix aliyun OSS Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-17 09:43:15 -07:00
Noah Treuhaft
12e73f01d2 Fix s3-goamz Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-10-17 09:43:15 -07:00
Derek McGowan
717ac0337f Merge pull request #2006 from RichardScothern/lint
Satisfy the latest go lint rules
2016-10-17 09:38:33 -07:00
Richard Scothern
8be8b60ce5 Merge pull request #2004 from mstanleyjones/toml_to_yaml
Convert Markdown frontmatter to YAML
2016-10-17 09:35:40 -07:00
Qiao Anran
3d703a0b82
Fix documentation typo
Signed-off-by: Qiao Anran <qiaoanran@gmail.com>
2016-10-17 14:58:29 +08:00
Richard Scothern
bd91493195 Satisfy the latest go lint rules
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-14 17:03:08 -07:00
Misty Stanley-Jones
f180e9a934 Convert Markdown frontmatter to YAML
Some frontmatter such as the weights, menu stuff, etc is no longer used
'draft=true' becomes 'published: false'

Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2016-10-14 15:59:19 -07:00
Misty Stanley-Jones
c372264f17 Merge pull request #1985 from johndmulhausen/master
Remove old documentation source, add README on migration
2016-10-14 13:13:32 -07:00
Derek McGowan
04c8db562d Merge pull request #2001 from aaronlehmann/simplify-splithostname
reference: Simplify SplitHostname
2016-10-13 15:37:33 -07:00
Lloyd Ramey
c8ea7840d3 Allow using ec2 roles when specifying region endpoint
Signed-off-by: Lloyd Ramey <lnr0626@gmail.com>
2016-10-13 18:07:37 -04:00
Aaron Lehmann
ab6ae11bc1 reference: Simplify SplitHostname
This nil check is not necessary.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-10-13 14:51:37 -07:00
Misty Stanley-Jones
3ac8dea66a Add back configuration.md and add clarification to README.md
Signed-off-by: Misty Stanley-Jones <misty@docker.com>
2016-10-13 11:49:49 -07:00
John Mulhausen
da8bcbb302 Remove old documentation source, add README on migration
Signed-off-by: John Mulhausen <john@docker.com>
2016-10-13 11:49:17 -07:00
Jess Frazelle
809774c33f
storage/garbagecollect: add emit commits for when dryRun is false
Signed-off-by: Jess Frazelle <acidburn@google.com>
2016-10-13 08:04:19 -07:00
Richard Scothern
252cc27ab1 Merge pull request #1988 from nwt/fix-s3-delete
Fix S3 Delete method's notion of subpaths
2016-10-11 11:14:29 -07:00
Richard Scothern
8261bd94f0 Merge pull request #1993 from dmcgowan/cleanup-makefile
Cleanup makefile
2016-10-11 10:46:09 -07:00
Derek McGowan
d9dc966209
Cleanup makefile
Use find instead of wildcard to get go files since wildcard does not
do recursive lookups via `**`, missing most go files.
Do lazy evaluations of variables which are not needed immediately.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-10-10 17:30:47 -07:00
Richard Scothern
1921dde3f1 Merge pull request #1942 from sergeyfd/auth-proxy
Support for custom authentication URL in proxying registry
2016-10-10 10:40:51 -07:00
Richard Scothern
c5b2ad51c1 Merge pull request #1991 from HuKeping/fix-unit2
Fix unit test
2016-10-10 10:38:56 -07:00
HuKeping
b984f91b32 Fix unit test
Signed-off-by: Hu Keping <hukeping@huawei.com>
2016-10-10 11:37:30 +08:00
Noah Treuhaft
76226c61a9 Fix S3 Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-10-06 11:21:55 -07:00
Richard Scothern
b89a594355 Merge pull request #1987 from dmcgowan/fix-lint
Update to fix lint errors
2016-10-06 10:06:54 -07:00
Derek McGowan
d35d94dcec
Update to fix lint errors
Context should use type values instead of strings.
Updated direct calls to WithValue, but still other uses of string keys.
Update Acl to ACL in s3 driver.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-10-05 17:47:12 -07:00
Richard Scothern
99cb7c0946 Merge pull request #1973 from dmcgowan/clarify-oauth-docs
Add note about implementation of oauth2
2016-09-26 16:28:29 -07:00
Derek McGowan
f193270c89
Add note about implementation of oauth2
Reading the oauth2 token documentation is misleading as it makes
no mention of it being a newer feature which may not be supported
by the token server. Add a note mentioning if it is not supported
to refer to the token documentation for getting a token.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-09-26 15:39:00 -07:00
Richard Scothern
431cfa3179 Merge pull request #1966 from nwt/fix-redis-tests
Fix connection pool exhaustion in Redis tests
2016-09-19 16:27:56 -07:00
Richard Scothern
8cb75f692f Merge pull request #1868 from stanhu/fix-missing-credentials-with-default-port
Fix missing auth headers with PATCH HTTP request when pushing to default port
2016-09-19 15:55:12 -07:00
Noah Treuhaft
cafeae4ecd Fix connection pool exhaustion in Redis tests
The Redis tests were failing with a "connection pool exhausted" error
from Redigo. Closing the connection used for FLUSHDB fixes the problem.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-19 15:30:32 -07:00
Stan Hu
6d952c92cf Rename CanonicalAddr -> canonicalAddr
Update comment for hasPort

Signed-off-by: Stan Hu <stanhu@gmail.com>
2016-09-19 14:16:02 -07:00
Richard Scothern
cfad4321c1 Merge pull request #1957 from nwt/notification-filtering
Add notification filtering by target media type
2016-09-19 10:41:03 -07:00
Noah Treuhaft
ad6bb66faf Add notification filtering by target media type
The Hub registry generates a large volume of notifications, many of
which are uninteresting based on target media type.  Discarding them
within the notification endpoint consumes considerable resources that
could be saved by discarding them within the registry.  To that end,
this change adds registry configuration options to restrict the
notifications sent to an endpoint based on target media type.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-16 12:01:03 -07:00
Richard Scothern
b6e0cfbdaa Merge pull request #1959 from nwt/disable-access-logging
Add configuration option to disable access logging
2016-09-15 09:45:56 -07:00
Noah Treuhaft
4034ff65f0 Add configuration option to disable access logging
Access logging is great.  Access logging you can turn off is even
better.  This change adds a configuration option for that.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-14 14:33:30 -07:00
Richard Scothern
835b04c004 Merge pull request #1958 from spacexnice/master
remoteURL parameter error cause  challenge cache not working which eventually lead to client manifest request timeout
2016-09-14 12:33:18 -07:00
Serge Dubrouski
81c5870c86 Support for custom authentication URL in proxying registry
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-09-13 17:07:20 -06:00
Derek McGowan
17fb0bb6b3 Merge pull request #1934 from jheiss/token_ssl_pem_type
Check PEM block type when reading token cert file
2016-09-13 09:45:06 -07:00
spacexnice
3c4b551a7c remoteURL parameter error cause challenge cache not working,critical area consume to much time, which eventually lead to client manifest request timeout(90s) in massive(which i test 80 concurrent) concurrent manifest request situation.
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-09-13 09:58:12 +08:00
Richard Scothern
cb744efe8b Merge pull request #1943 from sergeyfd/listener
manifestServiceListener.Get to pass down options parameter
2016-09-12 11:15:44 -07:00
Richard Scothern
405b53c019 Merge pull request #1900 from thaJeztah/improve-invalid-reference-errormessage
Improve "invalid reference" error message
2016-09-12 10:06:21 -07:00
Sebastiaan van Stijn
744ae974a5
Improve "invalid reference" error message
Use the same error message as is used in
docker/reference/reference.go to provide
slightly more information about the error.

This change tests if the reference passes
after lowercasing its characters, to determine
if the reference was invalid due to
it containing uppercase characters.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2016-09-12 15:13:14 +02:00
Stephen Day
5e8d18f615 Merge pull request #1945 from stevvooe/better-report-on-invalid-secret
handlers: provide better log message on mismatched secret
2016-09-07 12:55:48 -07:00
Derek McGowan
4df7c18266 Merge pull request #1951 from nwt/downgrade-token-auth-jwt-logging
Downgrade token auth JWT logging from error to info
2016-09-07 11:52:47 -07:00
Noah Treuhaft
91f268e5a5 Downgrade token auth JWT logging from error to info
The token auth package logs JWT validation and verification failures at
the `error` level.  But from the server's perspective, these aren't
errors.  They're the expected response to bad input.  Logging them at
the `info` level better reflects that distinction.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-07 10:45:06 -07:00
Derek McGowan
ed0b3aebb3 Merge pull request #1935 from vieux/update+plugin+media
update plugin MediaType
2016-09-06 09:51:20 -07:00
Derek McGowan
cbdca10ab0 Merge pull request #1949 from mikebrow/building-link
fixes link to building.md
2016-09-06 09:42:15 -07:00
Mike Brown
d1383450e9 fixes link to building.md
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2016-09-06 11:16:03 -05:00
Victor Vieux
de9aa8466f upate plugin MediaType
Signed-off-by: Victor Vieux <vieux@docker.com>
2016-09-06 03:34:24 -07:00
Stephen J Day
668b0a5f40
handlers: provide better log message on mismatched secret
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-09-02 15:24:35 -07:00
Serge Dubrouski
a1a2757fb0 manifestServiceListener.Get to pass down options parameter
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-09-01 18:50:56 -06:00
Richard Scothern
49c1a62cb2 Merge pull request #1940 from dmcgowan/lets-encrypt-port-note
Add note about required let's encrypt port
2016-09-01 16:13:18 -07:00
Derek McGowan
279c02a3ce
Add note about required let's encrypt port
Let's Encrypt uses tls-sni to validate the certificate
on the standard https port 443. If the registry is
outwardly listening on a different port Let's Encrypt
will not issue a certificate.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-09-01 15:11:44 -07:00
Stan Hu
462bb55c3f Fix missing auth headers with PATCH HTTP request when pushing to default port
If a user specifies `mydomain.com:443` in the `Host` configuration, the
PATCH request for the layer upload will fail because the challenge does not
appear to be in the map. To fix this, we normalize the map keys to always
use the Host:Port combination.

Closes https://github.com/docker/docker/issues/18469

Signed-off-by: Stan Hu <stanhu@gmail.com>
2016-09-01 14:46:31 -07:00
Fabio Berchtold
7dcac52f18 Add v2 signature signing to S3 storage driver (#1800)
* Add v2 signature signing to S3 storage driver

Closes #1796
Closes #1606

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* use Logrus for debug logging

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* use 'date' instead of 'x-amz-date' in request header

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* only allow v4 signature signing against AWS S3

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>
2016-09-01 13:52:40 -07:00
Jason Heiss
d04481e388 Check PEM block type when reading token cert file
closes #1909

Signed-off-by: Jason Heiss <jheiss@twosigma.com>
2016-09-01 16:48:55 -04:00
Richard Scothern
49da29ee46 Merge pull request #1925 from dmcgowan/reenable-race-detector
Re-enable race detector in circleci
2016-09-01 13:36:12 -07:00
Derek McGowan
41f383fb9a Merge pull request #1928 from ghostplant/master
Dynamically Parsing the Latest HTPassword File
2016-08-30 10:29:35 -07:00
Aaron Lehmann
2f16e6e7b3 Merge pull request #1932 from stevvooe/manifest-urls-fix
spec/manifest: clarify relationship between urls and foreign layers
2016-08-29 18:45:16 -07:00
Stephen J Day
6bcdb38b92
spec/manifest: clarify relationship between urls and foreign layers
Previously, the specificiation incorrectly bound the fates of `urls` and
foreign layers. These are complementary but unrelated features, in that
the `urls` field may be populated for layers that aren't foreign. The
type of the layer only dictates the push behavior of the layer, rather
than involving where it came from.

For example, one may pull a foreign layer from a registry, but they may
not push it back to another registry. Conversely, a layer that has no
restrictions on push/pull behavior may be fetched via `urls` entries.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-29 18:34:55 -07:00
Derek McGowan
205e606a4c Update proxy scheduler test to account for race
Running with the race detector may cause some parts
of the code to run slower causing a race in the scheduler
ordering.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-29 14:27:32 -07:00
Derek McGowan
6cd63c8bc0 Disable registry handlers race tests
Registry handlers tests currently takes up too
many resources for the race test to handle on circle ci.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-29 13:41:53 -07:00
Derek McGowan
b3e276ff93 Add hash map locking to proxy tests
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-29 13:39:24 -07:00
Derek McGowan
a50ce1ab93 Add locking to repository access in memory cache
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 18:10:42 -07:00
Derek McGowan
dabdc5e52b Fix access race in proxy scheduler
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 17:40:21 -07:00
Derek McGowan
b1b100cf01 Merge pull request #1923 from spacexnice/master
fix simpleAuthChallge concurrent problem
2016-08-26 17:31:09 -07:00
Derek McGowan
798b7331ca Add shutdown to test environment
Ensures test http server is fully closed before moving on with test.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 16:23:07 -07:00
Derek McGowan
38ee36eee4 Disable upload purging during test
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 16:23:07 -07:00
a-palchikov
82609180a1 tag service: properly handle error responses on HEAD requests by (#1918)
* tag service: properly handle error responses on HEAD requests by
re-issuing requests as GET for proper error details.

Fixes #1911.

Signed-off-by: dmitri <deemok@gmail.com>

* Simplify handling of failing HEAD requests in TagService and
make a GET request for cases:
  - if the server does not handle HEAD
  - if the response was an error to get error details

Signed-off-by: dmitri <deemok@gmail.com>

* Add a missing http.Response.Body.Close call for the GET request.

Signed-off-by: dmitri <deemok@gmail.com>
2016-08-26 14:35:04 -07:00
cuiwei13
169ff1d098 Dynamically Parsing the Latest HTPassword File
To parse the latest account list dynamically instead of restarting the distribution service frequently.

Signed-off-by: CUI Wei <ghostplant@qq.com>
2016-08-27 04:28:33 +08:00
spacexnice
68243306fb correct onerror behavior
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-26 09:41:54 +08:00
spacexnice
22cf89c02b gofmt -s do code format
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-25 07:34:36 +08:00
Matthew Green
dea554fc7c Swift driver now bulk deletes in chunks specified by the server (#1915)
Swift driver now bulk deletes in chunks specified by the server

Signed-off-by: Matthew Green <matthew.green@uk.ibm.com>
2016-08-24 10:09:25 -07:00
spacexnice
ca2b78dc0a adjust for loop
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-24 09:31:27 +08:00
Derek McGowan
6f43d2d4f0 Re-enable race detector in circle ci
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-23 17:31:40 -07:00
Richard Scothern
c24e10f70a Merge pull request #1924 from bowlofeggs/docs-comma_fix
Fix an erroneous comma in documentation JSON.
2016-08-23 15:26:26 -07:00
Randy Barlow
63b2e74b46
Fix an erroneous comma in documentation JSON.
Signed-off-by: Randy Barlow <randy@electronsweatshop.com>
2016-08-23 13:39:24 -04:00
spacexnice
44b8157572 fix simpleAuthChallge concurrent problem
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-23 22:09:18 +08:00
Richard Scothern
4107cdb633 Merge pull request #1919 from SvenDowideit/add-jenkinsfile
Add docs checking Jenkinsfile
2016-08-22 11:07:35 -07:00
Sven Dowideit
98f81b154f Add docs checking Jenkinsfile
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-08-22 12:19:30 +10:00
Richard Scothern
fd4dd8dd16 Merge pull request #1912 from bbodenmiller/patch-1
improve command formatting
2016-08-19 11:35:14 -07:00
Ben Bodenmiller
1f248a80a6 improve command formatting
Signed-off-by: Ben Bodenmiller <bbodenmiller@hotmail.com>
2016-08-19 02:33:02 -07:00
Richard Scothern
010e063270 Merge pull request #1906 from nwt/s3-multipart-copy
Use multipart upload API in S3 Move method
2016-08-17 15:22:32 -07:00
Derek McGowan
77b9d2997a Merge pull request #1901 from anusha-ragunathan/add-plugin-mediatype
Add plugin mediatype to distribution manifest.
2016-08-16 13:53:36 -07:00
Richard Scothern
c4297ef9da Merge pull request #1839 from adamvduke/adamvduke/allow-http2-registry-clients
Allow registry clients to connect via http2
2016-08-16 11:48:06 -07:00
Noah Treuhaft
63468ef4a8 Use multipart upload API in S3 Move method
This change to the S3 Move method uses S3's multipart upload API to copy
objects whose size exceeds a threshold.  Parts are copied concurrently.
The level of concurrency, part size, and threshold are all configurable
with reasonable defaults.

Using the multipart upload API has two benefits.

* The S3 Move method can now handle objects over 5 GB, fixing #886.

* Moving most objects, and espectially large ones, is faster.  For
  example, moving a 1 GB object averaged 30 seconds but now averages 10.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-08-16 10:53:24 -07:00
Richard Scothern
c810308d1b Merge pull request #1902 from majewsky/swift/support-tenant-domain-parameter
[swift] support different user-domain and tenant-domain
2016-08-15 10:45:12 -07:00
Stefan Majewsky
a7c6bfd59f [swift] support different user-domain and tenant-domain
This is already supported by ncw/swift, so we just need to pass the
parameters from the storage driver.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-08-15 11:21:42 +02:00
Adam Duke
ac009c86f1 Allow registry clients to connect via http2
Http2 will be enabled by default and can be disabled with a configuration option.

Signed-off-by: Adam Duke <adam.v.duke@gmail.com>
2016-08-13 22:07:42 -04:00
Anusha Ragunathan
c202010f8a Add plugin mediatype to distribution manifest.
This is required for github.com/docker/docker/pull/25582

Signed-off-by: Anusha Ragunathan <anusha@docker.com>
2016-08-11 19:11:11 -07:00
Richard Scothern
c9fd26e9ef Merge pull request #1895 from hinshun/improve-enumerate
Improve catalog enumerate runtime by an order of magnitude
2016-08-11 10:53:36 -07:00
Richard Scothern
baca174469 Merge pull request #1892 from stevvooe/use-math-rand-reader
testutil, storage: use math/rand.Read where possible
2016-08-11 10:04:36 -07:00
Stephen J Day
040db51795
testutil, storage: use math/rand.Read where possible
Use the much faster math/rand.Read function where cryptographic
guarantees are not required. The unit test suite should speed up a
little bit but we've already optimized around this, so it may not
matter.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-10 14:26:12 -07:00
Edgar Lee
2f81b3b058 Improve catalog enumerate runtime by an order of magnitude
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-09 17:42:26 -07:00
Richard Scothern
9000745401 Merge pull request #1888 from stevvooe/more-efficient-path-comparison
registry/storage: more efficient path compare in catalog
2016-08-09 13:45:07 -07:00
Stephen Day
c270829c6e Merge pull request #1887 from hinshun/fix-catalog-enumerate
Stop ErrFinishedWalk from escaping from Repositories walk
2016-08-09 13:44:20 -07:00
Derek McGowan
2f356d69c8 Merge pull request #1894 from RichardScothern/circle
Remove godep check from build and increase test timout
2016-08-09 12:06:30 -07:00
Richard Scothern
a8ef842a9b Remove godep check from build and increase test timout
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-08-09 11:50:44 -07:00
Stephen J Day
308faf00f1
catalog: add benchmarks for overridden path comparison
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 17:09:49 -07:00
Stephen J Day
bba5a0d05c
registry/storage: more efficient path compare in catalog
Previous component-wise path comparison is recursive and generates a
large amount of garbage. This more efficient version simply replaces the
path comparison with the zero-value to sort before everything. We do
this by replacing the byte-wise comparison that swaps a single character
inline for the separator comparison, such that separators sort first.

The resulting implementation provides component-wise path comparison
with no cost incurred for allocation or stack frame.

Direction of the comparison is also reversed to match Go style.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 15:55:26 -07:00
Edgar Lee
379312c148 Stop ErrFinishedWalk from escaping from Repositories walk
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-05 17:21:48 -07:00
Richard Scothern
bfa0a9c097 Merge pull request #1877 from spacexnice/master
function copyContent() cause socket leak.
2016-08-02 09:15:23 -07:00
yaoyao.xyy
64c8dd3252 when deploy registry as a pull through cache ,function copeContent() may cause a socket leak when docker user canceled its pull operation.
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-08-02 11:10:10 +08:00
Michal Minář
7365003236 Provide stat descriptor for Create method during cross-repo mount (#1857)
* Allow precomputed stats on cross-mounted blobs

Signed-off-by: Michal Minář <miminar@redhat.com>

* Extended cross-repo mount tests

Signed-off-by: Michal Minář <miminar@redhat.com>
2016-08-01 19:07:11 -07:00
Frank Chen
87917f3052 Add 'objectAcl' Option to the S3 Storage Backend (#1867)
* Add Object ACL Support to the S3 Storage Backend

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Made changes per @RichardScothern's comments

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Fix Typos

Signed-off-by: Frank Chen <frankchn@gmail.com>
2016-07-27 12:26:57 -07:00
Richard Scothern
4abae2ae56 Merge pull request #1861 from adamvduke/adamvduke/reorder-dockerfile-for-layer-caching-optimization
reorder Dockerfile steps for better layer caching
2016-07-22 11:07:24 -07:00
Adam Duke
41c87559c6 reorder Dockerfile steps for better layer caching
Running `apk add` before copying source into the image takes better
adavantage of layer caching when developing and regularly building the
image. This avoids source code changes invalidating the `apk add` layer
and causing that step to run on every image build.

Signed-off-by: Adam Duke <adam.v.duke@gmail.com>
2016-07-22 13:51:31 -04:00
Richard Scothern
c2a201dabf Merge pull request #1860 from Seb-Solon/master
Fix #1854
2016-07-22 10:04:01 -07:00
Richard Scothern
08d0d3756c Merge pull request #1862 from RichardScothern/fix-build
Fix the build.
2016-07-22 09:38:27 -07:00
Richard Scothern
2287f1c42f Fix the build.
Pass the manifestURL directly into the schema2 manifest handler instead of
accessing through the repository as it has since the reference is now an
interface.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-21 17:16:47 -07:00
Richard Scothern
2b72dd3927 Merge pull request #1829 from nwt/foreign-layer-host-whitelist
Add a foreign layer URL host whitelist
2016-07-21 16:02:20 -07:00
Aaron Lehmann
f56b143dc8 Merge pull request #1858 from RichardScothern/decouple
Decouple storage components by redefining dependencies as interfaces
2016-07-21 11:12:05 -07:00
Sebastien Coavoux
a9c9daf0f2 Fix: Compare path properly when list repository in catalog. #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:46:00 -04:00
Sebastien Coavoux
3290e2bef5 Enh: Test - Edit catalog_test to reproduce #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:23:19 -04:00
Richard Scothern
2a5fcacdf0 Decouple storage components by redefining dependencies as interfaces instead of concrete types
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-20 15:09:11 -07:00
Richard Scothern
857d0f15c0 Merge pull request #1757 from miminar/cross-mount-middleware-repository
Allow to use repo middleware wrapper during cross-repo mounts
2016-07-19 17:43:10 -07:00
Richard Scothern
dd355e95af Merge pull request #1686 from marcusmartins/nbf-leeway
Add leeway to JWT nbf checking
2016-07-19 17:41:57 -07:00
Noah Treuhaft
61e5803b56 Add control over validation of URLs in pushed manifests
Until we have some experience hosting foreign layer manifests, the Hub
operators wish to limit foreign layers on Hub. To that end, this change
adds registry configuration options to restrict the URLs that may appear
in pushed manifests.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-07-19 14:38:42 -07:00
Richard Scothern
7b5958798e Merge pull request #1852 from jfindley/load-middleware-redirect
Load the middleware/redirect driver
2016-07-19 09:10:25 -07:00
Richard Scothern
96a1aec65d Merge pull request #1850 from runcom/typo
manifest: versioned: fix typo
2016-07-19 09:08:42 -07:00
Richard Scothern
f27ceb7ab5 Merge pull request #1710 from majewsky/swift/wait-for-dlo-segments-during-read
[Swift] add simple heuristic to detect incomplete DLOs during read ops
2016-07-19 09:07:44 -07:00
James Findley
6b98c68d12 Load the middleware/redirect driver
Signed-off-by: James Findley <jfindley@fastmail.com>
2016-07-19 15:37:10 +01:00
Antonio Murdaca
2ea0a41872
manifest: versioned: fix typo
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-07-19 13:54:41 +02:00
Marcus Martins
db1bf93098
Add leeway to JWT nbf and exp checking
Adds a constant leeway (60 seconds) to the nbf and exp claim check to
account for clock skew between the registry servers and the
authentication server that generated the JWT.

The leeway of 60 seconds is a bit arbitrary but based on the RFC
recommendation and hub.docker.com logs/metrics where we don't see
drifts of more than a second on our servers running ntpd.

I didn't attempt to make the leeway configurable as it would add extra
complexity to the PR and I am not sure how Distribution prefer to
handle runtime flags like that.

Also, I am simplifying the exp and nbf check for readability as the
previous `NOT (A AND B)` with cmp operators was not very friendly.

Ref:
https://tools.ietf.org/html/rfc7519#section-4.1.5

Signed-off-by: Marcus Martins <marcus@docker.com>
2016-07-18 17:47:30 -07:00
Richard Scothern
aa3de8fd4f Merge pull request #1847 from RichardScothern/go-1.7
Fix vet errors with go 1.7
2016-07-18 14:01:47 -07:00
Richard Scothern
fcea6145f5 Fix vet errors with go 1.7
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-18 13:41:19 -07:00
Derek McGowan
f8083b7ff3 Merge pull request #1836 from hinshun/catalog-walk-return-error
Fix storage drivers dropping non EOF errors when listing repositories
2016-07-15 14:29:50 -07:00
Derek McGowan
b1ab3bfde5 Merge pull request #1838 from dmcgowan/search-v2-auth-test
Search v2 auth test
2016-07-15 14:27:56 -07:00
Derek McGowan
42a7e6e171 Merge pull request #1837 from RichardScothern/catalog-test
Use registry APIs to generate catalog test environment
2016-07-15 14:23:22 -07:00
Edgar Lee
a82f661ef0 Use typecast over reflect for error type checking
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 15:03:18 -07:00
Edgar Lee
3bfd03cbe6 Refactor errVal named parameter for catalog repositories to err
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 13:28:08 -07:00
Edgar Lee
aeb9a29499 Handle new errors returned from catalog repository listing
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 13:24:16 -07:00
Richard Scothern
b126d6643e Use distribution APIs to generate catalog test environment
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-14 09:58:19 -07:00
Derek McGowan
fc07e0380e Add v1 search with v2 token auth test
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-07-13 17:03:14 -07:00
Edgar Lee
5a0b35ca10 Fix storage drivers dropping non EOF errors when listing repositories
This fixes errors other than io.EOF from being dropped when a storage driver
lists repositories. For example, filesystem driver may point to a missing
directory and errors, which then gets subsequently dropped.

Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-13 16:41:51 -07:00
Derek McGowan
37b5e3e81b Merge pull request #1833 from aaronlehmann/document-toomanyrequests
Document TOOMANYREQUESTS error code
2016-07-13 10:41:01 -07:00
Richard Scothern
07f32ac183 Merge pull request #1830 from dmcgowan/allow-v2-auth-on-v1-search
Add support for using v2 ping challenges for v1
2016-07-12 17:28:54 -07:00
Derek McGowan
145abeea7b Add support for using v2 ping challenges for v1
Allows using v2 for v1 endpoints.
The primary use case being for search which does not have a v2 specification.
Added a user scope for allowing v2 search

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-07-12 17:15:56 -07:00
Derek McGowan
022416c502 Add support for registry type in scope
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-07-12 17:13:43 -07:00
Richard Scothern
c3c5277007 Merge pull request #1834 from dmcgowan/aws-vendor-update
Vendor update for aws sdk
2016-07-12 15:26:16 -07:00
Derek McGowan
acae5dcfff Vendor update for aws sdk
Updated to latest version of go aws sdk.
Use vendored sub pakages within aws sdk.
Adds missing vendor packages for letsencrypt

Fixes #1832

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-07-12 12:47:37 -07:00
Aaron Lehmann
b0099004e2 Document TOOMANYREQUESTS error code
Add entries with this error code in registry/api/v2/descriptors.go.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-07-12 12:18:54 -06:00
Richard Scothern
2052f29be6 Merge pull request #1827 from docker/ec2-creds
Allow EC2 IAM roles to be used when authorizing region endpoints
2016-07-11 13:05:41 -07:00
Richard Scothern
b7d43c862f Merge pull request #1826 from jchorl/cas
Add transport field to notifications endpoint config
2016-07-11 13:03:29 -07:00
Derek McGowan
9d84a6a18e Merge pull request #1828 from nwt/foreign-layer-url-validation
Properly validate multi-URL foreign layers
2016-07-11 13:01:04 -07:00
Noah Treuhaft
042bc06175 Properly validate multi-URL foreign layers
The existing code effectively ignored errors from all but the last of a
foreign layer's URLs.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-07-11 12:13:42 -07:00
Josh Chorlton
a62f212544 Add transport field to EndpointConfig struct
The EndpointConfig struct in the notifications package has some config
fields for a notification endpoint. This commit adds the ability to pass
in an *http.Transport to use when notifying that endpoint of an event.
This is especially useful for endpoints that use self-signed CAs.

Signed-off-by: Josh Chorlton <josh.chorlton@docker.com>
2016-07-11 11:52:23 -07:00
Richard Scothern
3da5f9088d Allow EC2 IAM roles to be used when authorizing region endpoints
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-11 10:54:57 -07:00
Stefan Majewsky
1f03d4e77d [Swift] add simple heuristic to detect incomplete DLOs during read ops
This is similar to waitForSegmentsToShowUp which is called during
Close/Commit. Intuitively, you wouldn't expect missing segments to be a
problem during read operations, since the previous Close/Commit
confirmed that all segments are there.

But due to the distributed nature of Swift, the read request could be
hitting a different storage node of the Swift cluster, where the
segments are still missing.

Load tests on my team's staging Swift cluster have shown this to occur
about once every 100-200 layer uploads when the Swift proxies are under
high load. The retry logic, borrowed from waitForSegmentsToShowUp, fixes
this temporary inconsistency.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-07-08 13:47:41 +02:00
Michal Minář
3f1434525b Export storage.CreateOptions in top-level package
Let the options for `BlobStore.Create()` be modified in middleware
wrappers.

Signed-off-by: Michal Minar <miminar@redhat.com>
2016-07-06 16:40:46 +02:00
Derek McGowan
4e17ab5d31 Merge pull request #1817 from cezarsa/master
Avoid formatting errors with %#v
2016-07-01 10:59:50 -07:00
Cezar Sa Espinola
e015cbadd6
registry: avoid formatting errors with %#v
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-07-01 11:30:58 -03:00
Derek McGowan
b49f8ed894 Merge pull request #1812 from RichardScothern/media-type
Override media type returned from Stat for existing manifests.
2016-06-29 15:28:34 -07:00
Richard Scothern
beb5d00474 Override media type returned from Stat for existing manifests.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-29 14:53:14 -07:00
Derek McGowan
c24b072e7d Merge pull request #1808 from BrianBland/clientAllTagsFollowsLinks
Changes the client Tags All() method to follow links
2016-06-29 12:02:48 -07:00
Brian Bland
a1f9f71e67 Changes the client Tags All() method to follow links
This returns all tags even when the registry forces pagination.

Signed-off-by: Brian Bland <brian.t.bland@gmail.com>
2016-06-28 15:49:14 -07:00
Richard Scothern
1f1d042f55 Merge pull request #1807 from docker/jchorl-master
fixed s3 Delete bug due to read-after-delete inconsistency
2016-06-28 15:09:35 -07:00
Josh Chorlton
2d0a5ecc0e fixed s3 Delete bug due to read-after-delete inconsistency
Signed-off-by: Josh Chorlton <josh.chorlton@docker.com>
2016-06-28 14:22:15 -07:00
Richard Scothern
5f7f871d8f Merge pull request #1805 from SvenDowideit/difuse-non-link
Stop hugo from making the example URL into a link
2016-06-27 10:41:36 -07:00
Sven Dowideit
a1c1349eac Stop hugo from making the example URL into a link
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-06-27 10:15:26 +10:00
Stephen Day
ffbe9b7c63 Merge pull request #1798 from SvenDowideit/replace-google-docs-png-with-local-image
Replace google docs image link with one in this repo
2016-06-22 13:25:24 -07:00
Derek McGowan
e3b4445b83 Merge pull request #1799 from liubin/fix-typos
fix typos
2016-06-22 07:06:04 -07:00
Sven Dowideit
82bdab7d48 Replace google docs image link with one in this repo
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-06-22 13:08:56 +00:00
bin liu
913e12c8ff fix typos
Signed-off-by: bin liu <liubin0329@gmail.com>
2016-06-22 12:40:21 +08:00
Richard Scothern
edd7cb5249 Merge pull request #1739 from cezarsa/master
[Swift] Expose EndpointType parameter in driver
2016-06-15 10:33:48 -07:00
Sven Dowideit
e472758825 Merge pull request #1788 from SvenDowideit/move-build-doc
Move the building.md doc to the top of the git repo
2016-06-15 10:57:13 +10:00
Sven Dowideit
51be30beb1 Move the building.md doc to the top of the git repo
Closes #1776

Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-06-15 20:26:16 +10:00
Charles Smith
879dc2fa38 Merge pull request #1791 from londoncalling/fix-broken-links-PR-23492
fixed broken link due to topic re-org in PR#23492
2016-06-14 14:20:35 -07:00
Victoria Bialas
c0987a9e1b fixed broken link due to topic re-org in PR#23492
Signed-off-by: Victoria Bialas <victoria.bialas@docker.com>
2016-06-14 13:56:17 -07:00
Richard Scothern
f4296d55fc Merge pull request #1787 from RichardScothern/nr
Re-add support for non-resumable digests
2016-06-14 10:37:47 -07:00
Richard Scothern
ccfa25cf00 If resumable digest support is disabled, detct this when closing the blobwriter
and allow the close to continue.  Also update the name of the function.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-13 17:35:06 -07:00
Richard Scothern
1fc752c718 Merge pull request #1706 from aibaars/registry-size-close
Blobwriter: call BlobWriter.Size after BlobWriter.Close
2016-06-13 16:29:35 -07:00
Richard Scothern
337ceb2b40 Merge pull request #1786 from RichardScothern/admin
Prepare for the release.
2016-06-13 16:09:09 -07:00
Richard Scothern
6e2dd4fa68 Prepare for the release.
Update .mailmap, AUTHORS and MAINTAINERS files.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-13 16:03:16 -07:00
Richard Scothern
6445220d0d Merge pull request #1785 from RichardScothern/changelog
Changelog for 2.5
2016-06-13 15:52:33 -07:00
Richard Scothern
c2af377920 Changelog for 2.5
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-13 15:37:26 -07:00
Cezar Sa Espinola
afb262bd5f
Update swift driver docs removing dup table and adding endpointtype
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-06-13 19:30:24 -03:00
Cezar Sa Espinola
7f72092940
Expose EndpointType parameter in swift storage driver
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-06-13 19:28:45 -03:00
Cezar Sa Espinola
5ad9d19ff6
Update vendored version of ncw/swift library
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-06-13 19:28:05 -03:00
Richard Scothern
fb106e167a Merge pull request #1465 from dmcgowan/token-server-oauth
Integration token server supporting oauth
2016-06-13 15:01:06 -07:00
Richard Scothern
4e08e7d1d6 Merge pull request #1775 from dmcgowan/get-content-digest
Add option to get content digest from manifest get
2016-06-13 14:59:05 -07:00
Richard Scothern
805b94eb2b Merge pull request #1770 from Windfarer/fix-link-patch
Fixing link patch
2016-06-13 14:34:42 -07:00
Richard Scothern
1bf52359f1 Merge pull request #1782 from tianon/accept-lists
Update "Accept" header parsing for list values
2016-06-13 14:33:46 -07:00
Aaron Lehmann
9198d642ba Merge pull request #1779 from dmcgowan/letsencrypt-support
Let's Encrypt support
2016-06-13 10:48:55 -10:00
Richard Scothern
015df6cdea Add documentation for let's encrypt
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-13 11:45:18 -07:00
Derek McGowan
be2ed961aa Add support for Let's Encrypt
Add configuration and certificate manager to use letsencrypt

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-13 11:30:42 -07:00
Tianon Gravi
8907f7d189 Update "Accept" header parsing for list values
In Go's header parsing, the same header multiple times results in multiple entries in the `r.Header[...]` slice, but Go does no further parsing beyond that (and in https://golang.org/cl/4528086 it was determined that until/unless the stdlib itself needs it, Go will not do so).

The consequence here for parsing of `Accept:` headers is that we support the way Go outputs headers, but not all language HTTP libraries have a facility to output multiple headers instead of a single list header.

This change ensures that the following (valid) header blocks all parse to the same result for the purposes of what is being tested here:

```
Accept: a/b
Accept: b/c
Accept: d/e
```

```
Accept: a/b; q=0.5, b/c
Accept: d/e
```

```
Accept: a/b; q=0.1, b/c; q=0.2, d/e; q=0.8
```

Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
2016-06-10 16:52:27 -07:00
Derek McGowan
1c99939221 Vendor letsencrypt packages
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-09 17:56:43 -07:00
Derek McGowan
125f4ff7d7 Add option to get content digest from manifest get
The client may need the content digest to delete a manifest using the digest used by the registry.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-08 17:02:29 -07:00
Richard Scothern
75882f079c Merge pull request #1774 from RichardScothern/catalog-clarifcation
Clarify API documentation around catalog fetch behavior
2016-06-08 12:59:09 -07:00
Richard Scothern
352924cd85 Clarify API documentation around catalog fetch behavior
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-08 10:39:17 -07:00
Richard Scothern
2cbf54df6c Merge pull request #1773 from zhouhaibing089/comment-fix
Fix incorrect comment about auth type
2016-06-08 10:31:49 -07:00
Richard Scothern
79102ecf85 Merge pull request #1772 from runcom/go1.6const
registry: use const for status code 429
2016-06-08 10:22:57 -07:00
zhouhaibing089
fc9aa2f351 incorrect comment about auth type
Signed-off-by: haibzhou <haibzhou@ebay.com>
2016-06-08 22:47:45 +08:00
Antonio Murdaca
4901f1b278 registry: use const for status code 429
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-06-08 10:19:15 +02:00
Eric Yang
a3965b93bc Fixing link in readme
Fixing link in readme

Signed-off-by: Qizhao Yang <windfarer@gmail.com>
2016-06-07 11:06:36 +08:00
Eric Yang
a31474626b Merge pull request #1 from docker/master
update
2016-06-07 10:58:38 +08:00
Sven Dowideit
feddf6cd4e Merge pull request #1767 from SvenDowideit/appease-the-oss-validator
Revert to using the full URL, as the OSS checker doesn't have access …
2016-06-02 11:22:55 -07:00
Sven Dowideit
b448455011 Revert to using the full URL, as the OSS checker doesn't have access to the non-OSS docs
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-06-02 18:22:02 +00:00
Richard Scothern
3828ecc654 Merge pull request #1765 from allencloud/fix-typos
fix typos
2016-06-02 10:36:17 -07:00
allencloud
db90724ab0 fix typos
Signed-off-by: allencloud <allen.sun@daocloud.io>
2016-06-02 23:03:27 +08:00
Sven Dowideit
bb330cd684 Merge pull request #1762 from SvenDowideit/more-docs-validation-cleanups
Next load of docs validation fixes
2016-06-01 11:25:54 -07:00
Richard Scothern
df2184c810 Merge pull request #1627 from luckyraul/swift_auth_url
Swift auth version param
2016-06-01 11:23:23 -07:00
Sven Dowideit
e4acec1806 Next load of docs validation fixes
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-06-01 18:18:17 +00:00
Sven Dowideit
e66f9c1440 Merge pull request #1759 from SvenDowideit/fix-new-doc-metadata
Add topics to the menu
2016-06-01 10:35:14 -07:00
Sven Dowideit
bf4eb92f56 Add topics to the menu
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-05-31 21:31:25 +00:00
Derek McGowan
d9c4f03bcc Merge pull request #1760 from RichardScothern/revert-config
Revert inadvertant filepath change
2016-05-31 11:16:19 -07:00
Richard Scothern
d153bfffbd Revert inadvertant filepath change
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-31 11:14:25 -07:00
Sven Dowideit
1b9ab303a4 Merge pull request #1735 from SvenDowideit/move-menu-page
Make the index.html files the visible overview menu entry
2016-05-31 09:32:53 -07:00
Richard Scothern
641f102967 Merge pull request #1687 from RichardScothern/signature-store
Remove signature store from registry.
2016-05-31 09:09:39 -07:00
Sven Dowideit
3069a04b8c Make the index.html files the visible overview menu entry
And move menu entry definition into a page that the user has no reason to navigate to

Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-05-28 01:02:40 +00:00
Richard Scothern
febcee6564 Add a deprecation document detailing signature store removal
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-27 14:51:59 -07:00
Richard Scothern
0c15ab6952 Remove signature store from registry. Return a generated signature for manifest
pull.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-27 13:19:26 -07:00
Richard Scothern
a5f72a6032 Merge pull request #1688 from tt/pass-through-known-errors
Pass through known errors
2016-05-27 11:35:34 -07:00
Richard Scothern
6a40198283 Merge pull request #1719 from gschukin/fix_ceph_last_modifed
Handle responce from ceph correctly if no Last-Modifed header specified
2016-05-27 11:34:39 -07:00
Richard Scothern
596ca8b86a Merge pull request #1744 from tonyhb/add-config-to-middleware
Pass in `app` as context to apply{N}Middleware
2016-05-25 14:08:09 -07:00
Richard Scothern
7f5c6dd882 Merge pull request #1748 from dmcgowan/fix-expires-in-documentation
Update auth documentation examples to show "expires in" as int
2016-05-25 12:48:01 -07:00
Derek McGowan
3207e0c112 Update auth documentation examples to show "expires in" as int
Go will fail to parse the examples since an int is expected rather than a string for the "expires in" value

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-05-25 11:43:36 -07:00
Derek McGowan
0bd7e971b0 Merge pull request #1743 from RichardScothern/tcp-bug
Correct yaml key names in configuration
2016-05-24 11:09:10 -07:00
Tony Holdstock-Brown
166c4a957f Pass in app as context to apply{N}Middleware
This lets us access registry config within middleware for additional
configuration of whatever it is that you're overriding.

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-24 11:08:57 -07:00
Richard Scothern
b9d6c7c9fb Merge pull request #1741 from leonhartX/master
fix broken markdown
2016-05-24 11:04:31 -07:00
Richard Scothern
588692f6c4 Correct yaml key names in configuration
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-24 10:42:29 -07:00
Richard Scothern
32e9779faf Merge pull request #1734 from legionus/blob-access-controller
Add support for  blobAccessController middleware
2016-05-24 10:34:36 -07:00
Ke Xu
79d6008a54 fix broken markdown
Signed-off-by: Ke Xu <leonhartx.k@gmail.com>
2016-05-24 16:33:30 +09:00
Aaron Lehmann
5bbf654999 Merge pull request #1725 from jstarks/foreign_layer
Add support for layers from foreign sources
2016-05-20 12:55:55 -07:00
John Starks
f0052b8434 Add support for layers from foreign sources
This will be used to support downloading Windows base layers from
Microsoft URLs.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-05-20 12:31:10 -07:00
Alexey Gladkov
f97eca5ad6 Add support for blobAccessController middleware
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
2016-05-19 14:02:15 +02:00
Nikita Tarasov
f18da7d3a3 update docs
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-05-16 16:09:26 +03:00
Derek McGowan
4a915d6efd Merge pull request #1721 from cyli/change-auth-scope-grammar
Hostnames can be optionally supported by the auth spec scope grammar
2016-05-11 16:30:28 -07:00
cyli
8854eed7ab Update the auth spec scope grammar to reflect the fact that hostnames are optionally supported.
Signed-off-by: cyli <cyli@twistedmatrix.com>
2016-05-11 15:19:50 -07:00
Gleb Schukin
5d7600e3ca Handle rare case when ceph doesn't return Last-Modified for HEAD requests in case DLO manifest doesn't have any segments
Signed-off-by: Gleb Schukin <gs1983@gmail.com>
2016-05-11 15:13:13 +03:00
Gleb Schukin
52eecb556c Handle rare case when ceph doesn't return Last-Modified for HEAD requests in case DLO manifest doesn't have any segments
Signed-off-by: Gleb Schukin <gs1983@gmail.com>
2016-05-11 15:06:08 +03:00
Derek McGowan
e69f108924 Merge pull request #1716 from dmcgowan/install-certs
Integration Tests: Generate the certificate directory with a pretest script
2016-05-10 11:02:59 -07:00
Richard Scothern
050a30eac0 Merge pull request #1702 from RichardScothern/api.md
Remove the unimplemented monolithic upload section from the API
2016-05-10 13:43:25 +01:00
Richard Scothern
42a710f889 Merge pull request #1714 from RichardScothern/us-gov-west-1
Add 'us-gov-west-1' to the valid region list.
2016-05-10 11:44:18 +01:00
Derek McGowan
db713e127b Generate the certificate directory with a pretest script
fixes #1690

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-05-09 14:21:53 -07:00
Richard Scothern
4f2ee029a2 Add 'us-gov-west-1' to the valid region list.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-09 16:38:16 +01:00
Arthur Baars
eca581cf36 StorageDriver: GCS: allow Cancel on a closed FileWriter
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
0490ff450b Blobwriter: call BlobWriter.Size after BlobWriter.Close
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
1d782c38f2 StorageDriver: Test case for #1698
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Richard Scothern
2a4deee441 Remove the unimplemented monolithic upload section from the API documentation.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-04 15:58:08 -07:00
Olivier Gambier
4ecea4b311 Merge pull request #1663 from moxiegirl/fix-flags
Fixing nginx flag issue identified by customer
2016-05-04 13:52:21 -07:00
Mary Anthony
b0db8d49bd Fixing issue identified by customer
Entering Seb's comment
Fix the flags

Signed-off-by: Mary Anthony <mary@docker.com>
2016-05-04 13:34:23 -07:00
Derek McGowan
48aac6834a Merge pull request #1701 from RichardScothern/updates
Update AUTHORS and version file
2016-05-04 11:04:48 -07:00
Richard Scothern
c1c230dee9 Merge pull request #1626 from RichardScothern/alpine
Use Alpine Linux as the parent image for the registry.
2016-05-04 11:03:42 -07:00
Richard Scothern
35dd23c649 Update AUTHORS and version file
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-04 10:22:07 -07:00
Richard Scothern
c047d34b22 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem
Add regulator to filesystem
2016-05-04 10:05:51 -07:00
Richard Scothern
9ec0d742d6 Merge pull request #1696 from runcom/go1.6-lol
registry: do not use http.StatusTooManyRequests
2016-05-04 09:45:18 -07:00
Tony Holdstock-Brown
c9c62380ff Don't wrap thead limits when using a negative int
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 16:03:44 -07:00
Antonio Murdaca
f60f275c29 registry: do not use http.StatusTooManyRequests
go1.5 doesn't export http.StatusTooManyRequests while
go1.6 does. Fix this by hardcoding the status code for now.

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 21:24:45 +02:00
Aaron Lehmann
d4be7016ef Merge pull request #1693 from runcom/too-many-requests
registry: type too many requests error
2016-05-03 10:16:21 -07:00
Antonio Murdaca
a264f9ae29 registry: type too many requests error
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 18:41:54 +02:00
Tony Holdstock-Brown
d0352a7448 Add documentation
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 09:33:25 -07:00
Tony Holdstock-Brown
33c448f147 Implement regulator in filesystem driver
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 09:33:22 -07:00
Sven Dowideit
b7088d29c6 Merge pull request #1680 from SvenDowideit/use-docs-base-oss
convert docs Dockerfiles to use docs/base:oss
2016-05-03 10:48:11 +10:00
Richard Scothern
5d08dfa70c Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments
[Swift] wait for DLO segments to show up when Close()ing the writer
2016-05-02 13:41:26 -07:00
Derek McGowan
a1d7463d67 Merge pull request #1669 from RichardScothern/close-after-commit
Clean uploads
2016-04-29 16:18:29 -07:00
Aaron Lehmann
024a9ed6c7 Merge pull request #1681 from aaronlehmann/preserve-authors-schema1
Preserve author information in schema1 manifests
2016-04-29 14:40:24 -07:00
Troels Thomsen
3730470b64 Pass through known errors
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2016-04-29 23:34:24 +02:00
Derek McGowan
72c9bfe1b5 Merge pull request #1683 from dmcgowan/update-golem-test-versions
Use official docker 1.11.1 image to test docker 1.11
2016-04-29 09:54:28 -07:00
Derek McGowan
8cded9db0d Use official docker 1.11.1 image to test docker 1.11
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-04-28 14:51:13 -07:00
Richard Scothern
ba927007b0 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error
Move GC into storage package and add tests
2016-04-28 14:09:58 -07:00
Aaron Lehmann
620a030217 Preserve author information in schema1 manifests
When we push a schema1 manifest, we encode history information from the
image JSON into v1Compatibility strings for the respective layers. The
"author" field was not being set in these v1Compatibility strings, so if
a parent layer had an author set, it would not be preserved after
pushing through a schema1 manifest and repulling, so the image ID would
change after the pull. This change preserves the authorship information
for parent layers so that the image ID does not change.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-04-28 11:57:48 -07:00
Sven Dowideit
e728c8bbed convert docs Dockerfiles to use docs/base:oss
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2016-04-28 12:57:10 +00:00
Richard Scothern
a7dda2ce93 Merge pull request #1665 from andrewhsu/middleware-redirect
add middleware storage driver for redirect
2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0 When a blob upload is committed prevent writing out hashstate in the
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03 Add a test with a missing _manifests directory
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a Move garbage collect code into storage package
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Tony Holdstock-Brown
a5aaae1f06 Ensure GC continues marking if _manifests is nonexistent
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-27 13:33:36 -07:00
Richard Scothern
6a992e1348 Merge pull request #1675 from sergeyfd/master
Fix wording for dry-run flag in usage message for garbage collector.
2016-04-27 10:08:53 -07:00
Serge Dubrouski
32193bdcf0 Fix wording for dry-run flag in useage message for garbage collector.
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-26 19:44:23 -06:00
Josh Hawn
e4dd3359cc Regulate filesystem driver to max of 100 calls
It's easily possible for a flood of requests to trigger thousands of
concurrent file accesses on the storage driver. Each file I/O call creates
a new OS thread that is not reaped by the Golang runtime. By limiting it
to only 100 at a time we can effectively bound the number of OS threads
in use by the storage driver.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-26 14:44:13 -07:00
Andrew Hsu
09a9b0cf90 separate the go/non-go imports and reorder
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-26 14:33:54 -07:00
Richard Scothern
47d14555c0 Merge pull request #1644 from fh1ch/clarify-kid-format
Clarify kid format for JWT token auth in docs
2016-04-25 17:01:15 -07:00
Richard Scothern
d654cfd985 Merge pull request #1623 from dmcgowan/docker-integration-readme
Integration test readme update
2016-04-25 16:10:19 -07:00
Richard Scothern
c83afea0c9 Merge pull request #1660 from jhaohai/cn-north-1-fix
Add cn-north-1 to valid check
2016-04-25 16:07:54 -07:00
Richard Scothern
ef32134592 Merge pull request #1666 from sergeyfd/master
Add blobWrtiter.Close() call into blobWriter.Commit()
2016-04-25 16:02:48 -07:00
Richard Scothern
96f796fb01 Merge pull request #1670 from vadmeste/fix_parts_sorting
s3 driver: Sorting completed parts by part number for a better accordance with S3 spec
2016-04-25 16:02:17 -07:00
Andrew Hsu
c4df027d41 modify redirect test to include port
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:46 -07:00
Andrew Hsu
fe9509f8f3 added config doc for redirect middleware
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:39 -07:00
Andrew Hsu
80248c3d3a scheme and host mandatory in baseurl
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:25 -07:00
Andrew Hsu
059bc5f5ef separate the go/non-go imports and reorder
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:03 -07:00
Anis Elleuch
987faca8a6 Sorting completed parts by part number for a better accordance with the S3 spec
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
2016-04-23 22:36:04 +01:00
Serge Dubrouski
21f38a74e6 Add blobWrtiter.Close() call into blobWriter.Commit()
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-22 19:23:17 -06:00
Andrew Hsu
4b217ccbf5 add middleware storage driver for redirect
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-21 16:02:52 -07:00
jhaohai
f76c622d8c add cn-north-1 to valid check
Signed-off-by: jhaohai <jhaohai@foxmail.com>
2016-04-21 11:51:34 +08:00
Richard Scothern
cd27f179f2 Merge pull request #1635 from hopkings2008/notifytype
use context.GetLogger to replace logrus in listener
2016-04-19 10:04:23 -07:00
Stefan Majewsky
9a67520af7 wait for DLO segments to show up when Close()ing the writer
Not just when Commit()ing the result. This fixes some errors I observed
when the layer (i.e. the DLO) is Stat()ed immediately after closing,
and reports the wrong file size because the container listing is not
yet up-to-date.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-19 14:31:49 +02:00
Nikita Tarasov
b4f060599a docs + fix test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 20:05:51 +03:00
Nikita Tarasov
b51607f9f0 fix test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>

Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 19:52:40 +03:00
Fabio Huser
17756eb43e Clarify kid format for JWT token auth in docs
The kid value can have an arbitrary format according JOSE specification, but Docker distribution expects a specific format (libtrust fingerprint) to work. This is not written in the documentation so far and is only mentioned in the libtrust source code itself.

Signed-off-by: Fabio Huser <fabio@fh1.ch>
2016-04-17 12:04:15 +02:00
Richard Scothern
9d491698cc Merge pull request #1641 from RichardScothern/ifollowtherules
Correction for JSON example.
2016-04-15 09:52:20 -07:00
Richard Scothern
b72d74464a Correction for JSON example.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-15 09:22:44 -07:00
yuzou
f2686b8db4 use context.GetLogger to replace logrus in listener
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-04-15 11:18:26 +08:00
Richard Scothern
e8feabc775 Use Alpine Linux as the parent image for the registry.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-14 11:29:04 -07:00
Richard Scothern
05b0ab0302 Merge pull request #1630 from hopkings2008/notifytype
fix typepo for log message of layer push event in blobServiceListener…
2016-04-14 10:39:58 -07:00
yuzou
098005177f fix typepo for log message of layer push event in blobServiceListener Put function.
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-04-14 16:41:35 +08:00
Richard Scothern
e90ff92895 Merge pull request #1625 from moxiegirl/fix-1598
Updated per conversation with Richard
2016-04-13 09:46:43 -07:00
Nikita
bcb7989fca test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:38 +03:00
Nikita
e6f8d7c28e Update swift.go
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:25 +03:00
Mary Anthony
6bce49d51d Updated per conversation with Richard
Removing draft
Richard's comments and some fixes

Signed-off-by: Mary Anthony <mary@docker.com>
2016-04-13 06:43:11 -07:00
Derek McGowan
ab2394446c Integration test readme update
Updates the readme to mention running the tests using golem.
Also provides instructions for making test development easier.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-04-12 14:03:56 -07:00
Richard Scothern
c6f63e298e Merge pull request #1619 from RichardScothern/gc-docs
Extend garbage collection documentation.
2016-04-12 10:25:43 -07:00
Richard Scothern
f9bcbd44ca Extend garbage collection documentation.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-11 17:43:25 -07:00
Richard Scothern
467fc068d8 Merge pull request #1622 from aaronlehmann/schema2-config-mediatype
Use correct media type for config blob in schema2 manifest
2016-04-11 17:23:03 -07:00
Aaron Lehmann
2de3f1a62a Use correct media type for config blob in schema2 manifest
The schema2 manifest builder fills in this part of the manifest based on
the descriptor it gets back from BlobIngester's Put method. It passes
the correct media type to Put, but Put ends up replacing this value with
application/octet-stream in its return value.

This commit works around the issue in the manifest builder. Arguably Put
should not be changing the media type in its return value, but this
commit is a targeted fix to keep it very low-risk for possible inclusion
in Docker 1.11.

Fixes #1621 (but maybe we should open a separate issue for the media
type behavior in the distribution client, and the unnecessary stat).

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-04-11 17:05:41 -07:00
Richard Scothern
4c119524f1 Merge pull request #1604 from ArdaXi/custom-s3-skip-region-check
Only check validity of S3 region if not using custom endpoint
2016-04-08 15:38:24 -07:00
Richard Scothern
437eeeda44 Merge pull request #1605 from majewsky/swift/deal-with-outdated-container-listings
registry/storage/swift: detect and fix outdated container listings
2016-04-08 15:38:06 -07:00
Richard Scothern
55f1b7651f Merge pull request #1590 from RichardScothern/s3-permission-scopes
Document required IAM permissions for S3 storage driver.
2016-04-06 14:46:12 -07:00
Richard Scothern
e4817cfc94 Remove ListAllMyBuckets from the S3 permission scope.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-06 14:22:08 -07:00
Stefan Majewsky
67321cb622 detect outdated container listings during Stat() and getAllSegments()
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-06 15:21:27 +02:00
Arien Holthuizen
dbb6e28da2 Only check validity of S3 region if not using custom endpoint
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
2016-04-06 13:38:09 +02:00
Richard Scothern
27e0be3e95 Merge pull request #1583 from dmcgowan/golem-integration-tests
Update docker integration tests to use golem
2016-04-05 15:38:01 -07:00
Richard Scothern
75e55632fd Merge pull request #1597 from tonyhb/log-upload-copy-errors
Ensure we log io.Copy errors and bytes copied/total in uploads
2016-04-05 09:46:12 -07:00
Derek McGowan
2ea61dc04f Add temporary cache directory
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-04-04 19:57:40 -07:00
Derek McGowan
17f7f60d77 Update docker integration tests to use golem
Use registry example from golem repository.
Use the golem test runner for the docker integration environment

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-04-04 19:57:40 -07:00
Tony Holdstock-Brown
25c5efdef9 Ensure we log io.Copy errors and bytes copied/total in uploads
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-04 17:21:36 -07:00
Richard Scothern
20fa47886d Merge pull request #1592 from estesp/manifest-spec-example-fix
Correct examples and architecture references in v2.2 spec
2016-04-04 10:39:54 -07:00
Phil Estes
bf9f80eaff Correct examples and architecture references in v2.2 spec
Add link to the official list of $GOOS and $GOARCH values and correct
values that were incorrectly listed in the spec examples.

Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
2016-04-03 17:50:22 -04:00
Olivier Gambier
ff6f38ccb6 Merge pull request #1588 from aaronlehmann/golint-godep-location
Makefile: don't look for golint and godep in specific places
2016-03-31 15:21:40 -07:00
Richard Scothern
0f4427b3c3 Document required IAM permissions for S3 storage driver.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-31 14:34:59 -07:00
Olivier Gambier
2c803d6fd0 Merge pull request #1587 from aaronlehmann/build-docs
docs: No need to change GOPATH to use vendored code
2016-03-31 14:04:57 -07:00
Aaron Lehmann
20525e390a Makefile: don't look for golint and godep in specific places
Using $GOPATH/bin/godep or $GOPATH/bin/golint is problematic because
$GOPATH can contain multiple colon-separated paths.

We should just run these like normal binaries. The user should make sure
their $PATH contains $GOPATH/bin, if necessary. This is part of normal
Go setup.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-31 11:00:28 -07:00
Aaron Lehmann
c3ec1745c0 docs: No need to change GOPATH to use vendored code
Now that we are using "native" Go vendoring, there is no need to
manipulate GOPATH.

Fixes #1586

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-31 10:46:02 -07:00
Aaron Lehmann
a11f6b6cfd Merge pull request #1418 from dmcgowan/oauth-spec
Add specification for using OAuth with the token server
2016-03-31 10:42:12 -07:00
Richard Scothern
04ff3c0359 Merge pull request #1578 from majewsky/do-not-swallow-error
don't swallow errors in Swift driver's GetContent()
2016-03-30 17:46:27 -07:00
Richard Scothern
13f72ef7dc Merge pull request #1560 from RichardScothern/gc
Fix signature handling with GC.
2016-03-30 17:46:04 -07:00
Richard Scothern
eb0b7f0173 Update the gc documentation.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:57:21 -07:00
Richard Scothern
4324b70c50 Fix signature handling with GC.
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist.  Handle this case.

If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist.  Disable the signature store in the garbage-collect command so
signatures are not fetched.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
80b310ca44 Add a --dry-run flag. If enabled this will print the mark and sweep process
with removing any files.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
ee8450ff13 Merge pull request #1581 from RichardScothern/docs
Manifest delete documentation enhancement
2016-03-30 16:05:57 -07:00
Richard Scothern
e5aa9c8e56 Merge pull request #1577 from xiekeyang/context
type judgment in context
2016-03-30 14:29:48 -07:00
Richard Scothern
791865af71 Merge pull request #1582 from aaronlehmann/garbagecollect-errors
garbagecollect: Clean up errors
2016-03-30 14:29:27 -07:00
Aaron Lehmann
267e268775 garbagecollect: Clean up errors
- Clean up error messages

- Add a missing error check on the result of blobService.Enumerate.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-30 11:35:24 -07:00
Richard Scothern
9b5caf64cf Add a note to the delete manifest section to aid getting the correct
digest to delete manifests.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 10:59:26 -07:00
Stefan Majewsky
4c9bed2507 don't swallow errors in Swift driver's GetContent()
In 326c3a9c49, which was only intended to
be a refactoring commit, the behavior of this block subtly changed so
that unknown types of errors would be swallowed instead of propagated.

I noticed this while investigating an error similar to #1539 aka
docker/docker#21290. It appears that during GetContent() for a
hashstate, the Swift proxy produces an error. Since this error was
silently swallowed, an empty []byte is used to restart the hash, then
producing the digest of the empty string instead of the layer's digest.

This PR will not fix the issue, but it should make the actual error more
visible by propagating it into `blobWriter#resumeDigest' and
'blobWriter#validateBlob', respectively.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-03-30 16:11:29 +02:00
xiekeyang
e388517d21 type judgment in context
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2016-03-30 17:35:48 +08:00
Stephen Day
0eefe17730 Merge pull request #1546 from arschles/doc-storage
Add documentation for how to register new StorageDrivers
2016-03-29 15:58:51 -07:00
Aaron Schlesinger
f4bdc6287a Remove the example
Instead, direct users to the one in the factory package

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:28 -07:00
Aaron Schlesinger
204ad474e4 Add documentation for how to register new StorageDrivers
This commit adds context-specific documentation on StorageDriver,
StorageDriverFactory, and the factory’s Register func, explaining how
the internal registration mechanism should be used.

This documentation follows from the thread starting at
https://github.com/deis/builder/pull/262/files#r56720200.

cc/ @stevvooe

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:19 -07:00
Richard Scothern
19ad02e6ac Merge pull request #1559 from moxiegirl/fix-925
Update the Duration value for Cloudfront
2016-03-29 10:30:19 -07:00
Mary Anthony
6018bdf58a Updating example
Signed-off-by: Mary Anthony <mary@docker.com>
2016-03-29 09:53:13 -07:00
Mary Anthony
188206eede Fixes and closes #925
Seconds to minutes as per code
Correction per Derek
Clarifying failure case

Signed-off-by: Mary Anthony <mary@docker.com>

Clarifying failure case

Signed-off-by: Mary Anthony <mary@docker.com>
2016-03-29 06:17:29 -07:00
Olivier Gambier
60b9f1756d Merge pull request #1570 from RichardScothern/version
Rev base version to 2.4.0
2016-03-28 13:56:24 -07:00
Richard Scothern
1ea9a3638d Rev base version to 2.4.0
Now that we've tagged a release candidate, we can now say all builds are
"2.4.0+unknown" if the makefile is not used.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-28 13:54:28 -07:00
Richard Scothern
72ec74834b Merge pull request #1569 from GameScripting/patch-1
added heading to get more focus into the headers that need to be set
2016-03-28 10:34:48 -07:00
Daniel Huhn
321e0420bf added heading to get more focus into the headers that need to be set
Signed-off-by: Daniel Huhn <daniel@danielhuhn.de>
2016-03-28 18:07:03 +02:00
Aaron Lehmann
f4b6fc8d68 Merge pull request #1557 from tonyhb/gc-logging
Utulize config log format within gc
2016-03-24 15:53:26 -07:00
Tony Holdstock-Brown
31d5dd5175 utulize config log format within gc
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-03-24 14:29:43 -07:00
Olivier Gambier
687fc1e164 Merge pull request #1558 from RichardScothern/AUTHORS
Update AUTHORS
2016-03-24 13:07:41 -07:00
Richard Scothern
ba60866139 Update AUTHORS
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-24 13:03:46 -07:00
Richard Scothern
f2d128e2b0 Merge pull request #1493 from thaJeztah/fix-storage-driver-links
Fix broken links in configuration
2016-03-24 10:57:58 -07:00
Richard Scothern
9cd193d395 Merge pull request #1556 from dmp42/6.fix-codecov
Fix codecov following vendor
2016-03-24 10:57:07 -07:00
Olivier Gambier
37c2b9e3f1 Fix codecov following vendor
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-24 10:30:14 -07:00
Sebastiaan van Stijn
a92c93d770 Fix broken links in configuration
HTML links are not converted by HUGO, so will work
on GitHub, but not in the online documentation.

Converted the HTML table (and links) to Markdown
to fix broken links.

Also added a header for the table, because none
was present.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2016-03-24 10:28:36 -07:00
Richard Scothern
b787035d8b Merge pull request #1545 from jonjohnsonjr/typos
Fix typos in docs/spec/api.md
2016-03-23 16:41:44 -07:00
Richard Scothern
87a997249d Merge pull request #1522 from RichardScothern/tag-events
Send tag events to notification listeners
2016-03-23 16:41:29 -07:00
Stephen Day
9e690c7fa2 Merge pull request #1491 from RichardScothern/relative-url
Return relative URLs
2016-03-23 16:19:39 -07:00
Richard Scothern
7e552e1972 Update documentation and examples in notifications.md
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 15:51:32 -07:00
Richard Scothern
bc9c820e4b Enable URLs returned from the registry to be configured as relative.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 15:14:32 -07:00
Richard Scothern
afe2bdd1c5 Propogate tag as a functional argument into the notification system to attach
tags to manifest push and pull event notifications.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 14:57:52 -07:00
Aaron Lehmann
1354eb9967 Merge pull request #1554 from felixonmars/patch-1
Replace a strange comma with a space.
2016-03-23 13:03:52 -07:00
Richard Scothern
d7fd863ba4 Merge pull request #1544 from dmp42/move-to-vendor
Move to vendor + go1.6
2016-03-23 12:43:44 -07:00
Olivier Gambier
abc9c65e34 Make dep-validate more selective
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-23 12:10:40 -07:00
Felix Yan
37606af003 Replace a strange comma with a space.
I believe it's a typo somehow? :)

Signed-off-by: Felix Yan <felixonmars@archlinux.org>
2016-03-23 17:35:30 +08:00
Olivier Gambier
d1444b56e9 Fix GCS
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-22 10:50:52 -07:00
Olivier Gambier
59401e277b Dependency validation enforce
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-22 10:45:50 -07:00
Olivier Gambier
53e3c1d7b2 Fix godeps
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-22 10:45:50 -07:00
Olivier Gambier
77e69b9cf3 Move to vendor
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-22 10:45:49 -07:00
Aaron Lehmann
c8d8e7e357 Merge pull request #1534 from jstarks/windows_platform
Add osversion and osvariant to platform specification
2016-03-21 16:17:18 -07:00
Richard Scothern
4562cb5a78 Merge pull request #1547 from jzwlqx/issue1535
fix manifest revision search, closes #1535
2016-03-21 16:01:16 -07:00
姜继忠
92a9ee34d4 fix manifest revision search, closes #1535
Signed-off-by: 姜继忠 <jizhong.jiangjz@alibaba-inc.com>
2016-03-19 19:11:25 +08:00
Jon Johnson
f64981b472 Fix typos in docs/spec/api.md
Fix grammatical and formatting issues.
Add links to sections or other documents where appropriate.

Signed-off-by: Jon Johnson <jonjohnson@google.com>
2016-03-18 15:01:20 -07:00
Derek McGowan
d06d6d3b09 Merge pull request #1542 from runcom/typed-errors
registry: client: auth: type errors
2016-03-18 10:37:55 -07:00
Antonio Murdaca
ea148448a2 registry: client: auth: type errors
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-18 09:34:50 +01:00
John Starks
5b0a484739 Add os.version and os.features to platform spec
These changes are needed to differentiate Windows images.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-03-17 16:54:21 -07:00
Richard Scothern
ec6ac0c05e Merge pull request #1531 from aaronlehmann/empty-errors-slice
Don't return empty errcode.Errors slices
2016-03-17 10:46:51 -07:00
Richard Scothern
609aa7cc53 Merge pull request #1538 from runcom/body-leaks
registry: client: repository: close response body
2016-03-16 15:58:22 -07:00
Antonio Murdaca
75b5a1b1f6 registry: client: repository: close response body
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-16 19:53:17 +01:00
Richard Scothern
2c635d1f7e Merge pull request #1532 from RichardScothern/azure-error-types
Update missing blob error checking with latest Azure API
2016-03-15 10:08:36 -07:00
Aaron Lehmann
3a2231fe39 Include status code in UnexpectedHTTPResponseError
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-15 09:03:56 -07:00
Richard Scothern
a5d372535e Merge pull request #1502 from RichardScothern/auth-url-endpoints
URL parse auth endpoints to normalize hostname to lowercase.
2016-03-14 16:28:24 -07:00
Richard Scothern
772e6f4057 Update missing blob error checking with latest Azure API
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-14 15:59:03 -07:00
Aaron Lehmann
cd6482ecb8 Don't return empty errcode.Errors slices
If this slice ends up empty after parsing the HTTP response body, it
means the body is not well-formed. We've probably encountered an error
message produced by something that uses a different JSON schema, or
an error that just happens to validate as JSON.

An empty errcode.Errors slice is not a very useful thing to return,
since its Error() output is just `<nil>`. Detect this case, and instend
return an UnexpectedHTTPResponseError.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-14 10:27:01 -07:00
Brian Bland
1dbe50bac4 Merge pull request #1523 from matt-duch/master
registry/storage/driver/s3-aws kms support
2016-03-11 16:14:46 -08:00
Matt Duch
fcb247dfce registry/storage/driver/s3-aws kms support
Signed-off-by: Matt Duch <matt@learnmetrics.com>
2016-03-11 17:19:01 -06:00
Brian Bland
c03b5fc5ee Merge pull request #1438 from BrianBland/newStorageDriverWriter
Adds new StorageDriver.FileWriter interface
2016-03-11 15:06:07 -08:00
Brian Bland
5967d33342 Removes ceph rados driver in favor of Swift API gateway support
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-10 16:49:08 -08:00
Brian Bland
ab49f85db8 Merge pull request #1512 from kmala/master
Added support to specifiy custom endpoint
2016-03-10 13:48:59 -08:00
Keerthan Mala
2be1b4ef4f Added support to specifiy custom endpoint
Signed-off-by: Keerthan Reddy Mala <keerthan.mala@gmail.com>
2016-03-09 16:12:20 -07:00
Olivier Gambier
db17a23b96 Merge pull request #1521 from dmcgowan/oauth-client-id-on-get
Add client ID to token fetch to GET endpoint
2016-03-09 13:40:47 -08:00
Derek McGowan
a59448784b Add client ID to token fetch to GET endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-09 12:44:55 -08:00
Derek McGowan
093fbdbfc4 Add client_id to get token endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-09 12:35:20 -08:00
Brian Bland
7fd1db9312 Updates Swift driver to support new storagedriver.FileWriter interface
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:38:39 -08:00
Li Yi
a9bf7a2aae Support FileWriter interface for OSS storage driver
Change-Id: Ie5533ad85f944800499ca1040fd67bf1378815e0
Signed-off-by: Li Yi <denverdino@gmail.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
307504713f Storagedriver: GCS: add chunksize parameter
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
7162cb19c6 Storagedriver: GCS: implement resumable uploads
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
666273d9f6 StorageDriver: Testsuite: call Close before getting Size
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:38 -08:00
Brian Bland
ff03381d49 Adds new storagedriver.FileWriter interface
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
 + Inmemory
 + Filesystem
 + S3
 + Azure

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:37:44 -08:00
Olivier Gambier
46b2c3fadf Merge pull request #1517 from aaronlehmann/fix-2-misspellings
Fix two misspellings in source code comments
2016-03-08 15:25:09 -08:00
Aaron Lehmann
87e34bd307 Fix two misspellings in source code comments
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-08 15:13:24 -08:00
Brian Bland
b61928652c Merge pull request #1514 from BrianBland/s3awsParameterParsing
[driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
2016-03-07 16:13:26 -08:00
Richard Scothern
8bd613aa11 URL parse auth endpoints to normalize hostname to lowercase.
It is possible for a middlebox to lowercase the URL at somepoint causing a
lookup in the auth challenges table to fail.  Lowercase hostname before
using as keys to challenge map.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-07 14:08:21 -08:00
Brian Bland
302cc39937 [driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
Mirrors changes from #1414 into the newer driver

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-07 11:52:34 -08:00
Richard Scothern
500fca56a1 Merge pull request #1513 from Windfarer/patch-1
command correction in documentation
2016-03-07 11:19:54 -08:00
Eric Yang
85eadae19a command correction in documentation
the original ```$GOPATH/bin/registry $GOPATH/src/github.com/docker/distribution/cmd/registry/config-example.yml``` leads to the error like

```
Error: unknown command "/Users/EricYang/go/src/github.com/docker/distribution/cmd/registry/config-example.yml" for "registry"
Run 'registry --help' for usage.
```

I think the correct command should be ```registry serve```

Signed-off-by: Eric Yang <EricYang@EricdeMacBook-Pro.local>
2016-03-07 10:42:35 +08:00
Aaron Lehmann
33c4a6522e Merge pull request #1511 from dmcgowan/fix-oauth-cross-repo-push
Fix oauth cross repository push
2016-03-04 15:29:08 -08:00
Derek McGowan
a8aa2f78de Fix oauth cross repository push
Cross repository push tokens were not being cached and could not be used,
now any returned token will be used and the caching is hidden in the getToken function.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 15:13:27 -08:00
Aaron Lehmann
274825f403 Merge pull request #1510 from dmcgowan/oauth-offline-access
Add offline token option
2016-03-04 14:47:48 -08:00
Derek McGowan
efd7ea4624 Add offline token option
Login needs to add an offline token flag to ensure a refresh token is returned by the token endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 14:32:51 -08:00
Derek McGowan
1126e32234 Add post token implementation
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 13:53:06 -08:00
Richard Scothern
5b8592b72c Merge pull request #1475 from dmcgowan/oauth-registry-client
Add oauth support to registry client auth
2016-03-04 11:51:53 -08:00
Derek McGowan
2ef7a872de Add options struct to initialize handler
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 11:32:48 -08:00
Derek McGowan
c9880e6b05 Fetch token by credentials and refresh token
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 00:34:17 -08:00
Derek McGowan
66d6eaa83f Update scope list to use space separator
The oauth spec defines using a space to separate parts of a scope.
To better comply with future implementations built on oauth use a space to separate the resource scopes.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-03 17:44:09 -08:00
Derek McGowan
d51f76f903 Update oauth documentation to include returned scope
Add post response values
Update password grant type to match oauth spec

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-03 17:44:09 -08:00
Richard Scothern
62d46ef07d Merge pull request #1388 from aibaars/gcs-simplify-move
StorageDriver: GCS: remove support for directory Moves
2016-03-03 10:20:51 -08:00
Arthur Baars
1593c2413f StorageDriver: GCS: remove support for directory Moves
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-03 13:28:13 +00:00
Richard Scothern
c8dff1bb57 Merge pull request #1364 from aaronlehmann/content-type-in-api-docs
Add information about manifest content types to API spec
2016-03-01 13:53:49 -08:00
Richard Scothern
5cbac00dd4 Merge pull request #1386 from andrewnguyen/garbage_collect_pick_from_1050
garbage collection
2016-02-29 15:01:48 -08:00
Andrew T Nguyen
feab4aafbc Implements garbage collection subcommand
- Includes a change in the command to run the registry. The registry
  server itself is now started up as a subcommand.
- Includes changes to the high level interfaces to support enumeration
  of various registry objects.

Signed-off-by: Andrew T Nguyen <andrew.nguyen@docker.com>
2016-02-29 14:15:21 -08:00
Brian Bland
3ae504384c Merge pull request #1473 from miminar/pass-blob-size
Commit uploaded blob with size
2016-02-29 10:26:45 -08:00
Aaron Lehmann
a5bd3e6144 Merge pull request #1474 from miminar/respect-errors
Respect errors returned from middleware code
2016-02-26 23:28:37 -08:00
Aaron Lehmann
770b6728db Merge pull request #1481 from stevvooe/remove-AUTHORS-from-default-make-target
Makefile: remove AUTHORS from default target
2016-02-26 23:26:40 -08:00
Brian Bland
e430d77342 Merge pull request #1482 from stweil/master
Fix some typos in comments and strings
2016-02-23 14:08:21 -08:00
Stefan Weil
615c6dfced Fix some typos in comments and strings
All of them were found and fixed by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-23 22:33:38 +01:00
Aaron Lehmann
7571572305 Merge pull request #1471 from RichardScothern/tag-headers
Enable proxying registries to downgrade fetched manifests to Schema 1.
2016-02-23 11:08:46 -08:00
Michal Minar
21a0878665 Commit blob with known size
Signed-off-by: Michal Minar <miminar@redhat.com>
2016-02-23 08:44:06 +01:00
Richard Scothern
7144936612 Enable proxying registries to downgrade fetched manifests to Schema 1.
Ensure Accept headers are sent with TagService.Get (which hits manifest
endpoints).  Add support for remote Get and Put for the proxied blobstore.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-22 17:56:48 -08:00
Stephen J Day
7230e9def7 Makefile: remove AUTHORS from default target
While it is generally good practice to look before jumping, many
developers enjoy running `make` without reading the Makefile and
indiscriminately use `git add -A`. This behavior, coupled with
generating AUTHORS in the default target result in garbage in various
PRs and extra round trips to remove said garbage.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-02-22 12:56:23 -08:00
Michal Minar
4d15bf071c Defined ErrAccessDenied error
Middleware code may perform additional checks on blobs written. Allow it
to return access denied errors that will result in 403 Forbidden.

Signed-off-by: Michal Minar <miminar@redhat.com>
2016-02-22 21:12:59 +01:00
Richard Scothern
ce05116517 Merge pull request #1477 from xiekeyang/unit-test
compare error output in tagstore unit test
2016-02-22 10:13:16 -08:00
xiekeyang
d58188c4d7 compare error output in tagstore unit test
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2016-02-22 17:34:22 +08:00
Richard Scothern
ad1b181291 Merge pull request #1466 from RichardScothern/proxy-lazy-auth
Lazily evaluate auth challenges
2016-02-19 15:06:35 -08:00
Richard Scothern
6344c99848 Merge pull request #1472 from tanksuzuki/deadlink-to-license-file
Fix deadlink to license file
2016-02-18 10:06:43 -08:00
Asuka Suzuki
b035d372e6 Fix deadlink to license file
Signed-off-by: Asuka Suzuki <hello@tanksuzuki.com>
2016-02-18 17:12:51 +09:00
Richard Scothern
4ce15476bd Extend authChallenger interface to remove type cast.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-17 13:07:14 -08:00
Brian Bland
5806f275bf Merge pull request #1469 from noxiouz/fix_storage_interface_desc
Fix description of StorageDriver.WriteStream
2016-02-17 10:46:11 -08:00
Aaron Lehmann
59a27406cd Merge pull request #1468 from HuKeping/master
Cleanup: remove unused log
2016-02-17 10:00:02 -08:00
Anton Tiurin
a048a4c8d5 Fix description of StorageDriver.WriteStream
Offset can be more than CurrentSize as long as this case is checked
by DriverSuite.testContinueStreamAppend.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2016-02-17 13:57:20 +03:00
HuKeping
5408d4e0a5 Cleanup: remove unused log
Signed-off-by: Hu Keping <hukeping@huawei.com>
2016-02-17 14:11:20 +08:00
Brian Bland
5a1863b76e Merge pull request #1467 from BrianBland/s3awsFixTest
[driver/s3aws] Fix TestStorageClass
2016-02-16 18:14:34 -08:00
Brian Bland
c770f28f04 [driver/s3aws] Fix TestStorageClass
Fixes bug in TestStorageClass for s3aws driver where the "standard" file
was checked for reduced-redundnancy storage.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-16 17:50:55 -08:00
Richard Scothern
56af60ad24 To avoid any network use unless necessary, delay establishing authorization
challenges with the upstream until any proxied data is found not to be local.

Implement auth challenges behind an interface and add to unit tests.  Also,
remove a non-sensical unit test.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-02-16 11:53:42 -08:00
Brian Bland
b9907c4853 Merge pull request #1385 from BrianBland/s3UseAWSLibrary
[driver/s3] Use aws/aws-sdk-go instead of goamz for s3 driver and cloudfront
2016-02-16 11:19:15 -08:00
Richard Scothern
d126752d97 Merge pull request #1462 from liuchang23/master
enhance log message of oss driver
2016-02-16 10:49:18 -08:00
Derek McGowan
98620458e3 Add credential authenticator interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-16 10:23:35 -08:00
liuchang0812
0758653b42 fix gofmt
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:42:09 +08:00
liuchang0812
dbba9cf299 closes #1461, enhance log message of oss driver
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:17:09 +08:00
Derek McGowan
3fc4e4cdc8 Add resource scope list definition
Allow providing multiple scopes in a single scope string

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-12 16:07:21 -08:00
Brian Bland
1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-12 15:59:26 -08:00
Derek McGowan
6d6c37c06a Update client section
Use client_id as defined in oauth rfc instead of custom client field

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-11 13:57:23 -08:00
Derek McGowan
32931689d5 Add scope documentation
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-11 13:57:23 -08:00
Derek McGowan
d8b59ab637 Add specification for using oauth with the token server
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-11 13:57:23 -08:00
Olivier Gambier
bf991fec01 Merge pull request #1455 from RichardScothern/grammar-correct
Reference name rather than repository in reference grammar
2016-02-11 10:40:22 -08:00
Richard Scothern
846328977e Reference name rather than repository in reference grammar
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-02-11 10:26:32 -08:00
Olivier Gambier
7b66c50bb7 Merge pull request #1452 from aaronlehmann/export-no-credentials-error
Export "no basic auth credentials" as an error value
2016-02-11 10:09:22 -08:00
Olivier Gambier
d438e75dcf Merge pull request #1453 from moxiegirl/remove-tutorials
Updating Dockerfile
2016-02-10 22:32:49 -08:00
Mary Anthony
78aa1d2bb1 Updating Dockerfile
Signed-off-by: Mary Anthony <mary@docker.com>
2016-02-10 21:12:45 -08:00
Olivier Gambier
3c80f6fb5e Merge pull request #1451 from aaronlehmann/typo-fixes
Typo fixes in comments
2016-02-10 17:12:00 -08:00
Aaron Lehmann
b046861e40 Export "no basic auth credentials" as an error value
Making this an exported error value will allow users of the
registry/client/auth module to have consistent behavior between
authentication failures and cases where no credentials are provided.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:46:57 -08:00
Aaron Lehmann
d6a9b20971 Merge pull request #1445 from dmcgowan/fix-manifest-digest-header
Fix schema1 manifest etag and docker content digest header
2016-02-10 16:27:18 -08:00
Aaron Lehmann
aa80478b64 Typo fixes in comments
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:26:29 -08:00
Aaron Lehmann
a3213ff331 Merge pull request #1420 from dmcgowan/configurable-trust-key
Add option to disable signatures
2016-02-10 16:15:59 -08:00
Derek McGowan
e8eca4db17 Add documentation for compatibility configuration
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-10 15:20:55 -08:00
Derek McGowan
b34e571bff Add option to disable signatures
Add option for specifying trust key for signing schema1 manifests.
Since schema1 signature key identifiers are not verified anywhere and deprecated, storing signatures is no longer a requirement.
Furthermore in schema2 there is no signature, requiring the registry to already add signatures to generated schema1 manifests.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-10 15:20:39 -08:00
Brian Bland
87ae10b867 Merge pull request #1446 from BrianBland/updateGoamz
[driver/s3] Adds support for ap-northeast-2 region
2016-02-10 14:44:09 -08:00
Richard Scothern
77534e7340 Merge pull request #1443 from igayoso/patch-1
Change certificate hypothetic path
2016-02-10 10:05:49 -08:00
Brian Bland
9259bd7133 [driver/s3] Adds support for ap-northeast-2 region
Updates docker/goamz dependency

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 19:42:02 -08:00
Derek McGowan
350234898a Fix schema1 manifest etag and docker content digest header
When schema2 manifests are rewritten as schema1 currently the etag and docker content digest header keep the value for the schema2 manifest.

Fixes #1444

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-09 18:28:43 -08:00
Richard Scothern
db34a8ba9c Merge pull request #1414 from BrianBland/s3ImproveConfigParsing
Improves flexibility of configuration handling for S3 driver
2016-02-09 18:10:03 -08:00
Brian Bland
52b4a328dd Improves flexibility of configuration handling for S3 driver
Treats nil parameters the same as unprovided parameters (fixes issues
where certain parameters are printed to "<nil>").
Accepts "true" and "false" string values for boolean parameters.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 17:27:16 -08:00
igayoso
a940a07fa9 Change certificate hypothetic path
Signed-off-by: igayoso <igayoso@gmail.com>
2016-02-10 01:03:28 +01:00
Richard Scothern
3f97c258a6 Merge pull request #1404 from RichardScothern/delete-events
Send manifest and blob delete events to the notifications subsystem.
2016-02-08 14:57:04 -08:00
Richard Scothern
f37b2ee16e Send manifest and blob delete events to the notifications subsystem.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-02-08 13:37:35 -08:00
Richard Scothern
fd7ccc0bdf Add documentation about what is sent for delete events
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-08 13:37:35 -08:00
Stephen Day
ee3b7ff76f Merge pull request #1434 from aaronlehmann/uppercase-in-hostname
Allow uppercase characters in hostnames
2016-02-08 10:53:18 -08:00
Aaron Lehmann
950d34d210 Allow uppercase characters in hostnames
This allows hostnames to contain uppercase characters, matching behavior
in Docker versions before 1.10. It does not attempt to canonicalize
hostnames into a lowercase format before parsing, since this could lead
to corner cases (for example, making Hostname.Domain.Com/ref ambiguous
on a daemon which contains references for both hostname.domain.com/ref
and Hostname.Domain.Com/ref).

Fixes: #1433

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-05 18:11:07 -08:00
Richard Scothern
55fbf54da0 Merge pull request #1432 from docker/ceph-build-fix
Attempt to unblock the ceph build process
2016-02-05 15:25:28 -08:00
Richard Scothern
bf1398e514 Attempt to unblock the ceph build process
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-02-05 14:02:01 -08:00
Richard Scothern
dd58349b35 Merge pull request #1419 from aaronlehmann/safer-header-copying
On redirect, only copy headers when they don't already exist in the redirected request
2016-02-05 12:14:01 -08:00
Aaron Lehmann
7761a70389 Add information about manifest content types to API spec
Bring the spec up to date for schema2 changes.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-05 11:27:20 -08:00
Aaron Lehmann
5e4ea38a2f Merge pull request #1425 from RichardScothern/fix-build
Correct type for repo reference
2016-02-04 17:47:03 -08:00
Richard Scothern
4a1d36c22d Correct type for repo reference
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-04 17:32:55 -08:00
Derek McGowan
ab9b433fca Merge pull request #1417 from RichardScothern/token-print
Print the correct token expiration time
2016-02-04 16:57:08 -08:00
Richard Scothern
2ad849704b Merge pull request #1413 from BrianBland/updateGoamz
Updates docker/goamz dependency to support AWS profile env
2016-02-04 16:50:15 -08:00
Richard Scothern
579981b979 Merge pull request #1350 from aibaars/storage-filewriter-pointer
Storage: remove bufferedFileWriter (dead code)
2016-02-04 16:47:52 -08:00
Richard Scothern
2cc6ccbded Merge pull request #1401 from BrianBland/s3StorageClass
Adds "storageclass" configuration parameter for S3 driver.
2016-02-04 16:44:13 -08:00
Richard Scothern
b737de4856 Merge pull request #1408 from aaronlehmann/repository-interface
Rename Name method of Repository to Named
2016-02-04 16:43:35 -08:00
Richard Scothern
7f46e7dce1 Merge pull request #1421 from hopkings2008/master
read the actual number of bytes according to the initial size.
2016-02-04 16:36:30 -08:00
Richard Scothern
11a71c5b88 Merge pull request #1423 from RichardScothern/credit
Update maintainers and authors
2016-02-04 11:10:52 -08:00
Olivier Gambier
0d9cb4f4f2 Merge pull request #1424 from RichardScothern/version-file
Update version file
2016-02-04 11:03:55 -08:00
Richard Scothern
7ba21d8b1d update version file
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-04 10:58:06 -08:00
Richard Scothern
9a0f037436 Update maintainers and authors
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-04 10:40:51 -08:00
Olivier Gambier
5596e787b6 Merge pull request #1409 from RichardScothern/recipe-menu
Move 'Recipes' into the Registry menu
2016-02-04 10:32:39 -08:00
Richard Scothern
97fbaaa5e2 Merge pull request #1422 from moxiegirl/add-docker-to-title
Update title of Registry to Docker Registry
2016-02-04 10:10:03 -08:00
Aaron Lehmann
b0989446eb Rename Name method of Repository to Named
This makes code that gets the name as a string read like
repo.Named().Name() instead of repo.Name().Name().

Requested in
https://github.com/docker/docker/pull/19887#discussion_r51479753

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-04 09:55:36 -08:00
Mary Anthony
8fdd7f5dfe Update title of Registry to Docker Registry
Signed-off-by: Mary Anthony <mary@docker.com>
2016-02-04 08:37:45 -08:00
yuzou
44c0b9ae87 read the actual number of bytes according to the initial size.
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-02-04 16:14:35 +08:00
Aaron Lehmann
4354f0a107 On redirect, only copy headers when they don't already exist in the redirected request
A changeset under consideration for Go 1.7 would automatically copy
headers on redirect. This change future-proofs our code so we won't make
duplicate copies of the headers if net/http does it automatically in the
future.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-03 13:19:44 -08:00
Richard Scothern
f5823ff0be Print the correct token expiration time
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-03 10:42:32 -08:00
Olivier Gambier
c149d45cc8 Merge pull request #1415 from omercnet/patch-1
Update configuration.md
2016-02-03 10:31:23 -08:00
Omer Cohen
5438838be4 fix yml indent
Signed-off-by: Omer Cohen <git@omer.io>
2016-02-03 17:59:49 +02:00
Omer Cohen
d842645de9 Update configuration.md
fixed broken links

Signed-off-by: Omer Cohen <git@omer.io>
2016-02-03 15:57:55 +02:00
Brian Bland
ee4e6e3691 Updates docker/goamz dependency to support AWS profile env
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-02 18:10:03 -08:00
Richard Scothern
bd8b81a686 Make 'Recipes' the last item in the list
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-02 10:18:35 -08:00
Richard Scothern
586111b11e Merge pull request #1407 from RichardScothern/docs
Add compatibility notes about pull by digest across Docker and Registry versions.
2016-02-02 09:44:27 -08:00
Olivier Gambier
830bef975d Merge pull request #1410 from aaronlehmann/failured
Correct ErrAuthenticationFailure message
2016-02-01 19:20:35 -08:00
Richard Scothern
4f7b32bc23 Add compatibility notes about pull by digest across Docker and Registry versions.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-01 17:05:27 -08:00
Aaron Lehmann
e387b63001 Correct ErrAuthenticationFailure message
This was "authentication failured". Change it to "authentication
failure".

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-01 17:03:41 -08:00
Aaron Lehmann
e10efb2ecc Merge pull request #1403 from dmcgowan/auth-const-keys
Update auth context keys to use constant
2016-02-01 16:29:07 -08:00
Aaron Lehmann
d3fb5c7064 Merge pull request #1390 from dmcgowan/token-server
Simple integration test token server
2016-02-01 16:27:49 -08:00
Brian Bland
695bac7b0f Adds test for S3 storage class configuration option
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:55 -08:00
Brian Bland
4688d1adc2 Adds "storageclass" configuration parameter for S3 driver.
Defaults to STANDARD, also supports REDUCED_REDUNDANCY.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:19 -08:00
Aaron Lehmann
2fc586d2a4 Merge pull request #1381 from BrianBland/s3CustomUAString
Adds custom registry User-Agent header to s3 HTTP requests
2016-02-01 15:40:36 -08:00
Richard Scothern
9710599f99 Make 'Recipes' into the Registry menu
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-01 15:22:25 -08:00
Brian Bland
2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 13:43:46 -08:00
Richard Scothern
f2164294f7 Merge pull request #1398 from moxiegirl/new-navigation-1.10
New navigation for 1.10
2016-02-01 11:31:40 -08:00
Mary Anthony
01bb31fc42 New navigation for 1.10
Capturing Olivier's comments

Signed-off-by: Mary Anthony <mary@docker.com>
2016-01-28 17:35:54 -08:00
Derek McGowan
648a1343db Update auth context keys to use constant
Prevent using strings throughout the code to reference a string key defined in the auth package.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-28 17:02:09 -08:00
Derek McGowan
e28c288444 Update to address comments
Add logging to resolve scope
Clarify response logs
Better messaging for tls setup error

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-28 15:47:22 -08:00
Aaron Lehmann
74c4a0e3e3 Merge pull request #1393 from RichardScothern/putbydgst
Add manifest put by digest to the registry client
2016-01-28 14:41:07 -08:00
Derek McGowan
70e85dde26 Merge pull request #1392 from aaronlehmann/client-range-requests
Support range requests in the client's httpReadSeeker
2016-01-28 13:44:43 -08:00
Richard Scothern
d793822290 Merge pull request #1394 from RichardScothern/invalidate-bdc
Invalidate the blob store descriptor cache
2016-01-28 10:04:09 -08:00
Aaron Lehmann
8fd84c4a57 Add a CheckRedirect function to the HTTP client
Use it to preserve Accept and Range headers that were added to the
original request.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-27 16:11:36 -08:00
Aaron Lehmann
ccf8154a44 Support range requests in the client's httpReadSeeker
Remove buffering on the reader, because it's not useful. Also remove
artificial io.EOF return.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-27 16:11:36 -08:00
Richard Scothern
a8861549cf Invalidate the blob store descriptor caches when content is removed from from
the proxy.  Also, switch to reference in the scheduler API.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:59 -08:00
Richard Scothern
584c9b517c Correct test digest lengths and enable all unit tests
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:46 -08:00
Richard Scothern
9f72e8442d Add manifest put by digest to the registry client
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 14:20:23 -08:00
Derek McGowan
08d1f035f0 Update create token to auth/token types
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-25 20:12:07 -08:00
Derek McGowan
fd17443988 Update token header struct to use json.RawMessage pointer
Since RawMessage json receivers take a pointer type, the Header structure should use points in order to call the json.RawMessage marshal and unmarshal functions

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-25 20:11:41 -08:00
Derek McGowan
eaa9da0be3 Add simple implementation of token server
Token server implementation currently functional with existing docker 1.9.x release and latest distribution release.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-25 15:42:05 -08:00
Arthur Baars
3d67dae525 Storage: remove bufferedFileWriter (dead code)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:08 +00:00
Arthur Baars
26e7268a86 Storage: blobwriter.Write/Seek test case
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:07 +00:00
Richard Scothern
69db5b7440 Merge pull request #1333 from aaronlehmann/use-reference-package
Use reference package
2016-01-22 15:54:19 -08:00
Aaron Lehmann
2b20b0167a Change URLBuilder methods to use references for tags and digests
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:49:29 -08:00
Aaron Lehmann
4441333912 Use reference package internally
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:47:05 -08:00
Olivier Gambier
caa2001e1f Merge pull request #1341 from moxiegirl/fix-issue-1254
Pulling driver specs out of configuruation.md
2016-01-21 16:57:23 -08:00
Richard Scothern
c301f8ab27 Merge pull request #1383 from aaronlehmann/default-manifest-type
If the media type for a manifest is unrecognized, default to schema1
2016-01-21 10:59:19 -08:00
Richard Scothern
70f4dac7f7 Merge pull request #1376 from aaronlehmann/use-mime-pkg
Use mime package to parse media type
2016-01-21 10:58:53 -08:00
Mary Anthony
cab284d6b2 - Pulling driver specs out of configuruation.md
- Fixing index metadata error
- Entering all the comments
- Updating with the build fixes
- Fix Microsoft link broken
- Fix collocate, colocate, to unambiguous co-locate

Signed-off-by: Mary Anthony <mary@docker.com>
2016-01-21 10:12:46 -08:00
Richard Scothern
1df70eb1da Merge pull request #1377 from RichardScothern/correct-event-digest
Ensure the canonical manifest digest and length are sent to notification endpoints.
2016-01-21 09:34:56 -08:00
Aaron Lehmann
c01fe47231 If the media type for a manifest is unrecognized, default to schema1
This is needed for compatibility with some third-party registries that
send an inappropriate Content-Type header such as text/html.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-21 09:34:06 -08:00
Stephen Day
08650825fe Merge pull request #1380 from aaronlehmann/avoid-empty-accept-header
Avoid empty Accept headers in client requests
2016-01-20 15:38:07 -08:00
Aaron Lehmann
acf02bead3 Avoid empty Accept headers in client requests
One of the keys in the manifest media type map is an empty string. This
should not be sent as an Accept header.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-20 15:22:38 -08:00
Stephen Day
182bddcf11 Merge pull request #1379 from aaronlehmann/token-error
Handle nonstandard token endpoint errors
2016-01-20 15:07:47 -08:00
Aaron Lehmann
ec636bbfd2 Handle nonstandard token endpoint errors
https://github.com/docker/distribution/pull/1249 changed token fetching
to parse HTTP error response bodies as serialized errcodes. However,
Docker Hub's authentication endpoint does not return error bodies in
this format. To work around this, convert its format into
ErrCodeUnauthorized or ErrCodeUnknown.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-20 14:45:08 -08:00
Richard Scothern
dd0d5a31f5 Merge pull request #1378 from stevvooe/update-specification-changelog
spec/api: add missing entries to changelog
2016-01-20 13:09:44 -08:00
Richard Scothern
fdd2e4d77e Merge pull request #1372 from aibaars/gcs-delete
StorageDriver GCS: improve test suite clean-up and add retrying to all GCS api calls
2016-01-20 12:21:22 -08:00
Stephen J Day
eb87f00afd spec/api: add missing entries to changelog
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-01-20 12:19:25 -08:00
Richard Scothern
4f822100f7 Ensure the canonical manifest digest and length are sent to notification endpoints.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-20 11:21:04 -08:00
Aaron Lehmann
93d019658f Use mime package to parse media type
This replaces custom parsing with the standard library's mime package.
This is simpler and more correct.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-20 09:59:19 -08:00
Arthur Baars
2a4345ca4b StorageDriver: GCS: retry all api calls
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Arthur Baars
6ee339464c StorageDriver: Test suite: improve cleanup
Verify that the file(s) have been deleted after calling Delete,
and retry if this is not the case. Furthermore, report the error
if a Delete operation fails.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Richard Scothern
47a064d419 Merge pull request #1367 from aaronlehmann/signed-manifest-content-type
Fix content type for schema1 signed manifests
2016-01-19 11:00:47 -08:00
Aaron Lehmann
8c1a000799 Fix content type for schema1 signed manifests
The Payload function for schema1 currently returns a signed manifest,
but indicates the content type is that of a manifest that isn't signed.

Note that this breaks compatibility with Registry 2.3 alpha 1 and
Docker 1.10-rc1, because they use the incorrect content type.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-18 16:08:02 -08:00
Richard Scothern
3cb403ae5b Merge pull request #1363 from aaronlehmann/media-type-charset
Do not require "charset=utf-8" for a schema1 with content type application/json
2016-01-18 15:20:01 -08:00
Richard Scothern
fc7ee6d789 Merge pull request #1355 from hopkings2008/master
In testsuites.go, enlarge the size of randomBytes to 128M to fix the …
2016-01-18 12:38:26 -08:00
Richard Scothern
775c5fbeba Merge pull request #1354 from RichardScothern/Boran-patch2
Carry #1329 : Fixes so that nginx will start
2016-01-18 11:58:35 -08:00
Sean Boran
07e68b3b04 Carry #1329 : Fixes so that nginx will start
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>

Signed-off-by: Sean Boran <Boran@users.noreply.github.com>
2016-01-18 11:46:39 -08:00
Richard Scothern
a1af43d2e0 Merge pull request #1357 from aibaars/gcs-test
StorageDriver GCS test suite: try google.DefaultTokenSource first
2016-01-18 11:29:01 -08:00
Aaron Lehmann
5a2664e0b1 Do not require "charset=utf-8" for a schema1 with content type application/json
For compatibility with other registries that don't use this exact
variant of the Content-Type header, we need to be more flexible about
what we accept. Any form of "application/json" should be allowed. The
charset should not be included in the comparison.

See docker/docker#19400.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-18 09:59:50 -08:00
Arthur Baars
19d7c9e051 StorageDriver GCS: try google.DefaultTokenSource first
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-15 11:47:47 +00:00
yuzou
cc82b0d48e In testsuites.go, enlarge the size of randomBytes to 128M to fix the crash of running TestConcurrentStreamReads
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-01-15 17:22:43 +08:00
Richard Scothern
7378e21678 Merge pull request #1332 from RichardScothern/gcs-params
Change the parameters to the GCS drivers to allow CircleCI testing.
2016-01-14 15:50:09 -08:00
Richard Scothern
1cdc683358 Merge pull request #1352 from tpounds/use-go-1.5.3
Use Go 1.5.3
2016-01-14 11:17:54 -08:00
Derek McGowan
cb08de17d7 Merge pull request #1351 from BrianBland/splitBlobCreateOptions
Splits up blob create options definitions to be package-specific
2016-01-14 11:10:54 -08:00
Brian Bland
8c5a6c13c0 Splits up blob create options definitions to be package-specific
Redefines privately in both storage and client packages

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-14 10:47:33 -08:00
Trevor Pounds
e5ac03f945 Update to Go 1.5.3.
Signed-off-by: Trevor Pounds <trevor.pounds@gmail.com>
2016-01-14 10:40:20 -08:00
Richard Scothern
df6cdb7bb6 Merge pull request #1347 from aaronlehmann/go-153
Upgrade CI to go 1.5.3
2016-01-14 10:31:45 -08:00
Brian Bland
2a9849e552 Merge pull request #1348 from BrianBland/fixCrossRepoBlobMount
Fixes cross-repo blob mounting in the BlobUploadHandler
2016-01-14 09:44:30 -08:00
Brian Bland
3ef917c269 Fixes cross-repo blob mounting in the BlobUploadHandler
Accidentally checked for err != nil instead of err == nil :/
Also now ensures that only a non-nil option is appended to the create
options slice

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-13 19:21:45 -08:00
Aaron Lehmann
050fcb7dbb Upgrade CI to go 1.5.3
Go 1.5.0 has some stack pointer bugs. This may have been causing some CI
failures. Upgrade to a newer version.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-13 18:25:20 -08:00
Richard Scothern
81dbebc7d3 Change the parameters to the GCS drivers to allow CircleCI testing.
Remove the requirement of file system access to run GCS unit tests.  Deconstruct
the input parameters to take the private key and email which can be specified on
the build system via environment variables.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-13 18:00:59 -08:00
Richard Scothern
5120357906 Merge pull request #1344 from BrianBland/blobCreateWithOptions
Adds functional options arguments to the Blobs Create method, remove Mount operation
2016-01-13 17:35:49 -08:00
Brian Bland
ce88d8a6f4 Adds functional options arguments to the Blobs Create method
Removes the Mount operation and instead implements this behavior as part
of Create a From option is provided, which in turn returns a rich
ErrBlobMounted indicating that a blob upload session was not initiated,
but instead the blob was mounted from another repository

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-13 16:42:59 -08:00
Richard Scothern
4378e1aa85 Merge pull request #1345 from BrianBland/fixErrorCodesInAPITemplate
Fixes error codes in the API spec template
2016-01-13 14:55:59 -08:00
Brian Bland
ac03070245 Fixes error codes in the API spec template
Uses UPPER_UNDERSCORE_CASE instead of the nice error message format
added in #911

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-13 11:53:40 -08:00
Richard Scothern
39ff320b82 Merge pull request #1335 from RichardScothern/fix-build
Fix manifest API unit tests
2016-01-11 14:17:36 -08:00
Richard Scothern
3ea0d70896 Fix manifest API unit tests
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-11 12:52:21 -08:00
Richard Scothern
3a1220de01 Merge pull request #1319 from RichardScothern/update-tags
Remove tags referencing deleted manifests.
2016-01-11 11:33:22 -08:00
Richard Scothern
93d9070c8b Merge pull request #1269 from BrianBland/crossRepositoryPush
Adds cross-repository blob mounting behavior
2016-01-08 14:37:00 -08:00
Brian Bland
613cfc861d Fires a new Mount event when blobs are cross-repo mounted
Adds an optional "fromRepository" field to the event target

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:30 -08:00
Brian Bland
3a35a2d953 Allows token authentication handler to request additional scopes
When an auth request provides the "from" query parameter, the token
handler will add a "pull" scope for the provided repository, refreshing
the token if the overall scope has increased

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:23 -08:00
Brian Bland
5df21570a7 Adds cross-repository blob mounting behavior
Extends blob upload POST endpoint to support mount and from query
parameters as described in #634

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:18 -08:00
Richard Scothern
cc0b6b86ea Merge pull request #1276 from denverdino/oss-test
Support large layer for OSS driver
2016-01-08 13:26:09 -08:00
Stephen Day
a7ae88da45 Merge pull request #1281 from aaronlehmann/new-manifest
Implement schema2 manifest formats
2016-01-07 17:19:56 -08:00
Stephen Day
5ebf6391a7 Merge pull request #911 from stevvooe/consistent-error-string
More consistent return from ErrorCode.Error()
2016-01-07 17:06:12 -08:00
Aaron Lehmann
6d17423a6d Move MediaType into manifest.Versioned
This makes content type sniffing cleaner. The document just needs to be
decoded into a manifest.Versioned structure. It's no longer a two-step
process.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann
697af09566 Recognize clients that don't support manifest lists
Convert a default platform's manifest to schema1 on the fly.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann
9c416f0e94 Add support for manifest list ("fat manifest")
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann
9284810356 Add API unit testing for schema2 manifest
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00
Aaron Lehmann
3f746a8207 Recognize clients that don't support schema2, and convert manifests to schema1 on the fly
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00
Aaron Lehmann
befd4d6e3c Factor out schema-specific portions of manifestStore
Create signedManifestHandler and schema2ManifestHandler. Use these to
unmarshal and put the respective types of manifests from manifestStore.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00
Aaron Lehmann
2ff77c00ba Add schema2 manifest support
Add schema2 manifest implementation.

Add a schema2 builder that creates a schema2 manifest from descriptors
and a configuration. It will add the configuration to the blob store if
necessary.

Rename the original schema1 manifest builder to ReferenceBuilder, and
create a ConfigBuilder variant that can build a schema1 manifest from an
image configuration and set of descriptors. This will be used to
translate schema2 manifests to the schema1 format for backward
compatibliity, by adding the descriptors from the existing schema2
manifest to the schema1 builder. It will also be used by engine-side
push code to create schema1 manifests from the new-style image
configration, when necessary to push a schema1 manifest.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00
Stephen Day
0be4b3f27d Merge pull request #1283 from djenriquez/master
CloudFront-s3-middleware
2016-01-07 14:58:35 -08:00
Aaron Lehmann
83ccbd18c0 Merge pull request #1324 from stevvooe/digest-from-bytes
digest: add NewDigestFromBytes for completeness
2016-01-08 01:04:27 +07:00
Stephen J Day
9f5ebd0a94 digest: add NewDigestFromBytes for completeness
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-01-06 18:00:20 -08:00
Richard Scothern
9f83bfc071 Merge pull request #1321 from aibaars/gcs-fix-retry
GCS storage driver: fix retry function
2016-01-06 12:00:12 -08:00
Richard Scothern
0acca63b93 Merge pull request #1323 from aibaars/gcs-fix-list-empty
GCS Storagedriver: fix test failure caused by #1187
2016-01-06 11:18:34 -08:00
Arthur Baars
bb1e9e0a82 GCS Storagedriver: fix test failure caused by #1187
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 18:36:28 +00:00
Arthur Baars
d3d40a8807 GCS driver: fix retry function
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 17:57:22 +00:00
DJ Enriquez
b49c4fff95 Applying recommended changes
- Adding clarification to the “chunksize” definition
- Changing short-hand “/w” to “with”

Signed-off-by: DJ Enriquez <dj.enriquez@infospace.com>
2016-01-05 15:22:35 -08:00
Richard Scothern
94347c8611 Remove tags referencing deleted manifests.
When a manifest is deleted by digest, look up the referenced tags in the tag
store and remove all associations.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-05 13:47:52 -08:00
Stephen Day
74d719d83f Merge pull request #1249 from k4leung4/print-error-msg
Print error for failed HTTP auth request.
2016-01-05 12:23:05 -08:00
Stephen Day
f277c660ae Merge pull request #1316 from dmcgowan/fix-janky-integration-tests
Fix broken daemon startup with master
2016-01-04 17:51:51 -08:00
Derek McGowan
8d1c44f148 Fix broken daemon startup with master
Since the daemon flag was deprecated and replaced by the daemon subcommand, the run engine should use the subcommand and only the flag for older versions

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-04 16:32:12 -08:00
Sven Dowideit
a580dae06f Merge pull request #1310 from thaJeztah/fix-hugo-linting
docs: mark manifest as "draft"
2016-01-04 18:44:11 +10:00
Stephen Day
0e73b8a37f Merge pull request #1097 from hopkings2008/master
use the scheme and host from x-forward-proto and x-forward-host if the…
2015-12-30 16:35:20 -08:00
Sebastiaan van Stijn
ef006ca1a1 docs: mark manifest as "draft"
Markdown linter produced an error on this page;

    running markdownlint
    ERROR (registry/spec/manifest-v2-2.md) frontmatter: Unexpected non-whitespace char: # Image Manifest Version 2, Schema 2

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2015-12-30 20:56:02 +01:00
Stephen Day
00b983b837 Merge pull request #1304 from BrianBland/fsDriverRelaxedPermissions
Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files)
2015-12-29 19:06:45 -08:00
Stephen Day
e63ad1e3d6 Merge pull request #1306 from stevvooe/from-bytes-algorithm
digest: make FromBytes available on digest.Algorithm
2015-12-29 18:59:47 -08:00
Stephen Day
0404e5a622 Merge pull request #1307 from stevvooe/panic-on-unavailable
digest: panic on unavailable hash algorithm
2015-12-29 18:47:02 -08:00
Stephen Day
2259d9c09b Merge pull request #1302 from liusdu/arm64
update panicwrap package to fix building error on aarch64 platform
2015-12-29 18:44:36 -08:00
Kenny Leung
7ff5042c1e Merge branch 'print-error-msg' of https://github.com/k4leung4/distribution into print-error-msg
Changed to use typed error instead of formatted string.
Added tests for new public method.

Signed-off-by: Kenny Leung <kleung@google.com>
2015-12-29 16:26:13 -08:00
Stephen J Day
95bee1895a digest: panic on unavailable hash algorithm
After running into a few nil pointer errors during development, it is clear
that having this function return nil when a hash is not available is the wrong
approach. Nearly every time, this lack of availability was due to a missing
import statement for the hash. This is always a programming error.

To avoid future confusion, we now appropriately panic when the hash function is
not imported by the application. More dynamic uses of the package should call
Algorithm.Available() before calling Algorithm.Hash() to avoid this panic.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-29 15:24:32 -08:00
Stephen J Day
4646282448 digest: make FromBytes available on digest.Algorithm
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-29 15:16:56 -08:00
Brian Bland
05d1abbcb4 Merge pull request #1303 from BrianBland/fixDisabledStorageRedirects
Serve blobs when a storage driver supports redirects but are disabled
2015-12-29 13:28:45 -08:00
Kenny Leung
c28278f7a2 Print error for failed HTTP auth request.
Signed-off-by: Kenny Leung <kleung@google.com>
2015-12-29 11:39:20 -08:00
Liu Hua
5b550f919f update panicwrap package to fix building error on aarch64 platform
Linux on aarch64 lacks of dup2, uses dup3 instead

Signed-off-by: Liu Hua <sdu.liu@huawei.com>
2015-12-29 19:11:59 +00:00
weiyuan.yl
75abeeca11 Replace 404 to http.StatusNotFound
Change-Id: Ia100975cb93c0a6d94ea5542b1c9ce386bc87649
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-29 12:09:04 +08:00
Brian Bland
f19deba809 Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files)
Leaves any further permissions restrictions to the process umask

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-28 15:22:28 -08:00
Brian Bland
cf487a7911 Serve blobs when a storage driver supports redirects but are disabled
Fixes issue where an error was returned instead of serving the blob

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-28 11:08:39 -08:00
yuzou
d724bb1d24 use the scheme and host from x-forward-proto and x-forward-host if they exits and correct the scheme for Location header during image upload
Signed-off-by: yuzou <zouyu7@huawei.com>
2015-12-28 17:28:32 +08:00
weiyuan.yl
3e4b554f8d In HEAD request for missing resource, only 404 NOT FOUND is returned
Change-Id: I73caf67b59519e6f4f82f7d78f5d4fd4ad9affcd
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-28 11:28:42 +08:00
weiyuan.yl
8c97a853d8 Update OSS client for more details in the error message
Change-Id: Ie2aa70dfeb69930ee5eda17d7e744e3f95a8b926
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-24 21:02:35 +08:00
DJ Enriquez
5dfc78997d Adding suggested changes
Adding changed suggested by moxiegirl.

Signed-off-by: DJ Enriquez <dj.enriquez@infospace.com>
2015-12-22 11:33:28 -09:00
Olivier Gambier
e02a0b0399 Merge pull request #1244 from carsonoid/carsonoid-nginx-doc-patch
Update nginx.md
2015-12-22 10:25:10 -08:00
Olivier Gambier
236c19fa30 Merge pull request #1255 from donhcd/convergeSwiftDocs
Help converge disagreeing swift docs
2015-12-22 10:23:13 -08:00
Olivier Gambier
29213af431 Merge pull request #1294 from rhcarvalho/patch-1
Fix godoc
2015-12-22 10:15:51 -08:00
Rodolfo Carvalho
e0fe14618f Fix godoc
`RepositoryNameTotalLengthMax` doesn't exist anymore.

Signed-off-by: Rodolfo Carvalho <rhcarvalho@gmail.com>
2015-12-22 12:26:05 +01:00
Olivier Gambier
9604f0843d Merge pull request #1290 from BrianBland/crossPackageCoverage
Enables cross-package code coverage reporting
2015-12-21 17:49:45 -08:00
Brian Bland
aaab336edc Enables cross-package code coverage reporting
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-21 17:09:38 -08:00
Olivier Gambier
c56d49b111 Merge pull request #1289 from BrianBland/codecov
Adds codecov coverage reporting to circle.yml
2015-12-21 11:41:12 -08:00
Brian Bland
dd0d28d541 Adds codecov coverage reporting to circle.yml
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-21 10:55:49 -08:00
Olivier Gambier
6b3908878f Merge pull request #1286 from thaJeztah/update-links
Update links to documentation
2015-12-20 21:20:49 -08:00
Sebastiaan van Stijn
bacfea16bb Update links to documentation
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2015-12-21 01:22:01 +01:00
Donald Huang
eb654589db help converge disagreeing swift docs
Signed-off-by: Donald Huang <don.hcd@gmail.com>
2015-12-20 23:22:19 +00:00
Stephen Day
ed3346eeb0 Merge pull request #1068 from aaronlehmann/manifest-proposal
docs/spec: Proposal for new manifest format
2015-12-18 16:25:14 -08:00
DJ Enriquez
8108539037 Relocating documentation to s3 storage-driver doc
Since the CloudFront middleware does not work without an S3 backend, it
became obvious that this documentation should exist within the S3
storage-driver documentation.

Signed-off-by: DJ Enriquez <dj.enriquez@infospace.com>
2015-12-18 12:48:13 -08:00
Aaron Lehmann
2e3f4934a7 docs/spec: Proposal for new manifest format
This is a follow-on to PR #62, and it borrows much of the format
from #993, but uses specific formats for the image manifest and manifest
list (fat manifest) instead of a combined generic format.

The intent of this proposed manifest format is to allow multi-arch, and
allow for full content-addressability of images in the Docker engine.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-18 12:26:02 -08:00
DJ Enriquez
4a9f6c7975 Applying recommended changes
- Corrected headings
- Added blank line after headings
- Changed example snippet to use `...`

Signed-off-by: DJ Enriquez <dj.enriquez@infospace.com>
2015-12-18 11:56:48 -08:00
DJ Enriquez
506905fd90 CloudFront-s3-middleware
Adding a more detailed document regarding how to use CloudFront as
middleware for an s3 backed registry.

Signed-off-by: DJ Enriquez <dj.enriquez@infospace.com>
2015-12-17 22:30:10 -08:00
Richard Scothern
67d3675d55 Merge pull request #1268 from RichardScothern/manifest-refactor-impl
Implementation of the Manifest Service API refactor.
2015-12-17 17:32:55 -08:00
Richard Scothern
cb6f002350 Implementation of the Manifest Service API refactor.
Add a generic Manifest interface to represent manifests in the registry and
remove references to schema specific manifests.

Add a ManifestBuilder to construct Manifest objects. Concrete manifest builders
will exist for each manifest type and implementations will contain manifest
specific data used to build a manifest.

Remove Signatures() from Repository interface.

Signatures are relevant only to schema1 manifests.  Move access to the signature
store inside the schema1 manifestStore.  Add some API tests to verify
signature roundtripping.

schema1
-------

Change the way data is stored in schema1.Manifest to enable Payload() to be used
to return complete Manifest JSON from the HTTP handler without knowledge of the
schema1 protocol.

tags
----

Move tag functionality to a seperate TagService and update ManifestService
to use the new interfaces.  Implement a driver based tagService to be backward
compatible with the current tag service.

Add a proxyTagService to enable the registry to get a digest for remote manifests
from a tag.

manifest store
--------------

Remove revision store and move all signing functionality into the signed manifeststore.

manifest registration
---------------------

Add a mechanism to register manifest media types and to allow different manifest
types to be Unmarshalled correctly.

client
------

Add ManifestServiceOptions to client functions to allow tags to be passed into Put and
Get for building correct registry URLs.  Change functional arguments to be an interface type
to allow passing data without mutating shared state.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2015-12-17 17:09:14 -08:00
Stephen Day
7f41cd7587 Merge pull request #1257 from aaronlehmann/remove-extra-stat
Remove unnecessary stat from blob Get method
2015-12-17 14:10:12 -08:00
Stephen Day
9162760443 Merge pull request #1271 from aaronlehmann/remove-tarsum
Remove tarsum support for digest package
2015-12-16 14:24:11 -08:00
Aaron Lehmann
4c850e7165 Remove tarsum support for digest package
tarsum is not actually used by the registry. Remove support for it.

Convert numerous uses in unit tests to SHA256.

Update docs to remove mentions of tarsums (which were often inaccurate).

Remove tarsum dependency.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-15 17:22:18 -08:00
Stephen Day
200cbe8b8e Merge pull request #1264 from aaronlehmann/hash-errors
Simplify digest.FromBytes calling convention
2015-12-15 16:50:50 -08:00
Richard Scothern
9a06c58d70 Merge pull request #1258 from thaJeztah/fix-docs-headers
Fix headers in documentation
2015-12-15 14:55:00 -08:00
Olivier Gambier
bf09bb1314 Merge pull request #1267 from gabriellhrn/typo
Fix typo in docs/configuration.md
2015-12-15 10:22:07 -08:00
gabriell nascimento
74255cd4f2 Fix typo in docs/configuration.md
Signed-off-by: Gabriell Nascimento <gabriellhrn@gmail.com>
2015-12-15 16:14:13 -02:00
Aaron Lehmann
31047c8113 Simplify digest.FromBytes calling convention
The current implementation of digest.FromBytes returns an error. This
error can never be non-nil, but its presence in the function signature
means each call site needs error handling code for an error that is
always nil.

I verified that none of the hash.Hash implementations in the standard
library can return an error on Write. Nor can any of the hash.Hash
implementations vendored in distribution.

This commit changes digest.FromBytes not to return an error. If Write
returns an error, it will panic, but as discussed above, this should
never happen.

This commit also avoids using a bytes.Reader to feed data into the hash
function in FromBytes. This makes the hypothetical case that would panic
a bit more explicit, and should also be more performant.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-14 14:30:51 -08:00
Stephen Day
23e4099973 Merge pull request #1245 from carsonoid/carsonoid-nginx-doc-patch-2
Fix incorrectly interpolated config setting
2015-12-14 12:19:43 -08:00
Sebastiaan van Stijn
129c9a22f7 Fix headers in documentation
These headers were rendered as body text
because there was no space, but a "tab" after
the `#`.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2015-12-12 01:39:51 +01:00
Aaron Lehmann
be1f4114c7 Remove unnecessary stat from blob Get method
This calls Stat before Open, which should be unnecessary because Open
can handle the case of a nonexistent blob. Removing the Stat saves a
round trip.

This is similar to the removal of stat in Open in #1226.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-11 15:13:03 -08:00
Li Yi
54da47d636 Support large layer for OSS driver
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-11 10:56:50 +08:00
Richard Scothern
e6c60e79c5 Merge pull request #1253 from RichardScothern/go-1.5.2
Update go version to 1.5.2
2015-12-09 11:52:51 -08:00
Richard Scothern
a67000a814 Update go version to 1.5.2
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-12-09 11:33:18 -08:00
Richard Scothern
afa4448f38 Merge pull request #1250 from RichardScothern/AUTHORS
Update AUTHORS for release
2015-12-09 11:18:37 -08:00
Richard Scothern
5ffa6564b8 Update AUTHORS and .mailmap for release
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-12-09 10:16:50 -08:00
Richard Scothern
796d6e7915 Merge pull request #1187 from stevvooe/check-storage-drivers-list-path-not-found
[WIP] registry/storage/driver: checking that non-existent path returns PathNotFoundError
2015-12-08 16:32:02 -08:00
Stephen J Day
66cd2bf950 storage/driver/s3: adjust s3 driver to return unmunged path
This fixes both the s3 driver and the oss driver to return the unmunged path
when returning errors.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 15:48:55 -08:00
Stephen Day
e4208ad5a7 Merge pull request #1246 from stevvooe/decrease-driver-testsuite-allocations
storage/driver: decrease memory allocation done during testsuite
2015-12-08 14:52:59 -08:00
Kenny Leung
e128a821db Print error for failed HTTP auth request.
Signed-off-by: Kenny Leung <kleung@google.com>
2015-12-08 14:24:03 -08:00
Stephen Day
7d9b2774f0 Merge pull request #1237 from thaJeztah/update-maintainers
update maintainers file for parsing
2015-12-08 13:35:34 -08:00
Li Yi
b6756a3d89 Fix the issue for listing root directory
Change-Id: I1c6181fa4e5666bd2e6ec69cb608c4778ae0fe48
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
Li Yi
7161fa0559 Fix for stevvooe:check-storage-drivers-list-path-not-found in OSS driver
Change-Id: I5e96fe761d3833c962084fd2d597f47e8a72e7c2
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
davidli
79d4d7f546 driver/swift: treat empty object list as a PathNotFoundError
Swift returns an empty object list when trying to read a non-existing object path, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: David li <wenquan.li@hpe.com>
2015-12-08 12:04:04 -08:00
Vincent Giersch
f25ccea279 driver/rados: treat OMAP EIO as a PathNotFoundError
RADOS returns a -EIO when trying to read a non-existing OMAP, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
2015-12-08 12:04:04 -08:00
Stephen J Day
a889f46223 storage/driver/s3: correct response on list of missing directory
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
b45078eb44 storage/driver/base: use correct error format style
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
a4eae0917e driver/filesystem: address filesystem driver on behavior of List
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day
0322c3bf1f registry/storage/driver: checking that non-existent path returns PathNotFoundError
Issue #1186 describes a condition where a null tags response is returned when
using the s3 driver. The issue seems to be related to a missing
PathNotFoundError in s3. This change adds a test for that to get an idea of the
lack of compliance across storage drivers. If the failures are manageable,
we'll add this test condition and fix the s3 driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen Day
807a80ff06 Merge pull request #1238 from noxiouz/fix_comment
Fix comment for PathRegexp
2015-12-08 10:41:16 -08:00
Stephen Day
ce17efc71e Merge pull request #1241 from pdevine/catalog-speedup
Make the catalog more efficient
2015-12-08 10:20:51 -08:00
Richard Scothern
95e80a8283 Merge pull request #1243 from RichardScothern/client-header-message
Add clearer messaging around missing content-length headers.
2015-12-08 09:54:18 -08:00
Stephen J Day
9dc3c6df0e storage/driver: decrease memory allocation done during testsuite
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-07 19:05:13 -08:00
Carson A
2c2fbc4c24 Update nginx.md
Fixes a huge problem where a nginx variable was not escaped and nginx was adding the literal string "always" to all requests.
The most obvious result was that basic auth was entirely broken.

Signed-off-by: Carson Anderson <ca@carson-anderson.com>
2015-12-07 15:47:51 -07:00
Patrick Devine
1027f5e066 Make the catalog more efficient
This change removes the sort() from the Repositories() function since
we're now guaranteed to have a lexigraphically sorted walk.

Signed-off-by: Patrick Devine <patrick.devine@docker.com>
2015-12-07 12:33:01 -08:00
Carson A
de3263c9d6 Update nginx.md
Split commands out from their lead-ins.

Signed-off-by: Carson Anderson <ca@carson-anderson.com>
2015-12-07 12:32:03 -07:00
Richard Scothern
74bb6f89c8 Merge pull request #1157 from lebauce/swift-bulk-delete-fixes
Use bulk delete to remove segments in Swift driver
2015-12-07 10:56:32 -08:00
Stephen Day
7a584f21b8 Merge pull request #1239 from stevvooe/refactor-reference-grammar
reference: refactor grammar and regular expressions
2015-12-07 10:31:43 -08:00
Richard Scothern
f18bf5e343 Add clearer messaging around missing content-length headers.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-12-07 10:17:49 -08:00
Stephen J Day
3150937b9f reference: refactor grammar and regular expressions
To resolve some inconsistencies between the grammar and the regular
expressions, the grammar has been slightly refactored to match the intent. The
regular expressions have been redefined with the same structure to make it
easier to verify the grammar is correct.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-04 17:54:23 -08:00
Sebastiaan van Stijn
434aaf7e4e update maintainers file for parsing
this updates the MAINTAINERS file to the new format,
so that it can be parsed and collected in the docker/opensource
repository.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2015-12-05 01:07:56 +01:00
Sven Dowideit
226d8c4f07 Merge pull request #1234 from SvenDowideit/linkchecker-fixes
Small doc fixes found by the linkchecker
2015-12-05 09:30:58 +10:00
Anton Tiurin
d4435b79d9 Fix comment for PathRegexp
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-12-04 22:14:21 +03:00
Sylvain Baubeau
ef4ab7a885 Use bulk delete to remove segments in Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-12-04 12:16:49 +01:00
Stephen Day
568bf038af Merge pull request #1233 from kendru/patch-1
Escape dollar signs
2015-12-03 12:29:12 -08:00
Stephen Day
4cf93714f3 Merge pull request #1231 from tonistiigi/digest-length-check
Validate digest length on parsing
2015-12-03 12:17:16 -08:00
Stephen Day
d46b6a8796 Merge pull request #1211 from tt/remove-name-verification
Remove name verification
2015-12-03 11:59:21 -08:00
Richard Scothern
eef3382222 Merge pull request #1232 from pdevine/catalog-clarification
Clarify pagination for catalog API endpoint
2015-12-03 09:14:23 -08:00
Sven Dowideit
7d0d94814b Small doc fixes found by the linkchecker
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2015-12-03 21:26:41 +10:00
Andrew Meredith
6357b02236 Escape dollar signs
If this example was copied and pasted, the shell would try to interpolate `$upstream_http_docker_distribution_api_version` and `$docker_distribution_api_version`.

Signed-off-by: Andrew Meredith <andymeredith@gmail.com>
2015-12-02 23:45:09 -07:00
Stephen Day
b66bb1287c Merge pull request #1227 from stevvooe/walk-sorted
storage: enforce sorted traversal during Walk
2015-12-02 18:37:44 -08:00
Patrick Devine
b084c77c70 Clarify pagination for catalog API endpoint
This change clarifies the way the catalog endpoint returns results
when pagination was not explicitly requested.

Signed-off-by: Patrick Devine <patrick.devine@docker.com>
2015-12-02 18:02:51 -08:00
Stephen Day
2b63f65543 Merge pull request #1226 from aaronlehmann/avoid-stat-roundtrips
Avoid stat round-trips when fetching a blob
2015-12-02 16:09:29 -08:00
Tonis Tiigi
f015982f0f Validate digest length on parsing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2015-12-02 16:04:06 -08:00
Aaron Lehmann
6beeb935cd Avoid stat round-trips when fetching a blob
Without this commit, three round-trips are required to fetch a blob with
a progress bar. The first is a call to Stat (HEAD request), to determine
the size. Then Open is called, which also calls Stat, and finally
performs a GET request.

Only the GET request is actually needed. The size of the blob can be
sniffed from Content-Length in the GET response.

This commit changes HTTPReadSeeker to automatically detect the size from
Content-Length instead of requiring it to be passed in. The Stat call is
removed from Open because it is no longer necessary.

HTTPReadSeeker now takes an additional errorHandler callback argument which
translates an unsuccessful HTTP response into an appropriate API-level
error. Using a callback for this makes it possible to avoid leaking the
repsonse body to Read's caller, which would make lifecycle management
problematic.

Fixes #1223

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-02 14:21:13 -08:00
Stephen J Day
8aa3ee6923 storage: add further tests for Walk implementation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:55:10 -08:00
Stephen J Day
5d576bc0cd storage: enforce sorted traversal during Walk
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:25:30 -08:00
Stephen J Day
6ad10796ef storage: correctly handle error during Walk
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:25:29 -08:00
Troels Thomsen
300ce35c12 Map error type to error code
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-12-01 22:26:37 +01:00
Troels Thomsen
1ece510198 Use well-known error type
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-12-01 22:23:47 +01:00
Troels Thomsen
e2b4b426b4 Define error type
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-12-01 22:23:38 +01:00
Richard Scothern
329c353411 Merge pull request #1225 from SvenDowideit/DOCS-227
TOC for configuration page incorrect
2015-12-01 09:57:45 -08:00
Sven Dowideit
9ec49c04fd TOC for configuration page incorrect
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2015-12-01 14:07:55 +10:00
Richard Scothern
261ed7fa98 Merge pull request #1213 from SvenDowideit/move-to-docs-base-latest
Use FROM docs/base:latest again
2015-11-30 10:22:44 -08:00
Sven Dowideit
b43d0baec0 Use FROM docs/base:latest again
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2015-11-26 20:34:00 +10:00
Troels Thomsen
d309bce2d1 Verify manifest name format
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-26 10:28:35 +01:00
Troels Thomsen
34c8194c95 Verify manifest name length
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-26 10:28:28 +01:00
Troels Thomsen
e8f8f4034e Remove name verification
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-25 21:16:28 +01:00
Troels Thomsen
ca9f0451a6 Use case of type name
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-25 21:12:49 +01:00
Richard Scothern
4874f111ea Merge pull request #1208 from farmerworking/fix-oss-doc-error
Oss secure option default to true, Fix doc error
2015-11-25 11:25:25 -08:00
Richard Scothern
7647f7fe76 Merge pull request #1204 from MHBauer/gh1197
Formatting content digest calculation to avoid copyright symbols
2015-11-25 11:24:06 -08:00
farmerworking
8acf5a4d63 Oss secure option default to true, Fix doc error
Signed-off-by: farmerworking <farmerworking@gmail.com>
2015-11-25 12:38:44 +08:00
Olivier Gambier
d7a4f6cb6a Merge pull request #1207 from Djelibeybi/master
Adding additional command to enable the shared system certificates
2015-11-24 15:49:58 -08:00
Avi Miller
a7199f5d7c Fixed whitespace and made the command consistent with the other examples.
Signed-off-by: Avi Miller <avi.miller@oracle.com>
2015-11-25 10:41:40 +11:00
Avi Miller
b27d479ad5 Adding additional comment to enable the shared system certificates store.
Signed-off-by: Avi Miller <avi.miller@oracle.com>
2015-11-25 10:30:12 +11:00
Morgan Bauer
17d4e82944
Formatting content digest calculation to avoid copyright symbols
- resolves #1197

Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>
2015-11-24 10:13:34 -08:00
Richard Scothern
9c3c50faff Merge pull request #1198 from jonboulle/master
README: fix formatting of docker-registry link
2015-11-23 10:14:31 -08:00
Sven Dowideit
54f9743402 Merge pull request #1193 from SvenDowideit/docs-validation-changes
Doc fixes found by validation job
2015-11-23 12:17:22 +10:00
Jonathan Boulle
23b1b12b10 README: fix formatting of docker-registry link
Signed-off-by: Jonathan Boulle <jonathanboulle@gmail.com>
2015-11-20 17:18:25 -08:00
Sven Dowideit
64bc8af1b9 Doc fixes found by validation job
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2015-11-20 12:21:03 +10:00
Stephen Day
0a3acb2625 Merge pull request #1166 from treed/master
De-obfuscate error message
2015-11-11 17:18:25 -08:00
Richard Scothern
9577d3053a Merge pull request #910 from RichardScothern/go-1.5
Use go 1.5 on the build machine
2015-11-11 16:31:28 -08:00
Richard Scothern
c3c932f4e1 Use go 1.5 on the build machine
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-11 09:50:59 -08:00
Richard Scothern
bb745757de Merge pull request #1125 from rustyconover/fix-url-path-escaping
Fix failing test case for URL escaping problem.
2015-11-10 16:50:55 -08:00
Richard Scothern
a33095af80 Merge pull request #1141 from lebauce/swift-no-missing-segment
Ensure read after write for segments
2015-11-10 10:39:43 -08:00
Richard Scothern
00aeb6ece7 Merge pull request #1172 from aaronlehmann/swift-empty-delete-request
Fix empty delete requests with Swift fs driver
2015-11-10 10:36:19 -08:00
Aaron Lehmann
6664a1384e Fix empty delete requests with Swift fs driver
The Delete method lists objects under the given path and tries to delete
all of them with a bulk delete request. If the path has no objects
underneath it, the body of this request will be empty, which causes
HTTP-level issues. Specifically, Go's HTTP client senses the empty
request buffer and doesn't include a Content-Length, which causes the
Swift server to fail the request.

This commit fixes the problem by avoiding sending empty bulk delete
requests. This is the correct thing to do anyway, since there's no
reason to request deletion of zero objects.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-11-09 15:33:05 -08:00
Ted Reed
fddbf73a57 De-obfuscate error message
Previously, this error message would stringify as a pointer address,
which isn't particularly helpful.

This change breaks out the elements of the challenge object such that
the error is appropriately represented.

Signed-off-by: Ted Reed <ted.reed@gmail.com>
2015-11-06 17:10:28 -08:00
Richard Scothern
362ae9cc41 Merge pull request #1156 from RichardScothern/manifest-verification
Manifest Verification
2015-11-04 14:09:12 -08:00
Richard Scothern
7ed02ea1db Merge pull request #1161 from stevvooe/add-status-badges
Add nice status badges to README
2015-11-04 14:08:46 -08:00
Stephen J Day
5882b97ef2 Add nice status badges to README
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-11-04 12:48:18 -08:00
Richard Scothern
dd32fbe615 Before allowing a schema1 manifest to be stored in the registry, ensure that it
contains equal length History and FSLayer arrays.

This is required to prevent malformed manifests being put to the registry and
failing external verification checks.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-03 15:19:46 -08:00
Richard Scothern
603ffd58e1 Merge pull request #1150 from RichardScothern/carry-1070
Add Storage Driver Context.
2015-11-03 15:15:48 -08:00
Richard Scothern
bc6e4cdceb Add a generic error type to capture non-typed errors
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-03 11:19:44 -08:00
amitshukla
9637cb40cd Fix for issue 664: https://github.com/docker/distribution/issues/664
Errors thrown by storage drivers don't have the name of the driver, causing user
confusion about whether the error is coming from Docker or from a storage driver.
This change adds the storage driver name to each error message.

This required changing ErrUnsupportedDriver to a type, leading to code changes
whenever ErrUnsupportedDriver is used.  The tests check whether the driver name
appears in the error message.

Signed-off-by: Amit Shukla <amit.shukla@docker.com>
2015-11-03 11:19:17 -08:00
Richard Scothern
a9da0e5100 Merge pull request #1114 from lebauce/swift-temp-url
Redirect support in Swift driver
2015-11-03 09:34:01 -08:00
Sylvain Baubeau
3ff8af326b Ensure read after write for segments
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 17:09:52 +01:00
Sylvain Baubeau
0737bb7175 Update Swift bindings
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 17:09:52 +01:00
Sylvain Baubeau
7c3281861f Add support for temporary URL for Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 09:59:50 +01:00
Sylvain Baubeau
7759153f2f Update Swift bindings
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 09:57:39 +01:00
Richard Scothern
5fc31de717 Merge pull request #1152 from RichardScothern/credit2
Add github IDs to maintainers
2015-11-02 17:14:35 -08:00
Richard Scothern
34754f20d8 Update authors
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-02 16:41:36 -08:00
Olivier Gambier
9316c2c3bc Merge pull request #1151 from RichardScothern/credit
Update AUTHORS and MAINTAINERS
2015-11-02 16:07:21 -08:00
Richard Scothern
b313f011e3 Update authors
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-02 16:05:34 -08:00
Richard Scothern
e760111803 Update maintainers
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-02 16:05:06 -08:00
Richard Scothern
222e6e91c4 Merge pull request #1149 from stevvooe/close-filereader
registry/storage: close filereader after allocation
2015-11-02 13:22:32 -08:00
Stephen J Day
46dd057f5c registry/storage: close filereader after allocation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-11-02 11:55:11 -08:00
Richard Scothern
c6c9194e9c Merge pull request #1148 from aaronlehmann/withname-check-length
Check length of input in WithName
2015-11-02 11:54:34 -08:00
Aaron Lehmann
073bd89262 Check length of input in WithName
There is a constraint on the length of the name in a reference, so
WithName should be checking this instead of potentially creating a
reference that doesn't comply with the rules.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-11-02 09:12:21 -08:00
Richard Scothern
ef17db8d73 Merge pull request #1122 from RichardScothern/scheduler-coalesce
Buffer writes to scheduler state file.
2015-10-30 13:43:07 -07:00
Richard Scothern
326ec6c121 Merge pull request #1139 from aaronlehmann/reference-withtag-fix
Use correct regexp in reference.WithTag
2015-10-29 15:25:18 -07:00
Richard Scothern
33428c37e1 Buffer writing the scheduler entry state to disk by periodically checking for
changes to the entries index and saving it to the filesystem.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-10-29 11:35:52 -07:00
Richard Scothern
9e8aaf7b40 Merge pull request #756 from aibaars/storage-gcs
Storage driver for Google Cloud Storage
2015-10-29 10:41:17 -07:00
Aaron Lehmann
985ee759e7 Use correct regexp in reference.WithTag
This was using a different regexp from the intended one. This meant that
tags with uppercase characters were not accepted.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-28 18:22:00 -07:00
Richard Scothern
987a69dd05 Merge pull request #1096 from aaronlehmann/simplify-proxy-scheduler
Simplify proxy scheduler
2015-10-28 13:23:00 -07:00
Arthur Baars
59784ecdd0 Storage driver for: Google Cloud Storage (gcs)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2015-10-28 20:21:56 +00:00
Arthur Baars
41e71e9a46 Google Cloud Storage (gcs) dependencies (2)
Add files of new dependencies to Godeps/_workspace

Signed-off-by: Arthur Baars <arthur@semmle.com>
2015-10-28 11:26:43 +00:00
Arthur Baars
95db61e97f Google Cloud Storage (gcs) dependencies (1)
Add new dependencies to Godeps/Godeps.json

Signed-off-by: Arthur Baars <arthur@semmle.com>
2015-10-28 11:26:39 +00:00
Olivier Gambier
731c43e3db Merge pull request #1136 from RichardScothern/update-roadmap
Update distribution roadmap.
2015-10-27 20:39:52 -07:00
Richard Scothern
183d144520 Update distribution roadmap.
Update pull through cache status.
Add section for metadata storage.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-10-27 17:20:49 -07:00
Stephen Day
2d23e77284 Merge pull request #1047 from mattmoor/expires_in_proposal
Add `expires_in` and `issued_at` to the auth spec.
2015-10-27 14:29:08 -07:00
Stephen Day
6bcd8a1dec Merge pull request #1099 from RichardScothern/go-version
Add go version to logger output
2015-10-27 14:19:43 -07:00
Matt Moore
0c7cd3fcf8 Add expires_in and issued_at to the auth spec.
This extends the specification for the Bearer token response to include
information pertaining to when an issued Bearer token will expire.

This also allows the client to accept `access_token` as an alias for `token`.

Signed-off-by: Matt Moore <mattmoor@google.com>
2015-10-27 11:05:33 -07:00
Rusty Conover
5f2205981b Fix failing test case
When building a URL don't include path components in the host parameter.

Closes #1124

Signed-off-by: Rusty Conover <rusty@luckydinosaur.com>
2015-10-25 13:03:22 -04:00
Richard Scothern
1c5f8166e4 Merge pull request #1121 from thaJeztah/fix-doc-emoji
Docs: replace emoji with regular warning
2015-10-22 17:13:33 -07:00
Sebastiaan van Stijn
bf6ca6cb53 Docs: replace emoji with regular warning
The docs don't render emoji, so replaced the `⚠️` with
a `**Warning**:` to keep the formatting consisten with
`**Note**:` used in other parts of the docs.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2015-10-22 15:59:48 -07:00
Richard Scothern
a266f99418 Merge pull request #1095 from mrwacky42/api-version-annotated
Add annotated example for adding header; +spelling
2015-10-22 14:04:32 -07:00
Richard Scothern
e569236f1e Add go version to logger output
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-10-22 10:11:25 -07:00
Olivier Gambier
24a3f92616 Merge pull request #1118 from SvenDowideit/stop-linking-when-rendering-to-html
Prevent md->html link generation
2015-10-22 09:34:09 -07:00
Sven Dowideit
a5040860f2 Prevent md->html link generation
Signed-off-by: Sven Dowideit <SvenDowideit@home.org.au>
2015-10-22 17:27:21 +10:00
Stephen Day
e5d9837970 Merge pull request #1098 from denverdino/master
Redundant digest verification in validateBlob when pushing a new layer
2015-10-21 14:50:18 -07:00
Stephen Day
e6bb0aa7c0 Merge pull request #1115 from tianon/fix-auth-servehttp-comment
Update "type auth.Challenge" comment example code
2015-10-21 12:56:14 -07:00
Tianon Gravi
39d47e8eea Update "type auth.Challenge" comment example code
This interface was changed in a0fdfb9d4d, but the comment wasn't ever updated to match.

Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
2015-10-21 11:54:13 -07:00
Richard Scothern
7e9c832524 Merge pull request #1111 from aaronlehmann/golint-comments
Correct two golint comment issues
2015-10-20 09:53:14 -07:00
Aaron Lehmann
5608db616c Correct two golint comment issues
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-19 16:42:12 -07:00
Olivier Gambier
21b6f54796 Merge pull request #1110 from RichardScothern/env-var-clarification
Clearer documentation around environment variable overrides
2015-10-19 14:53:20 -07:00
Richard Scothern
f9317483d3 Clearer documentation around environment variable overrides
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-10-19 14:52:08 -07:00
Richard Scothern
d4910831a3 Merge pull request #1006 from RichardScothern/proxy-serialize
Fix a race condition in pull through cache population
2015-10-16 17:13:26 -07:00
Olivier Gambier
f6d22717e7 Merge pull request #1103 from moxiegirl/hugo-15-retooling
Updating for Hugo 15 patch
2015-10-16 13:51:18 -07:00
Mary Anthony
8e92a8ab00 Updating for Hugo 15 patch
Signed-off-by: Mary Anthony <mary@docker.com>
2015-10-16 12:20:38 -07:00
Stephen Day
dfe60f4cb1 Merge pull request #827 from aaronlehmann/read-only-mode-2
Add a read-only mode as a configuration option
2015-10-15 11:50:31 -07:00
Richard Scothern
539d6312bf Merge pull request #1053 from mrwacky42/patch-1
Fix spelling
2015-10-15 11:44:57 -07:00
Richard Scothern
d87cf10852 Merge pull request #1094 from aaronlehmann/correct-payload-serialization-order
Correct unmarshal order for SignedManifest
2015-10-15 11:44:12 -07:00
Aaron Lehmann
c56e8c2533 Simplify proxy scheduler
The proxy scheduler implemented its own timer state machine. It's
simpler and more efficient to leverage the Go runtime's timer heap by
using time.AfterFunc.

This commit adds a time.Timer to each scheduler entry, and starts and
stops those timers as necessary. Then the mainloop goroutine and its
associated logic are not needed.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-15 09:18:21 -07:00
Li Yi
ad14545745 Redundant digest verification in validateBlob
Change-Id: I03764edadae529db2cc3acf7ecca329570f18659
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-10-15 22:21:14 +08:00
Sharif Nassar
0249cc9cb2 Add annotated example for adding header; +spelling
This adds a variable to help nginx add the
Docker-Distribution-Api-Version when using basic auth, and not add the
header when it sees it from the upstream.

Also fix some minor spelling/grammar issues.

Signed-off-by: Sharif Nassar <sharif@mrwacky.com>
2015-10-14 15:21:18 -07:00
Stephen J Day
76624704c3 Correct unmarshal order for SignedManifest
To ensure that we only unmarshal the verified payload into the contained
manifest, we first copy the entire incoming buffer into Raw and then unmarshal
only the Payload portion of the incoming bytes. If the contents is later
verified, the caller can then be sure that the contents of the Manifest fields
can be trusted.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-14 14:37:34 -07:00
Richard Scothern
fa3c275d63 Merge pull request #1093 from aaronlehmann/reference-improvements
Usability improvements for reference package
2015-10-14 14:19:41 -07:00
Aaron Lehmann
16eea0cc47 Usability improvements for reference package
Various improvements motivated by early real-world use in engine code
under development:

- Make `WithDigest` return `Canonical`, since the return value always
  has a name and a digest.

- Introduce a `NamedTagged` type, which can be used for strongly typing
  cases where something must have a name and a tag.

- Rename `ParseNamed` to `WithName`, and create a
  `ParseNamed` that accepts tags and digests (returning a `Named` type).
  The new `ParseNamed` makes code using the reference package much less
  verbose, since typical use cases require a name, and this is an
  additional type assertion and error case after every `Parse` call.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-13 16:07:34 -07:00
Richard Scothern
4c4112bdcf Merge pull request #1035 from aaronlehmann/host-parameter-alternate
Add http.host parameter
2015-10-13 14:03:48 -07:00
Stephen Day
bfd1f423c5 Merge pull request #1033 from jlhawn/api_spec_update_auth_errors
[api spec] Update authN and authZ errors
2015-10-13 13:52:18 -07:00
Stephen Day
8aec18217a Merge pull request #963 from dmcgowan/reference-update
[Carry 690] Add a new reference package abstracting tags and digests
2015-10-13 13:15:43 -07:00
Richard Scothern
6a4cdab380 Merge pull request #1083 from BrianBland/fixSwiftDocs
Fixes documentation for swift driver "container" parameter
2015-10-13 11:57:43 -07:00
Sharif Nassar
929595c7d1 Fix spelling
Signed-off-by: Sharif Nassar <sharif@mrwacky.com>
2015-10-12 16:11:30 -07:00
Aaron Lehmann
b07d759241 Add WithTag and WithDigest combinator functions
These functions allow a Named type to be combined with a tag or a
digest. WithTag will replace the ImageReference function in
github.com/docker/docker/utils as the Docker Engine transitions to the
reference package.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-12 16:00:33 -07:00
Derek McGowan
3943c4165c Make Named,Tagged, and Digested implement Reference
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-10-09 16:14:08 -07:00
Derek McGowan
6bd5b8c24e Update regexp to support repeated dash and double underscore
In order to support valid hostnames as name components, supporting repeated dash was added.
Additionally double underscore is now allowed as a separator to loosen the restriction for previously supported names.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-10-09 16:14:08 -07:00
Derek McGowan
bcda04d6cd Add field type for serialization
Since reference itself may be represented by multiple types which implement the reference inteface, serialization can lead to ambiguous type which cannot be deserialized.
Field wraps the reference object to ensure that the correct type is always deserialized, requiring an extra unwrap of the reference after deserialization.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-10-09 16:05:49 -07:00
Derek McGowan
31a448a628 Update to provide small and clear interfaces
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-10-09 16:05:49 -07:00
Tibor Vass
f8c09b6a7d Add a new reference package abstracting repositories, tags and digests
There seems to be a need for a type that represents a way of pointing
to an image, irrespective of the implementation.

This patch defines a Reference interface and provides 3 implementations:
- TagReference: when only a tag is provided
- DigestReference: when a digest (according to the digest package) is
  provided, can include optional tag as well

Validation of references are purely syntactic.

There is also a strong type for tags, analogous to digests, as well
as a strong type for Repository from which clients can access the
hostname alone, or the repository name without the hostname, or both
together via the String() method.

For Repository, the files names.go and names_test.go were moved from
the v2 package.

Signed-off-by: Tibor Vass <tibor@docker.com>
2015-10-09 16:05:34 -07:00
Brian Bland
90fe4b9256 Fixes documentation for swift driver "container" parameter
No longer claims that an additional segments container is used

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-10-08 16:51:57 -07:00
Derek McGowan
1d266b00e1 Merge pull request #1079 from mattmoor/allow-hostname-parts
Allow hostname components in component names.
2015-10-08 16:49:42 -07:00
Richard Scothern
515b9596e7 Fix a race condition in pull through cache population by removing the functionality
of readers joining current downloads.  Concurrent requests for the same blob
will not block, but only the first instance will be comitted locally.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-10-08 16:12:29 -07:00
Matt Moore
74c32a0a51 Allow hostname components in component names.
Fixes https://github.com/docker/distribution/issues/1062

This relaxes the naming restrictions places on Docker images to permit
valid hostnames according to [RFC-2396](https://www.ietf.org/rfc/rfc2396.txt).

It deviates from the RFC in the following ways:
 1) Allow underscores where we allow hyphens (hostnames don't allow
 underscores, which we must for backwards compatibility).
 2) Leave "top-level" name segments unrestricted (domains require an
 alpha character to begin a top-level domain, e.g. "com").
 3) DO NOT allow a trailing dot, as permitted by FQDNs.

Signed-off-by: Matt Moore <mattmoor@google.com>
2015-10-08 11:29:54 -07:00
Richard Scothern
4946d0d035 Merge pull request #1072 from compnski/patch-1
Update configuration.md
2015-10-08 10:12:21 -07:00
Jason Freidman
1c1917fd5a Update configuration.md
It seems that enabling proxy stops my instance from accepting local pushes, but I can't find mention of that in the docs.

Signed-off-by: Jason Freidman <jason@periscope.io>
2015-10-07 20:02:58 -07:00
Richard Scothern
a0007130cf Merge pull request #1076 from harche/patch-2
fixing registry version command example
2015-10-07 10:09:15 -07:00
harche
4ff7656ba1 Typo in command to check registry version
Signed-off-by: Harshal <p.harshal@gmail.com>
2015-10-07 16:38:42 +05:30
Richard Scothern
89bedf1e7f Merge pull request #1048 from docker/docs-secure-default
Correct the default setting of s3's "secure" parameter in configuration.md
2015-10-01 10:26:38 -07:00
Richard Scothern
61a4a1362c Merge pull request #950 from jackpgriffin/master
Skip creating swift container if already exists
2015-10-01 10:15:54 -07:00
Aaron Lehmann
7f83bfd8af Correct the default setting of s3's "secure" parameter in configuration.md
This defaults to true in the code, but the doc claimed it defaults to
false.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-30 18:41:06 -07:00
Aaron Lehmann
0a6988195e Add http.host parameter
This allows the administrator to specify an externally-reachable URL for
the registry. It takes precedence over the X-Forwarded-Proto and
X-Forwarded-Host headers, and the hostname in the request.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-30 15:12:01 -07:00
Derek McGowan
618ffe118b Merge pull request #1005 from xiekeyang/spec
add description to manifest spec
2015-09-30 11:47:16 -07:00
Richard Scothern
e760fb2dc6 Merge pull request #988 from tt/fix-incorrect-default
Fix incorrect default
2015-09-30 10:40:11 -07:00
Josh Hawn
0f670bdc91 [api spec] Update authN and authZ errors
Associate HTTP 401s with Authentication errors rather than Authorization
errors. Changes the meaning of the UNAUTHORIZED error to be authentication
specific.

Defines DENIED error code to be associated with authorization
errors which result in HTTP 403 responses.

Add 'No Such Repository' errors to more endpoints.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-09-30 09:12:31 -07:00
Stephen Day
497133f567 Merge pull request #1045 from tt/remove-duplicate-dependency-definition
Remove duplicate dependency definition
2015-09-30 13:19:35 +02:00
Stephen Day
7f2ffe0ad1 Merge pull request #1036 from dmcgowan/digest-set-update
Add remove and list functions to digest set
2015-09-30 13:17:41 +02:00
Stephen Day
ca2156d558 Merge pull request #1024 from aaronlehmann/avoid-testing-import
Avoid importing "testing" in externally-facing code
2015-09-30 13:14:42 +02:00
Troels Thomsen
6a3496d9a5 Remove duplicate dependency definition
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-09-30 10:48:28 +02:00
Troels Thomsen
5fc0c2f0b7 Use revision of saved package
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-09-30 10:48:21 +02:00
xiekeyang
da8ef7585d add description to manifest spec
This PR add some description to manifest spec.
It clarifies the relationship between `fsLayers` and `history` fields.

Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2015-09-30 10:16:46 +08:00
Richard Scothern
64660c68f2 Merge pull request #1042 from tt/upgrade-amazon-library
Upgrade Amazon library
2015-09-29 17:50:41 -07:00
Troels Thomsen
cca9f74eec Upgrade Amazon library to revision aa6e716d710a0c7941cb2075cfbb9661f16d21f1
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-09-30 00:29:17 +02:00
Aaron Lehmann
6acb3a0d7e Avoid importing "testing" in externally-facing code
The "testing" package adds some flags in its init function, so utilities
that import distribution code may print a page of extra testing flags in
their help output.

This commit solves the issue by moving an import of "testing" in the
registry/storage/cache package to a new
registry/storage/cache/cachecheck package, which is only imported by
tests.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-29 10:06:39 -07:00
Derek McGowan
dc6c54489c Add remove and list functions to digest set
Add mutex protection around set access

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-09-29 10:04:15 -07:00
Aaron Lehmann
14a4b4d9d6 Merge pull request #1001 from RichardScothern/fix-atrocity
Don't return nil, nil from functions
2015-09-28 13:34:11 -07:00
Troels Thomsen
cfe50c9ef4 Fix incorrect default
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-09-28 10:08:17 +02:00
Richard Scothern
d03c56129c Merge pull request #991 from mattmoor/expires_in_proposal
Fixes #684
2015-09-25 15:23:28 -07:00
Richard Scothern
07b05a72bb Merge pull request #999 from nakedible-p/patch-1
Remove initial access check from S3 driver
2015-09-25 15:23:03 -07:00
Richard Scothern
794975e9e6 Merge pull request #1014 from shyr/patch-3
Minor grammar fix : duplicated coordinators.
2015-09-23 10:04:33 -07:00
Sungho Moon
7bf66b6855 Minor grammar fix : duplicated coordinators.
Signed-off-by: Sungho Moon <sungho.moon@navercorp.com>
2015-09-23 11:40:13 +09:00
Aaron Lehmann
a601f92336 Add an "enabled" parameter under "readonly", and make it as if the mutable handlers don't exist when read-only mode is enabled
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-22 15:49:26 -07:00
Aaron Lehmann
c9bb330b71 Add a read-only mode as a configuration option
Add "readonly" under the storage/maintenance section. When this is set
to true, uploads and deletions will return 503 Service Unavailable
errors.

Document the parameter and add some unit testing.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-22 15:47:48 -07:00
Nuutti Kotivuori
26d8344872 Remove initial access check from S3 driver
In the S3 storage driver there is currently an initial access permission check by listing the bucket. If this check fails, registry will panic and exit.

However, this check is broken in two ways. First of all it strips the final slash from the root directory path, meaning that any access permissions which limit access to a single directory will fail, because S3 treats the path as strict prefix match. Secondly it fails to strip any leading slash that might be present, unlike the other access places, which means that the path used is different as a leading slash is allowed and significant in a filename in S3.

Since there is also a periodic health check which correctly checks access permissions and shows the error more cleanly, the best solution seems to be to just remove this initial access check.

Signed-off-by: Nuutti Kotivuori <nuutti.kotivuori@poplatek.fi>
2015-09-19 12:47:12 +03:00
Richard Scothern
f36ab5a834 Don't return a nil array and a nil error if the Tags endpoint cannot be found
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-09-18 16:19:41 -07:00
Richard Scothern
924913b4c3 Avoid returning nil, nil when fetching a manifest by tag by introducing a new
error ErrManifestNotModified which can be checked by clients.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-09-18 16:19:37 -07:00
Matt Moore
ca7c845dc0 Fixes #684
Split the discussion of v2 authentication into two parts:
1) A specification of the handshake between the client, registry and
authentication service.
2) A description of how `docker/distribution` implements this using JWT.

This should make it clearer that `#2` is an implementation detail, and
that clients should regard tokens as opaque entities that only the
registry and authentication service should understand.

Signed-off-by: Matt Moore <mattmoor@google.com>
2015-09-18 15:26:32 -07:00
Richard Scothern
f8109a78f9 Merge pull request #997 from xiekeyang/typo
typo
2015-09-18 12:03:58 -07:00
xiekeyang
78c760f898 typo
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2015-09-18 17:27:01 +08:00
Aaron Lehmann
f651c5d68e Merge pull request #981 from LouisKottmann/master
Add exemple proxy mistake whith nginx sitting behind an ELB
2015-09-17 14:35:12 -07:00
Louis Kottmann
4075e2bb3d Add exemple proxy mistake whith nginx sitting behind an ELB
closes #970

Signed-off-by: Louis Kottmann <louis.kottmann@gmail.com>
2015-09-17 12:46:12 +02:00
Richard Scothern
4271c60d98 Merge pull request #973 from jlhawn/update_token_spec
[docs/spec/auth] Clarify the Token auth workflow
2015-09-16 10:41:08 -07:00
Stephen Day
ece8e132bf Merge pull request #974 from stevvooe/context-cleanup
context: WithVersion and context package cleanup
2015-09-14 19:24:33 -07:00
Stephen J Day
530afa5234 Add WithVersion to context and other cleanup
By adding WithVersion to the context package, we can simplify context setup in
the application. This avoids some odd bugs where instantiation order can lead
to missing instance.id or version from log messages.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-09-14 17:14:31 -07:00
Josh Hawn
fb481ef843 [docs/spec/auth] Clarify the Token auth workflow
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-09-14 15:57:16 -07:00
Stephen J Day
360c24d975 Allow interface{} keys when using logger
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-09-11 11:33:55 -07:00
Stephen Day
b0e94fb154 Merge pull request #977 from aaronlehmann/use-distribution-context
Only use the distribution/context package in registry.go
2015-09-11 11:33:19 -07:00
Aaron Lehmann
c4cf32c797 Only use the distribution/context package in registry.go
This solves a issue from #909 where instance.id was not printed in logs,
because this file was using the background context from
golang.org/x/net/context instead of
github.com/docker/distribution/context.

It's cleaner to standardize on one package, so this commit removes the
import of golang.org/x/net/context entirely. The Context interfaces
defined in both packages are the same, so other code using
golang.org/x/net/context can still pass its context to NewRegistry.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-11 09:54:15 -07:00
Richard Scothern
00d59d621f Merge pull request #909 from aaronlehmann/importable-registry
Move initialization code from main.go to the registry package
2015-09-10 17:48:11 -07:00
Stephen Day
1cdcc0462a Merge pull request #888 from aaronlehmann/config-env-vars
More flexible environment variable overrides
2015-09-10 17:01:30 -07:00
Olivier Gambier
6bf5a049ff Merge pull request #964 from aaronlehmann/pin-dind-dep
Download dind from v1.8.1 tag of docker/docker repo
2015-09-09 18:13:10 -07:00
Aaron Lehmann
540e3f1433 Update godeps for addition of cobra dependency
codegangsta/cli is being removed since the "dist" tool which used it is
no longer in the repo.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-09 14:39:31 -07:00
Aaron Lehmann
cbc9957e29 Add a cobra command that implements the entire main function for registry
Use this command in cmd/registry/main.go.

Move debug server to the main command, and change Serve to be a
ListenAndServe function.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-09 14:39:31 -07:00
Aaron Lehmann
9b69e40c93 Move initialization code from main.go to the registry package
This makes it easier to embed a registry instance inside another
application.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-09 14:39:31 -07:00
Aaron Lehmann
2b42e98a98 Download dind from v1.8.1 tag of docker/docker repo
Fixes #898.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-08 17:48:13 -07:00
Richard Scothern
7a305cc8cd Merge pull request #946 from humble00/master
Add TrustId parameter to swift driver
2015-09-08 15:59:29 -07:00
Jack Griffin
2749bcdc28 Skip creating swift container if already exists
Signed-off-by: Jack Griffin <jackpg14@gmail.com>
2015-09-04 18:53:17 +00:00
Aaron Lehmann
a49bf24abe More flexible environment variable overrides
Overriding configuration parameters with environment variables used to
work by walking the configuration structure and checking for a
corresponding environment variable for each item. This was very limiting
because only variables corresponding to items that already existed in
the configuration structure would be checked. For example, an
environment variable corresponding to nested maps would only be noticed
if the outer map's key already existed.

This commit changes environment variable overriding to iterate over the
environment instead. For environment variables beginning with the
REGISTRY_ prefix, it splits the rest of their names on "_", and
interprets that as a path to the variable to unmarshal into. Map keys
are created as necessary. If we encounter an empty interface partway
through following the path, it becomes an implicit
map[string]interface{}.

With the new unit tests added here, parser.go now has 89.2% test
coverage.

TestParseWithExtraneousEnvStorageParams was removed, because the limit
of one storage driver is no longer enforced while parsing environment
variables. Now, Storage.Type will panic if multiple drivers are
specified.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-03 16:11:53 -07:00
Stephen Day
7c5a9ae96d Merge pull request #949 from docker/amylindburg-patch-1
Delete stale info from Project Planning in ROADMAP.md
2015-09-02 13:09:44 -07:00
Amy Lindburg
1ecceb6192 Delete stale info from Project Planning section
Alter Project Planning area to point to Wiki so ROADMAP.md does not get stale.

Signed-off-by: Amy Lindburg <amy.lindburg@docker.com>
2015-09-02 11:37:10 -07:00
Hua Wang
ddb689ef8b Add TrustId parameter to swift driver
github/ncw/swift has added support for trust, so let's add it.

Signed-off-by: Hua Wang <wanghua.humble@gmail.com>
2015-09-02 15:05:29 +08:00
Olivier Gambier
dbbafe92e9 Merge pull request #941 from liubin/master
add indentations
2015-08-28 12:52:50 -07:00
Aaron Lehmann
bc010e506e Merge pull request #931 from dmp42/5.all-ur-proxy-are-belong-to-us
Documentation fixes
2015-08-28 12:51:27 -07:00
Olivier Gambier
0c42f0cd63 Granmar and speeling fixes
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-28 12:48:55 -07:00
Olivier Gambier
6197ca7b10 Adding mirror as a recipe
Normalized mirror.md to match other recipes.
Slightly tweaked content, emphasizing the privacy concerns about using login/password on the backend.

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-28 12:47:39 -07:00
Olivier Gambier
5df53c0681 Documentation enhancements
Making metadata:
- more consistent
- more specific (fixed copy pasting)
- refine coverage

Insecure information cleanup

Removing no longer used files:
- mkdocs is gone
- the registry diagram is not used, and is a bit silly :)

Minor fixes

Fixing links

Recipes:
- harmonized code sections style to the rest of the docs
- harmonized recipe "style"
- listing new recipes

Enhance deploying

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-28 12:46:53 -07:00
bin liu
e14e40b68c add indentations
Signed-off-by: bin liu <liubin0329@gmail.com>
2015-08-28 20:09:36 +08:00
Stephen Day
1927c6c0f2 Merge pull request #934 from stevvooe/sanitize-url-correctly
Correctly sanitize location url preserving parameters
2015-08-27 22:30:21 -07:00
Stephen J Day
7232daf692 Correctly sanitize location url preserving parameters
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-27 17:06:03 -07:00
Olivier Gambier
21edffdee1 Merge pull request #929 from RichardScothern/docs
Add some clarifying documentation for pull through cache configuration
2015-08-27 16:41:22 -07:00
Richard Scothern
246138598d Add some clarifying documentation for pull through cache configuration
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-08-27 10:18:36 -07:00
Olivier Gambier
c2b25053d3 Merge pull request #932 from avinson/master
add info about redis eviction policy recommended setting
2015-08-26 21:23:51 -07:00
Aaron Vinson
db36173705 add info about redis eviction policy recommended setting 2015-08-26 18:19:13 -07:00
Olivier Gambier
b78a858b5d Merge pull request #918 from aaronlehmann/json-spec
Update JSON spec to point out escaping of <, >, &
2015-08-24 16:58:11 -07:00
Aaron Lehmann
4cb5626d9e Update JSON spec to point out escaping of <, >, &
...and add blurb about not relying on canonicalization to ensure
identical output.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-24 16:28:42 -07:00
Richard Scothern
7a4920d8e5 Merge pull request #897 from dmp42/5.all-ur-proxy-are-belong-to-us
Starting with recipes
2015-08-24 11:28:17 -07:00
Aaron Lehmann
7c46c3fb96 Merge pull request #912 from stevvooe/manifest-refactoring-schema1
Move manifest package to schema1
2015-08-21 16:47:09 -07:00
Stephen J Day
6712e602b0 Move manifest package to schema1
As we begin our march towards multi-arch, we must prepare for the reality of
multiple manifest schemas. This is the beginning of a set of changes to
facilitate this. We are both moving this package into its target position where
it may live peacefully next to other manfiest versions.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-21 16:29:47 -07:00
Stephen Day
8c3fc2619c Merge pull request #907 from stevvooe/digest-tool
Add generic content digest tool
2015-08-21 11:11:40 -07:00
Richard Scothern
2e33d6dfc0 Merge pull request #876 from dmcgowan/integration-test-cleanup
Fix mismatched version error on integration tests
2015-08-21 11:01:42 -07:00
Richard Scothern
37d4ad081f Merge pull request #901 from aaronlehmann/configurable-health-checks
Add configurable file-existence and HTTP health checks
2015-08-21 10:53:32 -07:00
Stephen J Day
3f7f956f80 More consistent return from ErrorCode.Error()
To bring ErrorCode into liine with Go conventions, ErrorCode.Error() now
returns the "nice" value of the error code. This ensures error message assembly
works similar to commonly used Go conventions when directly using ErrorCode as
an error.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-20 17:36:24 -07:00
Aaron Lehmann
b67aab2f60 Add headers parameter for HTTP checker
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 15:11:16 -07:00
Aaron Lehmann
e8f088fea6 Add a TCP health checker
Also, add timeout and status code parameters to the HTTP checker, and
remove the threshold parameter for the file checker.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 15:11:15 -07:00
Aaron Lehmann
b9b9cafa8f Expose a Registry type in health package, so unit tests can stay isolated from each other
Update docs.

Change health_test.go tests to create their own registries and register
the checks there. The tests now call CheckStatus directly instead of
polling the HTTP handler, which returns results from the default
registry.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 15:07:54 -07:00
Stephen J Day
7835d261d8 Add generic content digest tool
Previously a useful gist, this changeset polishes the original tarsum tool into
a utility that can be used to calculate content digests. Any algorithm from the
digest package is supported with additional support from tarsum.

This tool is very useful for quickly checking backend digests and verifying
correctness.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-20 14:55:34 -07:00
Aaron Lehmann
79959f578a Switch tests to import "github.com/docker/distribution/context"
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 14:50:12 -07:00
Aaron Lehmann
216df32510 Add storagedriver section to health check configuration
Add default storagedriver health check to example configuration files
with parameters matching the previous hardcoded configuration.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 14:50:12 -07:00
Aaron Lehmann
b09b0ffcf9 Add configurable file-existence and HTTP health checks
Add a section to the config file called "health". Within this section,
"filecheckers" and "httpcheckers" list checks to run. Each check
specifies a file or URI, a time interval for the check, and a threshold
specifying how many times the check must fail to reach an unhealthy
state.

Document the new options in docs/configuration.md.

Add unit testing for both types of checkers. Add an UnregisterAll
function in the health package to support the unit tests, and an
Unregister function for consistency with Register.

Fix a string conversion problem in the health package's HTTP checker.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 14:50:12 -07:00
Olivier Gambier
065bda79bb Starting with recipes
- apache recipe
- streamline nginx recipe
- recipes file

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-19 16:15:36 -07:00
Stephen Day
e4b93d1e6d Merge pull request #893 from aaronlehmann/functional-opts
Functional options for NewRegistryWithDriver
2015-08-19 14:41:08 -07:00
Richard Scothern
8f77a639e7 Merge pull request #896 from aaronlehmann/closenotifier-warnings
Fix CloseNotifier handling and avoid "the ResponseWriter does not implement CloseNotifier" warnings in logs
2015-08-19 12:41:03 -07:00
Aaron Lehmann
5bef618ace Add a unit test which verifies the ResponseWriter endpoints see implements CloseNotifier
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-19 11:37:53 -07:00
Aaron Lehmann
dbbcf5fe42 Functional options for NewRegistryWithDriver
Clean up calling convention for NewRegistryWithDriver to use functional
arguments.

This is a first step towards the refactor described in #215. I plan to
add additional options in the process of moving configurable items from
the App structure to the registry structure.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-19 11:06:30 -07:00
Aaron Lehmann
0e7462f1dd Fix CloseNotifier handling and avoid "the ResponseWriter does not implement CloseNotifier" warnings in logs
A change in #763 to address review comments caused problems. Originally,
instrumentedResponseWriter implemented the CloseNotifier interface, and
would panic if it was wrapping something that did not implement that
interface. This was split into a separate instrumentedResponseWriterCN
type that implements CloseNotifier, so there's a fallback if
instrumentedResponseWriter ever needs to wrap something that does not
implement this interface.

instrumentedResponseWriter's Value method would end up upcasting either
type back to instrumentedResponseWriter, which does not implement the
interface. In effect, instrumentedResponseWriterCN was never visible to
the handler.

This fixes the problem by implementing a wrapper Value method for
instrumentedResponseWriterCN.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-18 17:28:51 -07:00
Olivier Gambier
ed7cc91e6f Merge pull request #894 from aaronlehmann/fix-tests
Fix tests after #846
2015-08-18 13:57:13 -07:00
Aaron Lehmann
98d9b5f589 Fix tests after #846
Change checkResponse to only expect the configured
X-Content-Type-Options header if it doesn't receive a 405 error, which
means the handler isn't registered for that method.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-18 13:33:26 -07:00
Richard Scothern
cb6fde2f71 Merge pull request #874 from stevvooe/load-balancer-endpoint
Provide yes man endpoint for inflexible load balancers
2015-08-18 12:54:58 -07:00
Richard Scothern
f169359798 Merge pull request #846 from aaronlehmann/http-header-configuration
Add a section to the config file for HTTP headers to add to responses
2015-08-18 12:53:05 -07:00
Aaron Lehmann
ea43d57832 Merge pull request #883 from tg123/master
enable aliyun oss support by default in docker image
2015-08-18 11:17:42 -07:00
Aaron Lehmann
ae2146c6a8 Merge pull request #889 from stevvooe/remove-pathmapper
Remove pathMapper object
2015-08-18 09:40:34 -07:00
Stephen J Day
641cdf3ba6 Remove pathMapper object
The use of the pathMapper is no longer needed the way we have organized the
code base. The extra level of indirection has proved unnecessary and confusing
so we've opted to clean it up. In the future, we may require more flexibility,
but now it is simply not required.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-17 18:51:05 -07:00
Derek McGowan
1a6949134e Fix mismatched version error on integration tests
When running a different version of docker outside of the default in the integration image, then commands will fail with mismatched version unless the docker binary is specified to the correct version.

Add various cleanups to run script.
Run all commands interacting with docker inside the container in an exec.
Remove port binding to outside of container since all commands run inside.
Trap docker rm to exit in case of failure which prevents final command from running.
Do no copy images when $DOCKER_VOLUME is specified, this allows for faster runs when mounting a volume with a warm image cache.
Move exec and graph driver defaulting into run engine script.
Remove duplicated update of /etc/hosts.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-08-17 17:00:58 -07:00
Olivier Gambier
8133b04efa Merge pull request #881 from samsong8610/update-example-config-to-use-blobdescriptor
Update example config to avoid deprecated option
2015-08-17 14:09:18 -07:00
Stephen J Day
ad995ab8fa Add guide on load balancing a registry
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-17 13:29:48 -07:00
tgic
85df63eb06 enable aliyun oss support by default in docker image
Signed-off-by: tgic <farmer1992@gmail.com>
2015-08-17 21:47:30 +08:00
xg.song
2f9f6c08ed Update example config to avoid deprecated option
The storage cache option layerinfo is deprecated,
so use blobdescriptor instead in example config files.

Signed-off-by: xg.song <xg.song@venusource.com>
2015-08-17 08:25:22 +08:00
Stephen J Day
74080b7225 Provide yes man endpoint for inflexible load balancers
Certain load balancers, such as Amazon's Elastic Load Balancer, have a very
limited notion of health. While a properly configured and operational registry
should always return a 401 when hitting "/v2/", such load balancers cannot be
configured to treat this response code as healthy. This changeset makes "/"
always return a 200 response, unless the health checks have failed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-13 15:17:19 -07:00
Richard Scothern
77c6d9deb9 Merge pull request #869 from mssola/delete-documentation
docs: added the "delete" subsection in the configuration.md file
2015-08-13 11:22:15 -07:00
Miquel Sabaté
d07ae75f90 docs: added the "delete" subsection in the configuration.md file
Signed-off-by: Miquel Sabaté <msabate@suse.com>
2015-08-13 20:18:35 +02:00
Richard Scothern
44c7fb9e6e Merge pull request #864 from stevvooe/use-correct-manifest-link
registry/storage: use correct manifest link
2015-08-12 15:13:45 -07:00
Richard Scothern
fee9a9a469 Merge pull request #865 from dmp42/4.docs-2.1
Fix basic auth documentation
2015-08-12 15:00:21 -07:00
Olivier Gambier
5fb6116c2d Fix basic auth documentation
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-12 13:37:33 -07:00
Stephen Day
0be508fdc5 Merge pull request #862 from stevvooe/dont-comment-on-closed-issues
Forbid commenting on closed issues
2015-08-12 13:14:35 -07:00
Stephen J Day
06a098c632 Maintain manifest link compatibility
Unfortunately, the 2.1 releease has written manfiest links into the wrong
directory. This doesn't affect new 2.1 deployments but fixing this to be 2.0
backwards compatible has broken 2.1.0 compatibility. To ensure we have
compatibility between 2.0, 2.1.0 and future releases, we now check one of
several locations to identify a manifest link.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-12 13:11:13 -07:00
Stephen J Day
d00586de9f Use correct path for manifest revision path
Unfortunately, the refactor used the incorrect path for manifest links within a
repository. While this didn't stop the registry from working, it did break
compatibility with 2.0 deployments for manifest fetches.

Tests were added to ensure these are locked down to the appropriate paths.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-12 13:10:15 -07:00
Stephen J Day
e28be265d1 Forbid commenting on closed issues
Commenting on closed issues is rude and confusing. Most of the time, the issues
aren't actually the same. It is much better to file a duplicate and merge them
later when root cause has been identified than to have conflicting information
throughout a ticket.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-12 10:45:42 -07:00
Aaron Lehmann
f813c51857 Merge pull request #856 from dmp42/4.docs-2.1
Fix docs.docker.com
2015-08-11 17:50:10 -07:00
Olivier Gambier
633ac16856 Fix docs.docker.com
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-11 17:48:53 -07:00
Olivier Gambier
94296c1623 Merge pull request #847 from aaronlehmann/harmonize-config-paths
In the build container, change the config file location to /etc/docker/registry/config.yml
2015-08-11 16:54:44 -07:00
Aaron Lehmann
9c3bed6b88 Add a section to the config file for HTTP headers to add to responses
The example configuration files add X-Content-Type-Options: nosniff.

Add coverage in existing registry/handlers unit tests.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-11 16:41:38 -07:00
Olivier Gambier
9ca7921603 Merge pull request #848 from RichardScothern/error-type
Error handling fixes
2015-08-11 15:53:13 -07:00
Richard Scothern
7b0d831e6d Merge pull request #853 from dmp42/4.docs-2.1
Additional fixes
2015-08-11 14:32:57 -07:00
Richard Scothern
c78bfc5c59 Merge pull request #852 from ahmetalpbalkan/docs/typo
docs: Small typo fix in config.md
2015-08-11 14:31:35 -07:00
Ahmet Alp Balkan
c471d986d7 docs: Small typo fix
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-08-11 21:29:33 +00:00
Olivier Gambier
c198f8f279 Additional fixes
Couples of nits that where not addressed.

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-11 14:19:49 -07:00
Richard Scothern
776a4ffbe8 Change some incorrect error types in proxy stores from API errors to
distribution errors.  Fill in missing checks for mutations on a registry pull-through
cache.  Add unit tests and update documentation.

Also, give v2.ErrorCodeUnsupported an HTTP status code, previously it was
defaulting to 500, now its 405 Method Not Allowed.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-08-11 14:16:24 -07:00
Richard Scothern
528442c015 Merge pull request #831 from stevvooe/add-driver-health-check
Provide simple storage driver health check
2015-08-11 13:30:17 -07:00
Olivier Gambier
202b29d30b Merge pull request #851 from ahmetalpbalkan/docs/redirect
docs: Fix markdown in configuration.md
2015-08-11 13:26:31 -07:00
Richard Scothern
d187e765b6 Merge pull request #821 from aaronlehmann/docker-integration-fixes-3
Redesign integration testing to copy images to nested containers
2015-08-11 12:07:06 -07:00
Ahmet Alp Balkan
047877c079 docs: Fix markdown in configuration.md
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-08-11 19:03:53 +00:00
Stephen J Day
a68e081de4 Do not expose health check data in health handler
Because health check errors may expose sensitive data, we shouldn't expose the
details of the failure to clients. Instead, an error is returned to the client
with a hint about where they could find further information on why the service
is down.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-11 11:50:58 -07:00
Stephen J Day
f141480d98 Move common error codes to errcode package
Several error codes are generally useful but tied to the v2 specification
definitions. This change moves these error code definitions into the common
package for use by the health package, which is not tied to the v2 API.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-11 11:50:58 -07:00
Stephen J Day
6ba799b69e Provide simple storage driver health check
To ensure the ensure the web application is properly operating, we've added a
periodic health check for the storage driver. If the health check fails three
times in a row, the registry will serve 503 response status for any request
until the condition is resolved. The condition is reported in the response body
and via the /debug/health endpoint.

To ensure that all drivers will properly operate with this health check, a
function has been added to the driver testsuite.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-11 11:50:52 -07:00
Olivier Gambier
1d5b311fc4 Merge pull request #845 from gierschv/add-rados-root-omap
driver/rados: Create OMAP for root directory
2015-08-10 16:20:01 -07:00
Vincent Giersch
6c39af6708 fix(rados): Create OMAP for root directory
When using the RADOS driver, the hierarchy of the files is stored
in OMAPs, but the root OMAP was not created and a call to List("/")
was returning an error instead of returned the first level files
stored. This patches creates an OMAP for "/" and excludes the listed
directory from the list of files returned.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
2015-08-10 23:46:33 +02:00
Aaron Lehmann
1fcf90b9c4 In the build container, change the config file location to /etc/docker/registry/config.yml
This makes it consistent with the new official image.

Paths in the docs were updated in
34067d7d43.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-10 14:32:57 -07:00
Aaron Lehmann
143aae8bc5 Merge pull request #837 from dmp42/4.docs-2.1
[WIP] Updating documentation for 2.1
2015-08-10 14:01:52 -07:00
Olivier Gambier
34067d7d43 Documentation work
- move away insecure & self-signed
- introduce native basic auth
- move "down" nginx based authentication
- overall shortening / smoothing of "deploying" documentation

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-10 13:57:35 -07:00
Richard Scothern
3cecbf36d8 Merge pull request #839 from n1tr0g/master
Added missing auth type in the documentation
2015-08-10 10:39:19 -07:00
Dejan Golja
8221fd39bd Added missing auth type in the documentation
Signed-off-by: Dejan Golja <dejan@golja.org>
2015-08-11 00:11:17 +10:00
Olivier Gambier
4f7cb60190 Merge pull request #836 from stevvooe/fix-mailmap-scothern
Fix double entry in AUTHORS file
2015-08-07 14:01:16 -07:00
Stephen J Day
a77bbe7edf Fix double entry in AUTHORS file
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-07 14:00:07 -07:00
Olivier Gambier
81052c060c Merge pull request #835 from stevvooe/update-authors
Update authors file for 2.1 release
2015-08-07 13:44:39 -07:00
Stephen J Day
9a6d63eb39 Update authors file for 2.1 release
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-07 13:42:54 -07:00
Stephen Day
3f2517c9a6 Merge pull request #833 from stevvooe/remove-dist-tool
Remove dist tool from distribution
2015-08-07 12:45:44 -07:00
Richard Scothern
723c2d9944 Merge pull request #834 from stevvooe/rev-default-version
Rev base version to 2.1.0
2015-08-07 10:15:00 -07:00
Aaron Lehmann
1e15b6e001 Redesign integration testing to copy images to nested containers
Previously, the strategy for avoiding lots of rebuilding and repulling
for each Docker version being tested was to use a mountpoint to persist
/var/lib/docker. This was pretty broken, and may not be a reliable
strategy. This commit changes the scripts to instead build/pull images
outside the innermost container, and copy them to the final test
environment with docker save/docker load.

This requires a fair amount of changes, since run.sh must now
communicate with the Docker engine that was formerly started by
test_runner.sh. The code that starts this engine has been broken out to
run_engine.sh so that starting the engine and running the tests under it
can be done separately (with the images loaded in between these steps).

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-06 18:06:18 -07:00
Stephen J Day
bc5835ba2d Remove dist tool from distribution
We are headed in a different direction. The dist tool analog will not be a part
of this repository.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-06 17:35:36 -07:00
Stephen J Day
6cb14b79f1 Rev base version to 2.1.0
Now that we've tagged a release candidate, we can now say all builds are
"2.1.0+unknown" if the makefile is not used.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-06 17:21:03 -07:00
Stephen Day
ec77b836dd Merge pull request #830 from aaronlehmann/no-closenotifier-panic
Don't panic when a http.ResponseWriter does not implement CloseNotifier
2015-08-06 16:32:10 -07:00
Olivier Gambier
642fb631f4 Merge pull request #818 from dmp42/3.fix-build-docs
Fix relinker
2015-08-06 16:05:24 -07:00
Olivier Gambier
8a0e5a2a5e Merge pull request #829 from aaronlehmann/aliyun-oss-doc-styling
Fix lack of styling on Aliyun OSS documentation page
2015-08-06 16:04:47 -07:00
Aaron Lehmann
10f602b158 Don't panic when a http.ResponseWriter does not implement CloseNotifier
Instead, provide a variant of instrumentedResponseWriter that does not
implement CloseNotifier, and use that when necessary. In
copyFullPayload, log instead of panicing when we encounter something
that doesn't implement CloseNotifier.

This is more complicated than I'd like, but it's necessary because
instrumentedResponseWriter must not embed CloseNotifier unless there's
really a CloseNotifier to embed.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-06 15:50:54 -07:00
Aaron Lehmann
1670502413 Fix lack of styling on Aliyun OSS documentation page
This page was missing styling once exported to HTML. Adding a
<!--[metadata]> block similar to the ones the other *.md files have
appears to solve the problem.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-06 14:59:07 -07:00
Olivier Gambier
3ad11ff363 Fix relinker
This fixes several flaws in the link rewriter:
- broken links on the docs side
- multiple links on one line being mangled
- byzantine logic

Also generalize the logic of the Dockerfile so it is no longer specific to "registry" (env variable), which is a first step in making it possible to upstream all this into the base image and docs project.

Added a number of tests (test.md) to validate the link replacer behavior (against test.compare.md), and embedded the test so that the doc build will fail if the link replacer does not behave.

This is still sed, unfortunately.

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-08-04 23:02:34 -07:00
Stephen Day
a0c63372fa Merge pull request #779 from RichardScothern/pull-through-cache
Add pull through cache ability to the Registry.
2015-08-04 17:04:56 -07:00
Richard Scothern
94935f39bc Add pull through cache functionality to the Registry which can be configured
with a new `proxy` section in the configuration file.

Create a new registry type which delegates storage to a proxyBlobStore
and proxyManifestStore.  These stores will pull through data if not present
locally.  proxyBlobStore takes care not to write duplicate data to disk.

Add a scheduler to cleanup expired content. The scheduler runs as a background
goroutine.  When a blob or manifest is pulled through from the remote registry,
an entry is added to the scheduler with a TTL.  When the TTL expires the
scheduler calls a pre-specified function to remove the fetched resource.

Add token authentication to the registry middleware.  Get a token at startup
and preload the credential store with the username and password supplied in the
config file.

Allow resumable digest functionality to be disabled at runtime and disable
it when the registry is a pull through cache.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-08-04 16:09:55 -07:00
Stephen Day
e920231b17 Merge pull request #816 from mbentley/add-migrator
Added information about docker/migrator
2015-08-04 15:19:46 -07:00
Stephen Day
ef1c72b978 Merge pull request #805 from RichardScothern/811
Spelling corrections
2015-08-04 14:50:32 -07:00
Richard Scothern
78caa1fc93 Merge pull request #817 from aaronlehmann/docker-integration-fixes-2
Two more Docker integration test fixes
2015-08-04 13:57:27 -07:00
Aaron Lehmann
a6346d64a1 Two more Docker integration test fixes
- Upgrade docker-compose to 1.3.3 to work around
  https://github.com/docker/compose/issues/1314

- Change run.sh to run the Docker container in the foreground so that
  the exit code is propagated.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-04 13:28:03 -07:00
Matt Bentley
1150cfe449 Added information about docker/migrator
Signed-off-by: Matt Bentley <matt@docker.com>
2015-08-04 15:22:39 -04:00
Aaron Lehmann
3a414deddb Merge pull request #514 from denverdino/master
Storage Driver for Aliyun OSS
2015-08-04 12:01:02 -07:00
Richard Scothern
2e7ac2ec62 Merge pull request #815 from aaronlehmann/docker-integration-fixes
Two fixes for Docker integration tests
2015-08-04 11:56:15 -07:00
Aaron Lehmann
f7a11aa258 Two fixes for Docker integration tests
- Add a command line argument to run_multiversion.sh which has it start
  a docker daemon. This allows it to run directly inside an outermost
  docker-integration container instead of running outside all containers.

- Install Docker 1.7.1 intead of 1.7.0-rc1 in the docker-integration
  container. 1.7.0 has a bug that prevents "docker cp" from working
  properly.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-04 11:49:55 -07:00
Li Yi
d7d34a6c5b Add the OSS link
Change-Id: I940fb50f467ce6fc5fbdd9ceb0f9d848e422ced7
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-08-04 09:18:18 +08:00
Stephen Day
c4e553a655 Merge pull request #814 from aaronlehmann/test-response-writer
Make testResponseWriter's CloseNotify return an actual channel instead of nil
2015-08-03 15:59:42 -07:00
Aaron Lehmann
cca86be6bf Make testResponseWriter's CloseNotify return an actual channel instead of nil
This channel never gets written to, but this only means that the mock
ResponseWriter will never signal a premature disconnect.

Based on feedback from #763.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-03 15:10:14 -07:00
Richard Scothern
12db5fc16b Merge pull request #783 from aaronlehmann/multiple-version-integration-testing
Run integration tests with multiple Docker engine versions
2015-08-03 13:57:52 -07:00
Richard Scothern
db12c889e1 Merge pull request #763 from aaronlehmann/close-notifier
Use CloseNotifier to supress spurious HTTP 400 errors on early disconnect
2015-08-03 13:57:32 -07:00
Stephen Day
de7196071b Merge pull request #803 from duglin/FixBuild
Fix vet issue
2015-08-03 12:56:50 -07:00
Doug Davis
633eec0f91 Fix vet issue
registry/storage/blob_test.go:149: arg d for printf verb %s of wrong type: github.com/docker/distribution.Descriptor

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-08-03 12:24:34 -07:00
Stephen Day
7a46b772ad Merge pull request #811 from aaronlehmann/fix-vet
Fix go vet in CI
2015-08-03 12:19:55 -07:00
Stephen Day
f9181687f3 Merge pull request #812 from stevvooe/update-s3-dependency
Update s3 bindings
2015-08-03 12:05:34 -07:00
Richard Scothern
fb9662ad5a Add blob delete entry to api description and regenerate api.md
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-08-03 11:59:19 -07:00
Stephen J Day
7fc9e2112a Update s3 bindings
The S3 library has made a few fixes to the retry logic. Updating the bindings
accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-03 11:50:48 -07:00
Aaron Lehmann
5a37b46421 Fix go vet in CI
It's necessary to compile the code first; otherwise go vet silently
fails to load imports.

Fixes #807.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-03 11:29:10 -07:00
Aaron Lehmann
609efa79e4 Set the response code to 499 when a client disconnects during an upload
The response code isn't actually sent to the client, because the
connection has already closed by this point. But it causes the status
code to appear as 499 in the logs instead of 0.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:39:30 -07:00
Aaron Lehmann
9c58954a6e Factor CloseNotifier use into a new function
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:30:26 -07:00
Aaron Lehmann
6cb5670ba5 Use CloseNotifier to supress spurious HTTP 400 errors on early disconnect
When a client disconnects without completing a HTTP request, we were
attempting to process the partial request, which usually leads to a 400
error. These errors can pollute the logs and make it more difficult to
track down real bugs.

This change uses CloseNotifier to detect disconnects. In combination
with checking Content-Length, we can detect a disconnect before sending
the full payload, and avoid logging a 400 error.

This logic is only applied to PUT, POST, and PATCH endpoints, as these
are the places where disconnects during a request are most likely to
happen.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:30:26 -07:00
Aaron Lehmann
1788ae4870 Update github.com/gorilla/handlers dependency
Support for exposing the CloseNotifier interface was just recently added
to its logging handler wrappers. This is needed for #597.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:30:26 -07:00
Richard Scothern
bffa20d6bb Spelling corrections
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-31 16:43:11 -07:00
Richard Scothern
0b3b55e723 Merge pull request #801 from alexwlchan/master
Fix a few typos in the docs
2015-07-31 09:15:35 -07:00
Alex Chan
51bd34eaed Fix a few typos in the docs
Signed-off-by: Alex Chan <alex.chan@metaswitch.com>
2015-07-31 13:36:43 +01:00
tgic
72c794a810 remove unused code and fix todo format
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
tgic
3eaab7da95 add link to Aliyun OSS doc
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
tgic
e53d99a837 fix goimports
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
tgic
806b890cd6 add build tag include_oss to circle.yml 2015-07-31 13:22:56 +08:00
tgic
0058b08eeb add include_oss build tag
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
Li Yi
708ad28114 Update the comment for the consistency model
Change-Id: Iee49afeda1c11d6af8c0f26c96d8ccc328c22757
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
Li Yi
539c7f5311 Update the comments for consistence model
Change-Id: I161522ee51f247fb17e42844b3699bd9031e34e8
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
76174ad57e update doc add endpoint 2015-07-31 13:22:55 +08:00
tgic
8e5c901a26 fix testcase TestReadStreamWithOffset incompatible with oss
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
2a8535626f fix oss: got 403 in TestContinueStreamAppendLarge
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
9627c6c5a5 godep update oss sdk
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
Li Yi
813543d6e3 Update the OSS test case for latest code change
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
3a240de22e check access key and secret before run 2015-07-31 13:22:55 +08:00
tgic
ee9971a5af update godep 2015-07-31 13:22:54 +08:00
tgic
3d30cb38f6 add endpoint support 2015-07-31 13:22:54 +08:00
Li Yi
38b23c8dff Update Godeps for Aliyun OSS
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Li Yi
c3b42db014 Add the secure access with HTTPS
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Li Yi
b8a276f2db Fix the warning of golint
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Li Yi
9e4975d8ff Support OSS driver
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Aaron Lehmann
1368e2ac53 Change run_multiversion.sh to get Docker engine versions from dind container
This is much better than downloading the builds with curl, because
Docker is intelligent about only pulling what it needs.

We use tagged versions of dind for the released version, and dind-master
for a master build.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-30 15:39:19 -07:00
Aaron Lehmann
e564231f58 Run integration tests with multiple Docker engine versions
Add a run_multiversion.sh script that downloads a variety of Docker
engines and runs the integration tests against each of them.

Skip some of the tests when running with Docker 1.6.0. It has a bug that
prevents these tests from completing successfully.

Fixes #751

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-30 15:39:19 -07:00
1950 changed files with 529582 additions and 59306 deletions

View file

@ -1,38 +0,0 @@
image: dmp42/go:stable
script:
# To be spoofed back into the test image
- go get github.com/modocache/gover
- go get -t ./...
# Go fmt
- test -z "$(gofmt -s -l -w . | tee /dev/stderr)"
# Go lint
- test -z "$(golint ./... | tee /dev/stderr)"
# Go vet
- go vet ./...
# Go test
- go test -v -race -cover ./...
# Helper to concatenate reports
- gover
# Send to coverall
- goveralls -service drone.io -coverprofile=gover.coverprofile -repotoken {{COVERALLS_TOKEN}}
# Do we want these as well?
# - go get code.google.com/p/go.tools/cmd/goimports
# - test -z "$(goimports -l -w ./... | tee /dev/stderr)"
# http://labix.org/gocheck
notify:
email:
recipients:
- distribution@docker.com
slack:
team: docker
channel: "#dt"
username: mom
token: {{SLACK_TOKEN}}
on_success: true
on_failure: true

3
.github/CODE_OF_CONDUCT.md vendored Normal file
View file

@ -0,0 +1,3 @@
## Docker Distribution Community Code of Conduct
Docker Distribution follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).

1
.gitignore vendored
View file

@ -35,3 +35,4 @@ bin/*
# Editor/IDE specific files. # Editor/IDE specific files.
*.sublime-project *.sublime-project
*.sublime-workspace *.sublime-workspace
.idea/*

20
.golangci.yml Normal file
View file

@ -0,0 +1,20 @@
linters:
enable:
- structcheck
- varcheck
- staticcheck
- unconvert
- gofmt
- goimports
- golint
- ineffassign
- vet
- unused
- misspell
disable:
- errcheck
run:
deadline: 2m
skip-dirs:
- vendor

View file

@ -1,6 +1,32 @@
Stephen J Day <stephen.day@docker.com> Stephen Day <stevvooe@users.noreply.github.com> Stephen J Day <stephen.day@docker.com> Stephen Day <stevvooe@users.noreply.github.com>
Stephen J Day <stephen.day@docker.com> Stephen Day <stevvooe@gmail.com> Stephen J Day <stephen.day@docker.com> Stephen Day <stevvooe@gmail.com>
Olivier Gambier <olivier@docker.com> Olivier Gambier <dmp42@users.noreply.github.com> Olivier Gambier <olivier@docker.com> Olivier Gambier <dmp42@users.noreply.github.com>
Brian Bland <brian.bland@docker.com> Brian Bland <r4nd0m1n4t0r@gmail.com> Brian Bland <brian.bland@docker.com> Brian Bland <r4nd0m1n4t0r@gmail.com>
Josh Hawn <josh.hawn@docker.com> Josh Hawn <jlhawn@berkeley.edu> Brian Bland <brian.bland@docker.com> Brian Bland <brian.t.bland@gmail.com>
Josh Hawn <josh.hawn@docker.com> Josh Hawn <jlhawn@berkeley.edu>
Richard Scothern <richard.scothern@docker.com> Richard <richard.scothern@gmail.com> Richard Scothern <richard.scothern@docker.com> Richard <richard.scothern@gmail.com>
Richard Scothern <richard.scothern@docker.com> Richard Scothern <richard.scothern@gmail.com>
Andrew Meredith <andymeredith@gmail.com> Andrew Meredith <kendru@users.noreply.github.com>
harche <p.harshal@gmail.com> harche <harche@users.noreply.github.com>
Jessie Frazelle <jessie@docker.com> <jfrazelle@users.noreply.github.com>
Sharif Nassar <sharif@mrwacky.com> Sharif Nassar <mrwacky42@users.noreply.github.com>
Sven Dowideit <SvenDowideit@home.org.au> Sven Dowideit <SvenDowideit@users.noreply.github.com>
Vincent Giersch <vincent.giersch@ovh.net> Vincent Giersch <vincent@giersch.fr>
davidli <wenquan.li@hp.com> davidli <wenquan.li@hpe.com>
Omer Cohen <git@omer.io> Omer Cohen <git@omerc.net>
Eric Yang <windfarer@gmail.com> Eric Yang <Windfarer@users.noreply.github.com>
Nikita Tarasov <nikita@mygento.ru> Nikita <luckyraul@users.noreply.github.com>
Yu Wang <yuwa@microsoft.com> yuwaMSFT2 <yuwa@microsoft.com>
Yu Wang <yuwa@microsoft.com> Yu Wang (UC) <yuwa@microsoft.com>
Olivier Gambier <olivier@docker.com> dmp <dmp@loaner.local>
Olivier Gambier <olivier@docker.com> Olivier <o+github@gambier.email>
Olivier Gambier <olivier@docker.com> Olivier <dmp42@users.noreply.github.com>
Elsan Li 李楠 <elsanli@tencent.com> elsanli(李楠) <elsanli@tencent.com>
Rui Cao <ruicao@alauda.io> ruicao <ruicao@alauda.io>
Gwendolynne Barr <gwendolynne.barr@docker.com> gbarr01 <gwendolynne.barr@docker.com>
Haibing Zhou 周海兵 <zhouhaibing089@gmail.com> zhouhaibing089 <zhouhaibing089@gmail.com>
Feng Honglin <tifayuki@gmail.com> tifayuki <tifayuki@gmail.com>
Helen Xie <xieyulin821@harmonycloud.cn> Helen-xie <xieyulin821@harmonycloud.cn>
Mike Brown <brownwm@us.ibm.com> Mike Brown <mikebrow@users.noreply.github.com>
Manish Tomar <manish.tomar@docker.com> Manish Tomar <manishtomar@users.noreply.github.com>
Sakeven Jiang <jc5930@sina.cn> sakeven <jc5930@sina.cn>

56
.travis.yml Normal file
View file

@ -0,0 +1,56 @@
dist: bionic
sudo: required
# setup travis so that we can run containers for integration tests
services:
- docker
jobs:
include:
- arch: amd64
- arch: s390x
language: go
go:
- "1.14.x"
go_import_path: github.com/docker/distribution
addons:
apt:
packages:
- python-minimal
env:
- TRAVIS_GOOS=linux DOCKER_BUILDTAGS="include_oss include_gcs" TRAVIS_CGO_ENABLED=1
before_install:
- uname -r
- sudo apt-get -q update
install:
- cd /tmp && go get -u github.com/vbatts/git-validation
# TODO: Add enforcement of license
# - go get -u github.com/kunalkushwaha/ltag
- cd $TRAVIS_BUILD_DIR
script:
- export GOOS=$TRAVIS_GOOS
- export CGO_ENABLED=$TRAVIS_CGO_ENABLED
- DCO_VERBOSITY=-q script/validate/dco
- GOOS=linux GO111MODULE=on script/setup/install-dev-tools
- script/validate/vendor
- go build -i .
- make check
- make build
- make binaries
# Currently takes too long
#- if [ "$GOOS" = "linux" ]; then make test-race ; fi
- if [ "$GOOS" = "linux" ]; then make coverage ; fi
after_success:
- bash <(curl -s https://codecov.io/bash) -F linux
before_deploy:
# Run tests with storage driver configurations

61
AUTHORS
View file

@ -1,61 +0,0 @@
Adam Enger <adamenger@gmail.com>
Adrian Mouat <adrian.mouat@gmail.com>
Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Alex Elman <aelman@indeed.com>
Amy Lindburg <amy.lindburg@docker.com>
Andrey Kostov <kostov.andrey@gmail.com>
Andy Goldstein <agoldste@redhat.com>
Anton Tiurin <noxiouz@yandex.ru>
Antonio Mercado <amercado@thinknode.com>
Arnaud Porterie <arnaud.porterie@docker.com>
BadZen <dave.trombley@gmail.com>
Ben Firshman <ben@firshman.co.uk>
bin liu <liubin0329@gmail.com>
Brian Bland <brian.bland@docker.com>
burnettk <burnettk@gmail.com>
Daisuke Fujita <dtanshi45@gmail.com>
Dave Trombley <dave.trombley@gmail.com>
David Lawrence <david.lawrence@docker.com>
David Xia <dxia@spotify.com>
Derek McGowan <derek@mcgstyle.net>
Diogo Mónica <diogo.monica@gmail.com>
Donald Huang <don.hcd@gmail.com>
Doug Davis <dug@us.ibm.com>
Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
Henri Gomez <henri.gomez@gmail.com>
Hu Keping <hukeping@huawei.com>
Ian Babrou <ibobrik@gmail.com>
Jeff Nickoloff <jeff@allingeek.com>
Jessie Frazelle <jfrazelle@users.noreply.github.com>
Jordan Liggitt <jliggitt@redhat.com>
Josh Hawn <josh.hawn@docker.com>
Julien Fernandez <julien.fernandez@gmail.com>
Kelsey Hightower <kelsey.hightower@gmail.com>
Kenneth Lim <kennethlimcp@gmail.com>
Mary Anthony <mary@docker.com>
Matt Robenolt <matt@ydekproductions.com>
Michael Prokop <mika@grml.org>
moxiegirl <mary@docker.com>
Nathan Sullivan <nathan@nightsys.net>
Nghia Tran <tcnghia@gmail.com>
Oilbeater <liumengxinfly@gmail.com>
Olivier Gambier <olivier@docker.com>
Philip Misiowiec <philip@atlashealth.com>
Richard Scothern <richard.scothern@docker.com>
Richard Scothern <richard.scothern@gmail.com>
Sebastiaan van Stijn <github@gone.nl>
Shawn Falkner-Horine <dreadpirateshawn@gmail.com>
Shreyas Karnik <karnik.shreyas@gmail.com>
Simon Thulbourn <simon+github@thulbourn.com>
Spencer Rinehart <anubis@overthemonkey.com>
Stephen J Day <stephen.day@docker.com>
Thomas Sjögren <konstruktoid@users.noreply.github.com>
Tianon Gravi <admwiggin@gmail.com>
Tibor Vass <teabee89@gmail.com>
Vincent Batts <vbatts@redhat.com>
Vincent Demeester <vincent@sbr.pm>
Vincent Giersch <vincent.giersch@ovh.net>
W. Trevor King <wking@tremily.us>
xiekeyang <xiekeyang@huawei.com>
Yann ROBERT <yann.robert@anantaplex.fr>
yuzou <zouyu7@huawei.com>

117
BUILDING.md Normal file
View file

@ -0,0 +1,117 @@
# Building the registry source
## Use-case
This is useful if you intend to actively work on the registry.
### Alternatives
Most people should use the [official Registry docker image](https://hub.docker.com/r/library/registry/).
People looking for advanced operational use cases might consider rolling their own image with a custom Dockerfile inheriting `FROM registry:2`.
OS X users who want to run natively can do so following [the instructions here](https://github.com/docker/docker.github.io/blob/master/registry/recipes/osx-setup-guide.md).
### Gotchas
You are expected to know your way around with go & git.
If you are a casual user with no development experience, and no preliminary knowledge of go, building from source is probably not a good solution for you.
## Build the development environment
The first prerequisite of properly building distribution targets is to have a Go
development environment setup. Please follow [How to Write Go Code](https://golang.org/doc/code.html)
for proper setup. If done correctly, you should have a GOROOT and GOPATH set in the
environment.
If a Go development environment is setup, one can use `go get` to install the
`registry` command from the current latest:
go get github.com/docker/distribution/cmd/registry
The above will install the source repository into the `GOPATH`.
Now create the directory for the registry data (this might require you to set permissions properly)
mkdir -p /var/lib/registry
... or alternatively `export REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/somewhere` if you want to store data into another location.
The `registry`
binary can then be run with the following:
$ $GOPATH/bin/registry --version
$GOPATH/bin/registry github.com/docker/distribution v2.0.0-alpha.1+unknown
> __NOTE:__ While you do not need to use `go get` to checkout the distribution
> project, for these build instructions to work, the project must be checked
> out in the correct location in the `GOPATH`. This should almost always be
> `$GOPATH/src/github.com/docker/distribution`.
The registry can be run with the default config using the following
incantation:
$ $GOPATH/bin/registry serve $GOPATH/src/github.com/docker/distribution/cmd/registry/config-example.yml
INFO[0000] endpoint local-5003 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] endpoint local-8083 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] listening on :5000 app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] debug server listening localhost:5001
If it is working, one should see the above log messages.
### Repeatable Builds
For the full development experience, one should `cd` into
`$GOPATH/src/github.com/docker/distribution`. From there, the regular `go`
commands, such as `go test`, should work per package (please see
[Developing](#developing) if they don't work).
A `Makefile` has been provided as a convenience to support repeatable builds.
Please install the following into `GOPATH` for it to work:
go get github.com/golang/lint/golint
Once these commands are available in the `GOPATH`, run `make` to get a full
build:
$ make
+ clean
+ fmt
+ vet
+ lint
+ build
github.com/docker/docker/vendor/src/code.google.com/p/go/src/pkg/archive/tar
github.com/sirupsen/logrus
github.com/docker/libtrust
...
github.com/yvasiyarov/gorelic
github.com/docker/distribution/registry/handlers
github.com/docker/distribution/cmd/registry
+ test
...
ok github.com/docker/distribution/digest 7.875s
ok github.com/docker/distribution/manifest 0.028s
ok github.com/docker/distribution/notifications 17.322s
? github.com/docker/distribution/registry [no test files]
ok github.com/docker/distribution/registry/api/v2 0.101s
? github.com/docker/distribution/registry/auth [no test files]
ok github.com/docker/distribution/registry/auth/silly 0.011s
...
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry-api-descriptor-template
+ binaries
The above provides a repeatable build using the contents of the vendor
directory. This includes formatting, vetting, linting, building,
testing and generating tagged binaries. We can verify this worked by running
the registry binary generated in the "./bin" directory:
$ ./bin/registry --version
./bin/registry github.com/docker/distribution v2.0.0-alpha.2-80-g16d8b2c.m
### Optional build tags
Optional [build tags](http://golang.org/pkg/go/build/) can be provided using
the environment variable `DOCKER_BUILDTAGS`.

View file

@ -1,14 +1,15 @@
# Contributing to the registry # Contributing to the registry
## Before reporting an issue... ## Before reporting an issue...
### If your problem is with... ### If your problem is with...
- automated builds - automated builds or your [Docker Hub](https://hub.docker.com/) account
- your account on the [Docker Hub](https://hub.docker.com/) - Report it to [Hub Support](https://hub.docker.com/support/)
- any other [Docker Hub](https://hub.docker.com/) issue - Distributions of Docker for desktop or Linux
- Report [Mac Desktop issues](https://github.com/docker/for-mac)
Then please do not report your issue here - you should instead report it to [https://support.docker.com](https://support.docker.com) - Report [Windows Desktop issues](https://github.com/docker/for-win)
- Report [Linux issues](https://github.com/docker/for-linux)
### If you... ### If you...
@ -16,10 +17,16 @@ Then please do not report your issue here - you should instead report it to [htt
- can't figure out something - can't figure out something
- are not sure what's going on or what your problem is - are not sure what's going on or what your problem is
Then please do not open an issue here yet - you should first try one of the following support forums: Please ask first in the #distribution channel on Docker community slack.
[Click here for an invite to Docker community slack](https://dockr.ly/slack)
- irc: #docker-distribution on freenode ### Reporting security issues
- mailing-list: <distribution@dockerproject.org> or https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution
The Docker maintainers take security seriously. If you discover a security
issue, please bring it to their attention right away!
Please **DO NOT** file a public issue, instead send your report privately to
[security@docker.com](mailto:security@docker.com).
## Reporting an issue properly ## Reporting an issue properly
@ -27,12 +34,13 @@ By following these simple rules you will get better and faster feedback on your
- search the bugtracker for an already reported issue - search the bugtracker for an already reported issue
### If you found an issue that describes your problem: ### If you found an issue that describes your problem:
- please read other user comments first, and confirm this is the same issue: a given error condition might be indicative of different problems - you may also find a workaround in the comments - please read other user comments first, and confirm this is the same issue: a given error condition might be indicative of different problems - you may also find a workaround in the comments
- please refrain from adding "same thing here" or "+1" comments - please refrain from adding "same thing here" or "+1" comments
- you don't need to comment on an issue to get notified of updates: just hit the "subscribe" button - you don't need to comment on an issue to get notified of updates: just hit the "subscribe" button
- comment if you have some new, technical and relevant information to add to the case - comment if you have some new, technical and relevant information to add to the case
- __DO NOT__ comment on closed issues or merged PRs. If you think you have a related problem, open up a new issue and reference the PR or issue.
### If you have not found an existing issue that describes your problem: ### If you have not found an existing issue that describes your problem:
@ -42,7 +50,7 @@ By following these simple rules you will get better and faster feedback on your
2. copy the output of: 2. copy the output of:
- `docker version` - `docker version`
- `docker info` - `docker info`
- `docker exec <registry-container> registry -version` - `docker exec <registry-container> registry --version`
3. copy the command line you used to launch your Registry 3. copy the command line you used to launch your Registry
4. restart your docker daemon in debug mode (add `-D` to the daemon launch arguments) 4. restart your docker daemon in debug mode (add `-D` to the daemon launch arguments)
5. reproduce your problem and get your docker daemon logs showing the error 5. reproduce your problem and get your docker daemon logs showing the error
@ -50,90 +58,72 @@ By following these simple rules you will get better and faster feedback on your
7. provide any relevant detail about your specific Registry configuration (e.g., storage backend used) 7. provide any relevant detail about your specific Registry configuration (e.g., storage backend used)
8. indicate if you are using an enterprise proxy, Nginx, or anything else between you and your Registry 8. indicate if you are using an enterprise proxy, Nginx, or anything else between you and your Registry
## Contributing a patch for a known bug, or a small correction ## Contributing Code
Contributions should be made via pull requests. Pull requests will be reviewed
by one or more maintainers or reviewers and merged when acceptable.
You should follow the basic GitHub workflow: You should follow the basic GitHub workflow:
1. fork 1. Use your own [fork](https://help.github.com/en/articles/about-forks)
2. commit a change 2. Create your [change](https://github.com/containerd/project/blob/master/CONTRIBUTING.md#successful-changes)
3. make sure the tests pass 3. Test your code
4. PR 4. [Commit](https://github.com/containerd/project/blob/master/CONTRIBUTING.md#commit-messages) your work, always [sign your commits](https://github.com/containerd/project/blob/master/CONTRIBUTING.md#commit-messages)
5. Push your change to your fork and create a [Pull Request](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork)
Additionally, you must [sign your commits](https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work). It's very simple: Refer to [containerd's contribution guide](https://github.com/containerd/project/blob/master/CONTRIBUTING.md#successful-changes)
for tips on creating a successful contribution.
- configure your name with git: `git config user.name "Real Name" && git config user.email mail@example.com` ## Sign your work
- sign your commits using `-s`: `git commit -s -m "My commit"`
Some simple rules to ensure quick merge: The sign-off is a simple line at the end of the explanation for the patch. Your
signature certifies that you wrote the patch or otherwise have the right to pass
it on as an open-source patch. The rules are pretty simple: if you can certify
the below (from [developercertificate.org](http://developercertificate.org/)):
- clearly point to the issue(s) you want to fix in your PR comment (e.g., `closes #12345`) ```
- prefer multiple (smaller) PRs addressing individual issues over a big one trying to address multiple issues at once Developer Certificate of Origin
- if you need to amend your PR following comments, please squash instead of adding more commits Version 1.1
## Contributing new features Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA
You are heavily encouraged to first discuss what you want to do. You can do so on the irc channel, or by opening an issue that clearly describes the use case you want to fulfill, or the problem you are trying to solve. Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
If this is a major new feature, you should then submit a proposal that describes your technical solution and reasoning. Developer's Certificate of Origin 1.1
If you did discuss it first, this will likely be greenlighted very fast. It's advisable to address all feedback on this proposal before starting actual work.
Then you should submit your implementation, clearly linking to the issue (and possible proposal). By making a contribution to this project, I certify that:
Your PR will be reviewed by the community, then ultimately by the project maintainers, before being merged. (a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
It's mandatory to: (b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
- interact respectfully with other community members and maintainers - more generally, you are expected to abide by the [Docker community rules](https://github.com/docker/docker/blob/master/CONTRIBUTING.md#docker-community-guidelines) (c) The contribution was provided directly to me by some other
- address maintainers' comments and modify your submission accordingly person who certified (a), (b) or (c) and I have not modified
- write tests for any new code it.
Complying to these simple rules will greatly accelerate the review process, and will ensure you have a pleasant experience in contributing code to the Registry. (d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
```
Have a look at a great, succesful contribution: the [Ceph driver PR](https://github.com/docker/distribution/pull/443) Then you just add a line to every git commit message:
## Coding Style Signed-off-by: Joe Smith <joe.smith@email.com>
Unless explicitly stated, we follow all coding guidelines from the Go Use your real name (sorry, no pseudonyms or anonymous contributions.)
community. While some of these standards may seem arbitrary, they somehow seem
to result in a solid, consistent codebase.
It is possible that the code base does not currently comply with these If you set your `user.name` and `user.email` git configs, you can sign your
guidelines. We are not looking for a massive PR that fixes this, since that commit automatically with `git commit -s`.
goes against the spirit of the guidelines. All new contributions should make a
best effort to clean up and make the code base better than they left it.
Obviously, apply your best judgement. Remember, the goal here is to make the
code base easier for humans to navigate and understand. Always keep that in
mind when nudging others to comply.
The rules:
1. All code should be formatted with `gofmt -s`.
2. All code should pass the default levels of
[`golint`](https://github.com/golang/lint).
3. All code should follow the guidelines covered in [Effective
Go](http://golang.org/doc/effective_go.html) and [Go Code Review
Comments](https://github.com/golang/go/wiki/CodeReviewComments).
4. Comment the code. Tell us the why, the history and the context.
5. Document _all_ declarations and methods, even private ones. Declare
expectations, caveats and anything else that may be important. If a type
gets exported, having the comments already there will ensure it's ready.
6. Variable name length should be proportional to its context and no longer.
`noCommaALongVariableNameLikeThisIsNotMoreClearWhenASimpleCommentWouldDo`.
In practice, short methods will have short variable names and globals will
have longer names.
7. No underscores in package names. If you need a compound name, step back,
and re-examine why you need a compound name. If you still think you need a
compound name, lose the underscore.
8. No utils or helpers packages. If a function is not general enough to
warrant its own package, it has not been written generally enough to be a
part of a util package. Just leave it unexported and well-documented.
9. All tests should run with `go test` and outside tooling should not be
required. No, we don't need another unit testing framework. Assertion
packages are acceptable if they provide _real_ incremental value.
10. Even though we call these "rules" above, they are actually just
guidelines. Since you've read all the rules, you now know that.
If you are having trouble getting into the mood of idiomatic Go, we recommend
reading through [Effective Go](http://golang.org/doc/effective_go.html). The
[Go Blog](http://blog.golang.org/) is also a great resource. Drinking the
kool-aid is a lot easier than going thirsty.

View file

@ -1,19 +1,31 @@
FROM golang:1.4 ARG GO_VERSION=1.13.8
RUN apt-get update && \ FROM golang:${GO_VERSION}-alpine3.11 AS build
apt-get install -y librados-dev apache2-utils && \
rm -rf /var/lib/apt/lists/*
ENV DISTRIBUTION_DIR /go/src/github.com/docker/distribution ENV DISTRIBUTION_DIR /go/src/github.com/docker/distribution
ENV GOPATH $DISTRIBUTION_DIR/Godeps/_workspace:$GOPATH ENV BUILDTAGS include_oss include_gcs
ENV DOCKER_BUILDTAGS include_rados
ARG GOOS=linux
ARG GOARCH=amd64
ARG GOARM=6
ARG VERSION
ARG REVISION
RUN set -ex \
&& apk add --no-cache make git file
WORKDIR $DISTRIBUTION_DIR WORKDIR $DISTRIBUTION_DIR
COPY . $DISTRIBUTION_DIR COPY . $DISTRIBUTION_DIR
COPY cmd/registry/config-dev.yml $DISTRIBUTION_DIR/cmd/registry/config.yml RUN CGO_ENABLED=0 make PREFIX=/go clean binaries && file ./bin/registry | grep "statically linked"
RUN make PREFIX=/go clean binaries
FROM alpine:3.11
RUN set -ex \
&& apk add --no-cache ca-certificates apache2-utils
COPY cmd/registry/config-dev.yml /etc/docker/registry/config.yml
COPY --from=build /go/src/github.com/docker/distribution/bin/registry /bin/registry
VOLUME ["/var/lib/registry"] VOLUME ["/var/lib/registry"]
EXPOSE 5000 EXPOSE 5000
ENTRYPOINT ["registry"] ENTRYPOINT ["registry"]
CMD ["cmd/registry/config.yml"] CMD ["serve", "/etc/docker/registry/config.yml"]

144
GOVERNANCE.md Normal file
View file

@ -0,0 +1,144 @@
# docker/distribution Project Governance
Docker distribution abides by the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
For specific guidance on practical contribution steps please
see our [CONTRIBUTING.md](./CONTRIBUTING.md) guide.
## Maintainership
There are different types of maintainers, with different responsibilities, but
all maintainers have 3 things in common:
1) They share responsibility in the project's success.
2) They have made a long-term, recurring time investment to improve the project.
3) They spend that time doing whatever needs to be done, not necessarily what
is the most interesting or fun.
Maintainers are often under-appreciated, because their work is harder to appreciate.
It's easy to appreciate a really cool and technically advanced feature. It's harder
to appreciate the absence of bugs, the slow but steady improvement in stability,
or the reliability of a release process. But those things distinguish a good
project from a great one.
## Reviewers
A reviewer is a core role within the project.
They share in reviewing issues and pull requests and their LGTM counts towards the
required LGTM count to merge a code change into the project.
Reviewers are part of the organization but do not have write access.
Becoming a reviewer is a core aspect in the journey to becoming a maintainer.
## Adding maintainers
Maintainers are first and foremost contributors that have shown they are
committed to the long term success of a project. Contributors wanting to become
maintainers are expected to be deeply involved in contributing code, pull
request review, and triage of issues in the project for more than three months.
Just contributing does not make you a maintainer, it is about building trust
with the current maintainers of the project and being a person that they can
depend on and trust to make decisions in the best interest of the project.
Periodically, the existing maintainers curate a list of contributors that have
shown regular activity on the project over the prior months. From this list,
maintainer candidates are selected and proposed in a pull request or a
maintainers communication channel.
After a candidate has been announced to the maintainers, the existing
maintainers are given five business days to discuss the candidate, raise
objections and cast their vote. Votes may take place on the communication
channel or via pull request comment. Candidates must be approved by at least 66%
of the current maintainers by adding their vote on the mailing list. The
reviewer role has the same process but only requires 33% of current maintainers.
Only maintainers of the repository that the candidate is proposed for are
allowed to vote.
If a candidate is approved, a maintainer will contact the candidate to invite
the candidate to open a pull request that adds the contributor to the
MAINTAINERS file. The voting process may take place inside a pull request if a
maintainer has already discussed the candidacy with the candidate and a
maintainer is willing to be a sponsor by opening the pull request. The candidate
becomes a maintainer once the pull request is merged.
## Stepping down policy
Life priorities, interests, and passions can change. If you're a maintainer but
feel you must remove yourself from the list, inform other maintainers that you
intend to step down, and if possible, help find someone to pick up your work.
At the very least, ensure your work can be continued where you left off.
After you've informed other maintainers, create a pull request to remove
yourself from the MAINTAINERS file.
## Removal of inactive maintainers
Similar to the procedure for adding new maintainers, existing maintainers can
be removed from the list if they do not show significant activity on the
project. Periodically, the maintainers review the list of maintainers and their
activity over the last three months.
If a maintainer has shown insufficient activity over this period, a neutral
person will contact the maintainer to ask if they want to continue being
a maintainer. If the maintainer decides to step down as a maintainer, they
open a pull request to be removed from the MAINTAINERS file.
If the maintainer wants to remain a maintainer, but is unable to perform the
required duties they can be removed with a vote of at least 66% of the current
maintainers. In this case, maintainers should first propose the change to
maintainers via the maintainers communication channel, then open a pull request
for voting. The voting period is five business days. The voting pull request
should not come as a surpise to any maintainer and any discussion related to
performance must not be discussed on the pull request.
## How are decisions made?
Docker distribution is an open-source project with an open design philosophy.
This means that the repository is the source of truth for EVERY aspect of the
project, including its philosophy, design, road map, and APIs. *If it's part of
the project, it's in the repo. If it's in the repo, it's part of the project.*
As a result, all decisions can be expressed as changes to the repository. An
implementation change is a change to the source code. An API change is a change
to the API specification. A philosophy change is a change to the philosophy
manifesto, and so on.
All decisions affecting distribution, big and small, follow the same 3 steps:
* Step 1: Open a pull request. Anyone can do this.
* Step 2: Discuss the pull request. Anyone can do this.
* Step 3: Merge or refuse the pull request. Who does this depends on the nature
of the pull request and which areas of the project it affects.
## Helping contributors with the DCO
The [DCO or `Sign your work`](./CONTRIBUTING.md#sign-your-work)
requirement is not intended as a roadblock or speed bump.
Some contributors are not as familiar with `git`, or have used a web
based editor, and thus asking them to `git commit --amend -s` is not the best
way forward.
In this case, maintainers can update the commits based on clause (c) of the DCO.
The most trivial way for a contributor to allow the maintainer to do this, is to
add a DCO signature in a pull requests's comment, or a maintainer can simply
note that the change is sufficiently trivial that it does not substantially
change the existing contribution - i.e., a spelling change.
When you add someone's DCO, please also add your own to keep a log.
## I'm a maintainer. Should I make pull requests too?
Yes. Nobody should ever push to master directly. All changes should be
made through a pull request.
## Conflict Resolution
If you have a technical dispute that you feel has reached an impasse with a
subset of the community, any contributor may open an issue, specifically
calling for a resolution vote of the current core maintainers to resolve the
dispute. The same voting quorums required (2/3) for adding and removing
maintainers will apply to conflict resolution.

127
Godeps/Godeps.json generated
View file

@ -1,127 +0,0 @@
{
"ImportPath": "github.com/docker/distribution",
"GoVersion": "go1.4.2",
"Packages": [
"./..."
],
"Deps": [
{
"ImportPath": "github.com/AdRoll/goamz/aws",
"Rev": "cc210f45dcb9889c2769a274522be2bf70edfb99"
},
{
"ImportPath": "github.com/AdRoll/goamz/cloudfront",
"Rev": "cc210f45dcb9889c2769a274522be2bf70edfb99"
},
{
"ImportPath": "github.com/AdRoll/goamz/s3",
"Rev": "cc210f45dcb9889c2769a274522be2bf70edfb99"
},
{
"ImportPath": "github.com/Azure/azure-sdk-for-go/storage",
"Rev": "97d9593768bbbbd316f9c055dfc5f780933cd7fc"
},
{
"ImportPath": "github.com/Sirupsen/logrus",
"Comment": "v0.7.3",
"Rev": "55eb11d21d2a31a3cc93838241d04800f52e823d"
},
{
"ImportPath": "github.com/bugsnag/bugsnag-go",
"Comment": "v1.0.2-5-gb1d1530",
"Rev": "b1d153021fcd90ca3f080db36bec96dc690fb274"
},
{
"ImportPath": "github.com/bugsnag/osext",
"Rev": "0dd3f918b21bec95ace9dc86c7e70266cfc5c702"
},
{
"ImportPath": "github.com/bugsnag/panicwrap",
"Rev": "e5f9854865b9778a45169fc249e99e338d4d6f27"
},
{
"ImportPath": "github.com/codegangsta/cli",
"Comment": "1.2.0-66-g6086d79",
"Rev": "6086d7927ec35315964d9fea46df6c04e6d697c1"
},
{
"ImportPath": "github.com/docker/docker/pkg/tarsum",
"Comment": "v1.4.1-3932-gb63ec6e",
"Rev": "b63ec6e4b1f6f5c77a6a74a52fcea9564538c575"
},
{
"ImportPath": "github.com/docker/libtrust",
"Rev": "fa567046d9b14f6aa788882a950d69651d230b21"
},
{
"ImportPath": "github.com/garyburd/redigo/internal",
"Rev": "535138d7bcd717d6531c701ef5933d98b1866257"
},
{
"ImportPath": "github.com/garyburd/redigo/redis",
"Rev": "535138d7bcd717d6531c701ef5933d98b1866257"
},
{
"ImportPath": "github.com/gorilla/context",
"Rev": "14f550f51af52180c2eefed15e5fd18d63c0a64a"
},
{
"ImportPath": "github.com/gorilla/handlers",
"Rev": "0e84b7d810c16aed432217e330206be156bafae0"
},
{
"ImportPath": "github.com/gorilla/mux",
"Rev": "e444e69cbd2e2e3e0749a2f3c717cec491552bbf"
},
{
"ImportPath": "github.com/noahdesu/go-ceph/rados",
"Comment": "v.0.3.0-29-gb15639c",
"Rev": "b15639c44c05368348355229070361395d9152ee"
},
{
"ImportPath": "github.com/stevvooe/resumable",
"Rev": "51ad44105773cafcbe91927f70ac68e1bf78f8b4"
},
{
"ImportPath": "github.com/mitchellh/mapstructure",
"Rev": "482a9fd5fa83e8c4e7817413b80f3eb8feec03ef"
},
{
"ImportPath": "github.com/ncw/swift",
"Rev": "22c8fa9fb5ba145b4d4e2cebb027e84b1a7b1296"
},
{
"ImportPath": "github.com/yvasiyarov/go-metrics",
"Rev": "57bccd1ccd43f94bb17fdd8bf3007059b802f85e"
},
{
"ImportPath": "github.com/yvasiyarov/gorelic",
"Comment": "v0.0.6-8-ga9bba5b",
"Rev": "a9bba5b9ab508a086f9a12b8c51fab68478e2128"
},
{
"ImportPath": "github.com/yvasiyarov/newrelic_platform_go",
"Rev": "b21fdbd4370f3717f3bbd2bf41c223bc273068e6"
},
{
"ImportPath": "golang.org/x/crypto/bcrypt",
"Rev": "c10c31b5e94b6f7a0283272dc2bb27163dcea24b"
},
{
"ImportPath": "golang.org/x/crypto/blowfish",
"Rev": "c10c31b5e94b6f7a0283272dc2bb27163dcea24b"
},
{
"ImportPath": "golang.org/x/net/context",
"Rev": "1dfe7915deaf3f80b962c163b918868d8a6d8974"
},
{
"ImportPath": "gopkg.in/check.v1",
"Rev": "64131543e7896d5bcc6bd5a76287eb75ea96c673"
},
{
"ImportPath": "gopkg.in/yaml.v2",
"Rev": "bef53efd0c76e49e6de55ead051f886bea7e9420"
}
]
}

5
Godeps/Readme generated
View file

@ -1,5 +0,0 @@
This directory tree is generated automatically by godep.
Please do not edit.
See https://github.com/tools/godep for more information.

2
Godeps/_workspace/.gitignore generated vendored
View file

@ -1,2 +0,0 @@
/pkg
/bin

View file

@ -1,57 +0,0 @@
package aws_test
import (
"github.com/AdRoll/goamz/aws"
"gopkg.in/check.v1"
"time"
)
func (S) TestAttemptTiming(c *check.C) {
testAttempt := aws.AttemptStrategy{
Total: 0.25e9,
Delay: 0.1e9,
}
want := []time.Duration{0, 0.1e9, 0.2e9, 0.2e9}
got := make([]time.Duration, 0, len(want)) // avoid allocation when testing timing
t0 := time.Now()
for a := testAttempt.Start(); a.Next(); {
got = append(got, time.Now().Sub(t0))
}
got = append(got, time.Now().Sub(t0))
c.Assert(got, check.HasLen, len(want))
const margin = 0.01e9
for i, got := range want {
lo := want[i] - margin
hi := want[i] + margin
if got < lo || got > hi {
c.Errorf("attempt %d want %g got %g", i, want[i].Seconds(), got.Seconds())
}
}
}
func (S) TestAttemptNextHasNext(c *check.C) {
a := aws.AttemptStrategy{}.Start()
c.Assert(a.Next(), check.Equals, true)
c.Assert(a.Next(), check.Equals, false)
a = aws.AttemptStrategy{}.Start()
c.Assert(a.Next(), check.Equals, true)
c.Assert(a.HasNext(), check.Equals, false)
c.Assert(a.Next(), check.Equals, false)
a = aws.AttemptStrategy{Total: 2e8}.Start()
c.Assert(a.Next(), check.Equals, true)
c.Assert(a.HasNext(), check.Equals, true)
time.Sleep(2e8)
c.Assert(a.HasNext(), check.Equals, true)
c.Assert(a.Next(), check.Equals, true)
c.Assert(a.Next(), check.Equals, false)
a = aws.AttemptStrategy{Total: 1e8, Min: 2}.Start()
time.Sleep(1e8)
c.Assert(a.Next(), check.Equals, true)
c.Assert(a.HasNext(), check.Equals, true)
c.Assert(a.Next(), check.Equals, true)
c.Assert(a.HasNext(), check.Equals, false)
c.Assert(a.Next(), check.Equals, false)
}

View file

@ -1,629 +0,0 @@
//
// goamz - Go packages to interact with the Amazon Web Services.
//
// https://wiki.ubuntu.com/goamz
//
// Copyright (c) 2011 Canonical Ltd.
//
// Written by Gustavo Niemeyer <gustavo.niemeyer@canonical.com>
//
package aws
import (
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"io/ioutil"
"net"
"net/http"
"net/url"
"os"
"os/user"
"path"
"regexp"
"strings"
"time"
)
// Regular expressions for INI files
var (
iniSectionRegexp = regexp.MustCompile(`^\s*\[([^\[\]]+)\]\s*$`)
iniSettingRegexp = regexp.MustCompile(`^\s*(.+?)\s*=\s*(.*\S)\s*$`)
)
// Defines the valid signers
const (
V2Signature = iota
V4Signature = iota
Route53Signature = iota
)
// Defines the service endpoint and correct Signer implementation to use
// to sign requests for this endpoint
type ServiceInfo struct {
Endpoint string
Signer uint
}
// Region defines the URLs where AWS services may be accessed.
//
// See http://goo.gl/d8BP1 for more details.
type Region struct {
Name string // the canonical name of this region.
EC2Endpoint string
S3Endpoint string
S3BucketEndpoint string // Not needed by AWS S3. Use ${bucket} for bucket name.
S3LocationConstraint bool // true if this region requires a LocationConstraint declaration.
S3LowercaseBucket bool // true if the region requires bucket names to be lower case.
SDBEndpoint string
SNSEndpoint string
SQSEndpoint string
SESEndpoint string
IAMEndpoint string
ELBEndpoint string
KMSEndpoint string
DynamoDBEndpoint string
CloudWatchServicepoint ServiceInfo
AutoScalingEndpoint string
RDSEndpoint ServiceInfo
KinesisEndpoint string
STSEndpoint string
CloudFormationEndpoint string
ElastiCacheEndpoint string
}
var Regions = map[string]Region{
APNortheast.Name: APNortheast,
APSoutheast.Name: APSoutheast,
APSoutheast2.Name: APSoutheast2,
EUCentral.Name: EUCentral,
EUWest.Name: EUWest,
USEast.Name: USEast,
USWest.Name: USWest,
USWest2.Name: USWest2,
USGovWest.Name: USGovWest,
SAEast.Name: SAEast,
CNNorth1.Name: CNNorth1,
}
// Designates a signer interface suitable for signing AWS requests, params
// should be appropriately encoded for the request before signing.
//
// A signer should be initialized with Auth and the appropriate endpoint.
type Signer interface {
Sign(method, path string, params map[string]string)
}
// An AWS Service interface with the API to query the AWS service
//
// Supplied as an easy way to mock out service calls during testing.
type AWSService interface {
// Queries the AWS service at a given method/path with the params and
// returns an http.Response and error
Query(method, path string, params map[string]string) (*http.Response, error)
// Builds an error given an XML payload in the http.Response, can be used
// to process an error if the status code is not 200 for example.
BuildError(r *http.Response) error
}
// Implements a Server Query/Post API to easily query AWS services and build
// errors when desired
type Service struct {
service ServiceInfo
signer Signer
}
// Create a base set of params for an action
func MakeParams(action string) map[string]string {
params := make(map[string]string)
params["Action"] = action
return params
}
// Create a new AWS server to handle making requests
func NewService(auth Auth, service ServiceInfo) (s *Service, err error) {
var signer Signer
switch service.Signer {
case V2Signature:
signer, err = NewV2Signer(auth, service)
// case V4Signature:
// signer, err = NewV4Signer(auth, service, Regions["eu-west-1"])
default:
err = fmt.Errorf("Unsupported signer for service")
}
if err != nil {
return
}
s = &Service{service: service, signer: signer}
return
}
func (s *Service) Query(method, path string, params map[string]string) (resp *http.Response, err error) {
params["Timestamp"] = time.Now().UTC().Format(time.RFC3339)
u, err := url.Parse(s.service.Endpoint)
if err != nil {
return nil, err
}
u.Path = path
s.signer.Sign(method, path, params)
if method == "GET" {
u.RawQuery = multimap(params).Encode()
resp, err = http.Get(u.String())
} else if method == "POST" {
resp, err = http.PostForm(u.String(), multimap(params))
}
return
}
func (s *Service) BuildError(r *http.Response) error {
errors := ErrorResponse{}
xml.NewDecoder(r.Body).Decode(&errors)
var err Error
err = errors.Errors
err.RequestId = errors.RequestId
err.StatusCode = r.StatusCode
if err.Message == "" {
err.Message = r.Status
}
return &err
}
type ServiceError interface {
error
ErrorCode() string
}
type ErrorResponse struct {
Errors Error `xml:"Error"`
RequestId string // A unique ID for tracking the request
}
type Error struct {
StatusCode int
Type string
Code string
Message string
RequestId string
}
func (err *Error) Error() string {
return fmt.Sprintf("Type: %s, Code: %s, Message: %s",
err.Type, err.Code, err.Message,
)
}
func (err *Error) ErrorCode() string {
return err.Code
}
type Auth struct {
AccessKey, SecretKey string
token string
expiration time.Time
}
func (a *Auth) Token() string {
if a.token == "" {
return ""
}
if time.Since(a.expiration) >= -30*time.Second { //in an ideal world this should be zero assuming the instance is synching it's clock
auth, err := GetAuth("", "", "", time.Time{})
if err == nil {
*a = auth
}
}
return a.token
}
func (a *Auth) Expiration() time.Time {
return a.expiration
}
// To be used with other APIs that return auth credentials such as STS
func NewAuth(accessKey, secretKey, token string, expiration time.Time) *Auth {
return &Auth{
AccessKey: accessKey,
SecretKey: secretKey,
token: token,
expiration: expiration,
}
}
// ResponseMetadata
type ResponseMetadata struct {
RequestId string // A unique ID for tracking the request
}
type BaseResponse struct {
ResponseMetadata ResponseMetadata
}
var unreserved = make([]bool, 128)
var hex = "0123456789ABCDEF"
func init() {
// RFC3986
u := "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234567890-_.~"
for _, c := range u {
unreserved[c] = true
}
}
func multimap(p map[string]string) url.Values {
q := make(url.Values, len(p))
for k, v := range p {
q[k] = []string{v}
}
return q
}
type credentials struct {
Code string
LastUpdated string
Type string
AccessKeyId string
SecretAccessKey string
Token string
Expiration string
}
// GetMetaData retrieves instance metadata about the current machine.
//
// See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html for more details.
func GetMetaData(path string) (contents []byte, err error) {
c := http.Client{
Transport: &http.Transport{
Dial: func(netw, addr string) (net.Conn, error) {
deadline := time.Now().Add(5 * time.Second)
c, err := net.DialTimeout(netw, addr, time.Second*2)
if err != nil {
return nil, err
}
c.SetDeadline(deadline)
return c, nil
},
},
}
url := "http://169.254.169.254/latest/meta-data/" + path
resp, err := c.Get(url)
if err != nil {
return
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
err = fmt.Errorf("Code %d returned for url %s", resp.StatusCode, url)
return
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return
}
return []byte(body), err
}
func GetRegion(regionName string) (region Region) {
region = Regions[regionName]
return
}
// GetInstanceCredentials creates an Auth based on the instance's role credentials.
// If the running instance is not in EC2 or does not have a valid IAM role, an error will be returned.
// For more info about setting up IAM roles, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
func GetInstanceCredentials() (cred credentials, err error) {
credentialPath := "iam/security-credentials/"
// Get the instance role
role, err := GetMetaData(credentialPath)
if err != nil {
return
}
// Get the instance role credentials
credentialJSON, err := GetMetaData(credentialPath + string(role))
if err != nil {
return
}
err = json.Unmarshal([]byte(credentialJSON), &cred)
return
}
// GetAuth creates an Auth based on either passed in credentials,
// environment information or instance based role credentials.
func GetAuth(accessKey string, secretKey, token string, expiration time.Time) (auth Auth, err error) {
// First try passed in credentials
if accessKey != "" && secretKey != "" {
return Auth{accessKey, secretKey, token, expiration}, nil
}
// Next try to get auth from the environment
auth, err = EnvAuth()
if err == nil {
// Found auth, return
return
}
// Next try getting auth from the instance role
cred, err := GetInstanceCredentials()
if err == nil {
// Found auth, return
auth.AccessKey = cred.AccessKeyId
auth.SecretKey = cred.SecretAccessKey
auth.token = cred.Token
exptdate, err := time.Parse("2006-01-02T15:04:05Z", cred.Expiration)
if err != nil {
err = fmt.Errorf("Error Parsing expiration date: cred.Expiration :%s , error: %s \n", cred.Expiration, err)
}
auth.expiration = exptdate
return auth, err
}
// Next try getting auth from the credentials file
auth, err = CredentialFileAuth("", "", time.Minute*5)
if err == nil {
return
}
//err = errors.New("No valid AWS authentication found")
err = fmt.Errorf("No valid AWS authentication found: %s", err)
return auth, err
}
// EnvAuth creates an Auth based on environment information.
// The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment
// variables are used.
func EnvAuth() (auth Auth, err error) {
auth.AccessKey = os.Getenv("AWS_ACCESS_KEY_ID")
if auth.AccessKey == "" {
auth.AccessKey = os.Getenv("AWS_ACCESS_KEY")
}
auth.SecretKey = os.Getenv("AWS_SECRET_ACCESS_KEY")
if auth.SecretKey == "" {
auth.SecretKey = os.Getenv("AWS_SECRET_KEY")
}
if auth.AccessKey == "" {
err = errors.New("AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment")
}
if auth.SecretKey == "" {
err = errors.New("AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY not found in environment")
}
return
}
// CredentialFileAuth creates and Auth based on a credentials file. The file
// contains various authentication profiles for use with AWS.
//
// The credentials file, which is used by other AWS SDKs, is documented at
// http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs
func CredentialFileAuth(filePath string, profile string, expiration time.Duration) (auth Auth, err error) {
if profile == "" {
profile = "default"
}
if filePath == "" {
u, err := user.Current()
if err != nil {
return auth, err
}
filePath = path.Join(u.HomeDir, ".aws", "credentials")
}
// read the file, then parse the INI
contents, err := ioutil.ReadFile(filePath)
if err != nil {
return
}
profiles := parseINI(string(contents))
profileData, ok := profiles[profile]
if !ok {
err = errors.New("The credentials file did not contain the profile")
return
}
keyId, ok := profileData["aws_access_key_id"]
if !ok {
err = errors.New("The credentials file did not contain required attribute aws_access_key_id")
return
}
secretKey, ok := profileData["aws_secret_access_key"]
if !ok {
err = errors.New("The credentials file did not contain required attribute aws_secret_access_key")
return
}
auth.AccessKey = keyId
auth.SecretKey = secretKey
if token, ok := profileData["aws_session_token"]; ok {
auth.token = token
}
auth.expiration = time.Now().Add(expiration)
return
}
// parseINI takes the contents of a credentials file and returns a map, whose keys
// are the various profiles, and whose values are maps of the settings for the
// profiles
func parseINI(fileContents string) map[string]map[string]string {
profiles := make(map[string]map[string]string)
lines := strings.Split(fileContents, "\n")
var currentSection map[string]string
for _, line := range lines {
// remove comments, which start with a semi-colon
if split := strings.Split(line, ";"); len(split) > 1 {
line = split[0]
}
// check if the line is the start of a profile.
//
// for example:
// [default]
//
// otherwise, check for the proper setting
// property=value
if sectMatch := iniSectionRegexp.FindStringSubmatch(line); len(sectMatch) == 2 {
currentSection = make(map[string]string)
profiles[sectMatch[1]] = currentSection
} else if setMatch := iniSettingRegexp.FindStringSubmatch(line); len(setMatch) == 3 && currentSection != nil {
currentSection[setMatch[1]] = setMatch[2]
}
}
return profiles
}
// Encode takes a string and URI-encodes it in a way suitable
// to be used in AWS signatures.
func Encode(s string) string {
encode := false
for i := 0; i != len(s); i++ {
c := s[i]
if c > 127 || !unreserved[c] {
encode = true
break
}
}
if !encode {
return s
}
e := make([]byte, len(s)*3)
ei := 0
for i := 0; i != len(s); i++ {
c := s[i]
if c > 127 || !unreserved[c] {
e[ei] = '%'
e[ei+1] = hex[c>>4]
e[ei+2] = hex[c&0xF]
ei += 3
} else {
e[ei] = c
ei += 1
}
}
return string(e[:ei])
}
func dialTimeout(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, time.Duration(2*time.Second))
}
func AvailabilityZone() string {
transport := http.Transport{Dial: dialTimeout}
client := http.Client{
Transport: &transport,
}
resp, err := client.Get("http://169.254.169.254/latest/meta-data/placement/availability-zone")
if err != nil {
return "unknown"
} else {
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "unknown"
} else {
return string(body)
}
}
}
func InstanceRegion() string {
az := AvailabilityZone()
if az == "unknown" {
return az
} else {
region := az[:len(az)-1]
return region
}
}
func InstanceId() string {
transport := http.Transport{Dial: dialTimeout}
client := http.Client{
Transport: &transport,
}
resp, err := client.Get("http://169.254.169.254/latest/meta-data/instance-id")
if err != nil {
return "unknown"
} else {
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "unknown"
} else {
return string(body)
}
}
}
func InstanceType() string {
transport := http.Transport{Dial: dialTimeout}
client := http.Client{
Transport: &transport,
}
resp, err := client.Get("http://169.254.169.254/latest/meta-data/instance-type")
if err != nil {
return "unknown"
} else {
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "unknown"
} else {
return string(body)
}
}
}
func ServerLocalIp() string {
transport := http.Transport{Dial: dialTimeout}
client := http.Client{
Transport: &transport,
}
resp, err := client.Get("http://169.254.169.254/latest/meta-data/local-ipv4")
if err != nil {
return "127.0.0.1"
} else {
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "127.0.0.1"
} else {
return string(body)
}
}
}
func ServerPublicIp() string {
transport := http.Transport{Dial: dialTimeout}
client := http.Client{
Transport: &transport,
}
resp, err := client.Get("http://169.254.169.254/latest/meta-data/public-ipv4")
if err != nil {
return "127.0.0.1"
} else {
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "127.0.0.1"
} else {
return string(body)
}
}
}

View file

@ -1,140 +0,0 @@
package aws_test
import (
"github.com/AdRoll/goamz/aws"
"gopkg.in/check.v1"
"io/ioutil"
"os"
"strings"
"testing"
"time"
)
func Test(t *testing.T) {
check.TestingT(t)
}
var _ = check.Suite(&S{})
type S struct {
environ []string
}
func (s *S) SetUpSuite(c *check.C) {
s.environ = os.Environ()
}
func (s *S) TearDownTest(c *check.C) {
os.Clearenv()
for _, kv := range s.environ {
l := strings.SplitN(kv, "=", 2)
os.Setenv(l[0], l[1])
}
}
func (s *S) TestEnvAuthNoSecret(c *check.C) {
os.Clearenv()
_, err := aws.EnvAuth()
c.Assert(err, check.ErrorMatches, "AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY not found in environment")
}
func (s *S) TestEnvAuthNoAccess(c *check.C) {
os.Clearenv()
os.Setenv("AWS_SECRET_ACCESS_KEY", "foo")
_, err := aws.EnvAuth()
c.Assert(err, check.ErrorMatches, "AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment")
}
func (s *S) TestEnvAuth(c *check.C) {
os.Clearenv()
os.Setenv("AWS_SECRET_ACCESS_KEY", "secret")
os.Setenv("AWS_ACCESS_KEY_ID", "access")
auth, err := aws.EnvAuth()
c.Assert(err, check.IsNil)
c.Assert(auth, check.Equals, aws.Auth{SecretKey: "secret", AccessKey: "access"})
}
func (s *S) TestEnvAuthAlt(c *check.C) {
os.Clearenv()
os.Setenv("AWS_SECRET_KEY", "secret")
os.Setenv("AWS_ACCESS_KEY", "access")
auth, err := aws.EnvAuth()
c.Assert(err, check.IsNil)
c.Assert(auth, check.Equals, aws.Auth{SecretKey: "secret", AccessKey: "access"})
}
func (s *S) TestGetAuthStatic(c *check.C) {
exptdate := time.Now().Add(time.Hour)
auth, err := aws.GetAuth("access", "secret", "token", exptdate)
c.Assert(err, check.IsNil)
c.Assert(auth.AccessKey, check.Equals, "access")
c.Assert(auth.SecretKey, check.Equals, "secret")
c.Assert(auth.Token(), check.Equals, "token")
c.Assert(auth.Expiration(), check.Equals, exptdate)
}
func (s *S) TestGetAuthEnv(c *check.C) {
os.Clearenv()
os.Setenv("AWS_SECRET_ACCESS_KEY", "secret")
os.Setenv("AWS_ACCESS_KEY_ID", "access")
auth, err := aws.GetAuth("", "", "", time.Time{})
c.Assert(err, check.IsNil)
c.Assert(auth, check.Equals, aws.Auth{SecretKey: "secret", AccessKey: "access"})
}
func (s *S) TestEncode(c *check.C) {
c.Assert(aws.Encode("foo"), check.Equals, "foo")
c.Assert(aws.Encode("/"), check.Equals, "%2F")
}
func (s *S) TestRegionsAreNamed(c *check.C) {
for n, r := range aws.Regions {
c.Assert(n, check.Equals, r.Name)
}
}
func (s *S) TestCredentialsFileAuth(c *check.C) {
file, err := ioutil.TempFile("", "creds")
if err != nil {
c.Fatal(err)
}
iniFile := `
[default] ; comment 123
aws_access_key_id = keyid1 ;comment
aws_secret_access_key=key1
[profile2]
aws_access_key_id = keyid2 ;comment
aws_secret_access_key=key2
aws_session_token=token1
`
_, err = file.WriteString(iniFile)
if err != nil {
c.Fatal(err)
}
err = file.Close()
if err != nil {
c.Fatal(err)
}
// check non-existant profile
_, err = aws.CredentialFileAuth(file.Name(), "no profile", 30*time.Minute)
c.Assert(err, check.Not(check.Equals), nil)
defaultProfile, err := aws.CredentialFileAuth(file.Name(), "default", 30*time.Minute)
c.Assert(err, check.Equals, nil)
c.Assert(defaultProfile.AccessKey, check.Equals, "keyid1")
c.Assert(defaultProfile.SecretKey, check.Equals, "key1")
c.Assert(defaultProfile.Token(), check.Equals, "")
profile2, err := aws.CredentialFileAuth(file.Name(), "profile2", 30*time.Minute)
c.Assert(err, check.Equals, nil)
c.Assert(profile2.AccessKey, check.Equals, "keyid2")
c.Assert(profile2.SecretKey, check.Equals, "key2")
c.Assert(profile2.Token(), check.Equals, "token1")
}

View file

@ -1,124 +0,0 @@
package aws
import (
"math"
"net"
"net/http"
"time"
)
type RetryableFunc func(*http.Request, *http.Response, error) bool
type WaitFunc func(try int)
type DeadlineFunc func() time.Time
type ResilientTransport struct {
// Timeout is the maximum amount of time a dial will wait for
// a connect to complete.
//
// The default is no timeout.
//
// With or without a timeout, the operating system may impose
// its own earlier timeout. For instance, TCP timeouts are
// often around 3 minutes.
DialTimeout time.Duration
// MaxTries, if non-zero, specifies the number of times we will retry on
// failure. Retries are only attempted for temporary network errors or known
// safe failures.
MaxTries int
Deadline DeadlineFunc
ShouldRetry RetryableFunc
Wait WaitFunc
transport *http.Transport
}
// Convenience method for creating an http client
func NewClient(rt *ResilientTransport) *http.Client {
rt.transport = &http.Transport{
Dial: func(netw, addr string) (net.Conn, error) {
c, err := net.DialTimeout(netw, addr, rt.DialTimeout)
if err != nil {
return nil, err
}
c.SetDeadline(rt.Deadline())
return c, nil
},
Proxy: http.ProxyFromEnvironment,
}
// TODO: Would be nice is ResilientTransport allowed clients to initialize
// with http.Transport attributes.
return &http.Client{
Transport: rt,
}
}
var retryingTransport = &ResilientTransport{
Deadline: func() time.Time {
return time.Now().Add(5 * time.Second)
},
DialTimeout: 10 * time.Second,
MaxTries: 3,
ShouldRetry: awsRetry,
Wait: ExpBackoff,
}
// Exported default client
var RetryingClient = NewClient(retryingTransport)
func (t *ResilientTransport) RoundTrip(req *http.Request) (*http.Response, error) {
return t.tries(req)
}
// Retry a request a maximum of t.MaxTries times.
// We'll only retry if the proper criteria are met.
// If a wait function is specified, wait that amount of time
// In between requests.
func (t *ResilientTransport) tries(req *http.Request) (res *http.Response, err error) {
for try := 0; try < t.MaxTries; try += 1 {
res, err = t.transport.RoundTrip(req)
if !t.ShouldRetry(req, res, err) {
break
}
if res != nil {
res.Body.Close()
}
if t.Wait != nil {
t.Wait(try)
}
}
return
}
func ExpBackoff(try int) {
time.Sleep(100 * time.Millisecond *
time.Duration(math.Exp2(float64(try))))
}
func LinearBackoff(try int) {
time.Sleep(time.Duration(try*100) * time.Millisecond)
}
// Decide if we should retry a request.
// In general, the criteria for retrying a request is described here
// http://docs.aws.amazon.com/general/latest/gr/api-retries.html
func awsRetry(req *http.Request, res *http.Response, err error) bool {
retry := false
// Retry if there's a temporary network error.
if neterr, ok := err.(net.Error); ok {
if neterr.Temporary() {
retry = true
}
}
// Retry if we get a 5xx series error.
if res != nil {
if res.StatusCode >= 500 && res.StatusCode < 600 {
retry = true
}
}
return retry
}

View file

@ -1,29 +0,0 @@
package aws
import (
"net/http"
"time"
)
// V4Signer:
// Exporting methods for testing
func (s *V4Signer) RequestTime(req *http.Request) time.Time {
return s.requestTime(req)
}
func (s *V4Signer) CanonicalRequest(req *http.Request) string {
return s.canonicalRequest(req, "")
}
func (s *V4Signer) StringToSign(t time.Time, creq string) string {
return s.stringToSign(t, creq)
}
func (s *V4Signer) Signature(t time.Time, sts string) string {
return s.signature(t, sts)
}
func (s *V4Signer) Authorization(header http.Header, t time.Time, signature string) string {
return s.authorization(header, t, signature)
}

View file

@ -1,265 +0,0 @@
package aws
var USGovWest = Region{
"us-gov-west-1",
"https://ec2.us-gov-west-1.amazonaws.com",
"https://s3-fips-us-gov-west-1.amazonaws.com",
"",
true,
true,
"",
"https://sns.us-gov-west-1.amazonaws.com",
"https://sqs.us-gov-west-1.amazonaws.com",
"",
"https://iam.us-gov.amazonaws.com",
"https://elasticloadbalancing.us-gov-west-1.amazonaws.com",
"",
"https://dynamodb.us-gov-west-1.amazonaws.com",
ServiceInfo{"https://monitoring.us-gov-west-1.amazonaws.com", V2Signature},
"https://autoscaling.us-gov-west-1.amazonaws.com",
ServiceInfo{"https://rds.us-gov-west-1.amazonaws.com", V2Signature},
"",
"https://sts.amazonaws.com",
"https://cloudformation.us-gov-west-1.amazonaws.com",
"",
}
var USEast = Region{
"us-east-1",
"https://ec2.us-east-1.amazonaws.com",
"https://s3.amazonaws.com",
"",
false,
false,
"https://sdb.amazonaws.com",
"https://sns.us-east-1.amazonaws.com",
"https://sqs.us-east-1.amazonaws.com",
"https://email.us-east-1.amazonaws.com",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.us-east-1.amazonaws.com",
"https://kms.us-east-1.amazonaws.com",
"https://dynamodb.us-east-1.amazonaws.com",
ServiceInfo{"https://monitoring.us-east-1.amazonaws.com", V2Signature},
"https://autoscaling.us-east-1.amazonaws.com",
ServiceInfo{"https://rds.us-east-1.amazonaws.com", V2Signature},
"https://kinesis.us-east-1.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.us-east-1.amazonaws.com",
"https://elasticache.us-east-1.amazonaws.com",
}
var USWest = Region{
"us-west-1",
"https://ec2.us-west-1.amazonaws.com",
"https://s3-us-west-1.amazonaws.com",
"",
true,
true,
"https://sdb.us-west-1.amazonaws.com",
"https://sns.us-west-1.amazonaws.com",
"https://sqs.us-west-1.amazonaws.com",
"",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.us-west-1.amazonaws.com",
"https://kms.us-west-1.amazonaws.com",
"https://dynamodb.us-west-1.amazonaws.com",
ServiceInfo{"https://monitoring.us-west-1.amazonaws.com", V2Signature},
"https://autoscaling.us-west-1.amazonaws.com",
ServiceInfo{"https://rds.us-west-1.amazonaws.com", V2Signature},
"https://kinesis.us-west-1.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.us-west-1.amazonaws.com",
"https://elasticache.us-west-1.amazonaws.com",
}
var USWest2 = Region{
"us-west-2",
"https://ec2.us-west-2.amazonaws.com",
"https://s3-us-west-2.amazonaws.com",
"",
true,
true,
"https://sdb.us-west-2.amazonaws.com",
"https://sns.us-west-2.amazonaws.com",
"https://sqs.us-west-2.amazonaws.com",
"https://email.us-west-2.amazonaws.com",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.us-west-2.amazonaws.com",
"https://kms.us-west-2.amazonaws.com",
"https://dynamodb.us-west-2.amazonaws.com",
ServiceInfo{"https://monitoring.us-west-2.amazonaws.com", V2Signature},
"https://autoscaling.us-west-2.amazonaws.com",
ServiceInfo{"https://rds.us-west-2.amazonaws.com", V2Signature},
"https://kinesis.us-west-2.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.us-west-2.amazonaws.com",
"https://elasticache.us-west-2.amazonaws.com",
}
var EUWest = Region{
"eu-west-1",
"https://ec2.eu-west-1.amazonaws.com",
"https://s3-eu-west-1.amazonaws.com",
"",
true,
true,
"https://sdb.eu-west-1.amazonaws.com",
"https://sns.eu-west-1.amazonaws.com",
"https://sqs.eu-west-1.amazonaws.com",
"https://email.eu-west-1.amazonaws.com",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.eu-west-1.amazonaws.com",
"https://kms.eu-west-1.amazonaws.com",
"https://dynamodb.eu-west-1.amazonaws.com",
ServiceInfo{"https://monitoring.eu-west-1.amazonaws.com", V2Signature},
"https://autoscaling.eu-west-1.amazonaws.com",
ServiceInfo{"https://rds.eu-west-1.amazonaws.com", V2Signature},
"https://kinesis.eu-west-1.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.eu-west-1.amazonaws.com",
"https://elasticache.eu-west-1.amazonaws.com",
}
var EUCentral = Region{
"eu-central-1",
"https://ec2.eu-central-1.amazonaws.com",
"https://s3-eu-central-1.amazonaws.com",
"",
true,
true,
"https://sdb.eu-central-1.amazonaws.com",
"https://sns.eu-central-1.amazonaws.com",
"https://sqs.eu-central-1.amazonaws.com",
"",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.eu-central-1.amazonaws.com",
"https://kms.eu-central-1.amazonaws.com",
"https://dynamodb.eu-central-1.amazonaws.com",
ServiceInfo{"https://monitoring.eu-central-1.amazonaws.com", V2Signature},
"https://autoscaling.eu-central-1.amazonaws.com",
ServiceInfo{"https://rds.eu-central-1.amazonaws.com", V2Signature},
"https://kinesis.eu-central-1.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.eu-central-1.amazonaws.com",
"",
}
var APSoutheast = Region{
"ap-southeast-1",
"https://ec2.ap-southeast-1.amazonaws.com",
"https://s3-ap-southeast-1.amazonaws.com",
"",
true,
true,
"https://sdb.ap-southeast-1.amazonaws.com",
"https://sns.ap-southeast-1.amazonaws.com",
"https://sqs.ap-southeast-1.amazonaws.com",
"",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.ap-southeast-1.amazonaws.com",
"https://kms.ap-southeast-1.amazonaws.com",
"https://dynamodb.ap-southeast-1.amazonaws.com",
ServiceInfo{"https://monitoring.ap-southeast-1.amazonaws.com", V2Signature},
"https://autoscaling.ap-southeast-1.amazonaws.com",
ServiceInfo{"https://rds.ap-southeast-1.amazonaws.com", V2Signature},
"https://kinesis.ap-southeast-1.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.ap-southeast-1.amazonaws.com",
"https://elasticache.ap-southeast-1.amazonaws.com",
}
var APSoutheast2 = Region{
"ap-southeast-2",
"https://ec2.ap-southeast-2.amazonaws.com",
"https://s3-ap-southeast-2.amazonaws.com",
"",
true,
true,
"https://sdb.ap-southeast-2.amazonaws.com",
"https://sns.ap-southeast-2.amazonaws.com",
"https://sqs.ap-southeast-2.amazonaws.com",
"",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.ap-southeast-2.amazonaws.com",
"https://kms.ap-southeast-2.amazonaws.com",
"https://dynamodb.ap-southeast-2.amazonaws.com",
ServiceInfo{"https://monitoring.ap-southeast-2.amazonaws.com", V2Signature},
"https://autoscaling.ap-southeast-2.amazonaws.com",
ServiceInfo{"https://rds.ap-southeast-2.amazonaws.com", V2Signature},
"https://kinesis.ap-southeast-2.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.ap-southeast-2.amazonaws.com",
"https://elasticache.ap-southeast-2.amazonaws.com",
}
var APNortheast = Region{
"ap-northeast-1",
"https://ec2.ap-northeast-1.amazonaws.com",
"https://s3-ap-northeast-1.amazonaws.com",
"",
true,
true,
"https://sdb.ap-northeast-1.amazonaws.com",
"https://sns.ap-northeast-1.amazonaws.com",
"https://sqs.ap-northeast-1.amazonaws.com",
"",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.ap-northeast-1.amazonaws.com",
"https://kms.ap-northeast-1.amazonaws.com",
"https://dynamodb.ap-northeast-1.amazonaws.com",
ServiceInfo{"https://monitoring.ap-northeast-1.amazonaws.com", V2Signature},
"https://autoscaling.ap-northeast-1.amazonaws.com",
ServiceInfo{"https://rds.ap-northeast-1.amazonaws.com", V2Signature},
"https://kinesis.ap-northeast-1.amazonaws.com",
"https://sts.amazonaws.com",
"https://cloudformation.ap-northeast-1.amazonaws.com",
"https://elasticache.ap-northeast-1.amazonaws.com",
}
var SAEast = Region{
"sa-east-1",
"https://ec2.sa-east-1.amazonaws.com",
"https://s3-sa-east-1.amazonaws.com",
"",
true,
true,
"https://sdb.sa-east-1.amazonaws.com",
"https://sns.sa-east-1.amazonaws.com",
"https://sqs.sa-east-1.amazonaws.com",
"",
"https://iam.amazonaws.com",
"https://elasticloadbalancing.sa-east-1.amazonaws.com",
"https://kms.sa-east-1.amazonaws.com",
"https://dynamodb.sa-east-1.amazonaws.com",
ServiceInfo{"https://monitoring.sa-east-1.amazonaws.com", V2Signature},
"https://autoscaling.sa-east-1.amazonaws.com",
ServiceInfo{"https://rds.sa-east-1.amazonaws.com", V2Signature},
"",
"https://sts.amazonaws.com",
"https://cloudformation.sa-east-1.amazonaws.com",
"https://elasticache.sa-east-1.amazonaws.com",
}
var CNNorth1 = Region{
"cn-north-1",
"https://ec2.cn-north-1.amazonaws.com.cn",
"https://s3.cn-north-1.amazonaws.com.cn",
"",
true,
true,
"",
"https://sns.cn-north-1.amazonaws.com.cn",
"https://sqs.cn-north-1.amazonaws.com.cn",
"",
"https://iam.cn-north-1.amazonaws.com.cn",
"https://elasticloadbalancing.cn-north-1.amazonaws.com.cn",
"",
"https://dynamodb.cn-north-1.amazonaws.com.cn",
ServiceInfo{"https://monitoring.cn-north-1.amazonaws.com.cn", V4Signature},
"https://autoscaling.cn-north-1.amazonaws.com.cn",
ServiceInfo{"https://rds.cn-north-1.amazonaws.com.cn", V4Signature},
"",
"https://sts.cn-north-1.amazonaws.com.cn",
"",
"",
}

View file

@ -1,136 +0,0 @@
package aws
import (
"math/rand"
"net"
"net/http"
"time"
)
const (
maxDelay = 20 * time.Second
defaultScale = 300 * time.Millisecond
throttlingScale = 500 * time.Millisecond
throttlingScaleRange = throttlingScale / 4
defaultMaxRetries = 3
dynamoDBScale = 25 * time.Millisecond
dynamoDBMaxRetries = 10
)
// A RetryPolicy encapsulates a strategy for implementing client retries.
//
// Default implementations are provided which match the AWS SDKs.
type RetryPolicy interface {
// ShouldRetry returns whether a client should retry a failed request.
ShouldRetry(target string, r *http.Response, err error, numRetries int) bool
// Delay returns the time a client should wait before issuing a retry.
Delay(target string, r *http.Response, err error, numRetries int) time.Duration
}
// DefaultRetryPolicy implements the AWS SDK default retry policy.
//
// It will retry up to 3 times, and uses an exponential backoff with a scale
// factor of 300ms (300ms, 600ms, 1200ms). If the retry is because of
// throttling, the delay will also include some randomness.
//
// See https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/retry/PredefinedRetryPolicies.java#L90.
type DefaultRetryPolicy struct {
}
// ShouldRetry implements the RetryPolicy ShouldRetry method.
func (policy DefaultRetryPolicy) ShouldRetry(target string, r *http.Response, err error, numRetries int) bool {
return shouldRetry(r, err, numRetries, defaultMaxRetries)
}
// Delay implements the RetryPolicy Delay method.
func (policy DefaultRetryPolicy) Delay(target string, r *http.Response, err error, numRetries int) time.Duration {
scale := defaultScale
if err, ok := err.(*Error); ok && isThrottlingException(err) {
scale = throttlingScale + time.Duration(rand.Int63n(int64(throttlingScaleRange)))
}
return exponentialBackoff(numRetries, scale)
}
// DynamoDBRetryPolicy implements the AWS SDK DynamoDB retry policy.
//
// It will retry up to 10 times, and uses an exponential backoff with a scale
// factor of 25ms (25ms, 50ms, 100ms, ...).
//
// See https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/retry/PredefinedRetryPolicies.java#L103.
type DynamoDBRetryPolicy struct {
}
// ShouldRetry implements the RetryPolicy ShouldRetry method.
func (policy DynamoDBRetryPolicy) ShouldRetry(target string, r *http.Response, err error, numRetries int) bool {
return shouldRetry(r, err, numRetries, dynamoDBMaxRetries)
}
// Delay implements the RetryPolicy Delay method.
func (policy DynamoDBRetryPolicy) Delay(target string, r *http.Response, err error, numRetries int) time.Duration {
return exponentialBackoff(numRetries, dynamoDBScale)
}
// NeverRetryPolicy never retries requests and returns immediately on failure.
type NeverRetryPolicy struct {
}
// ShouldRetry implements the RetryPolicy ShouldRetry method.
func (policy NeverRetryPolicy) ShouldRetry(target string, r *http.Response, err error, numRetries int) bool {
return false
}
// Delay implements the RetryPolicy Delay method.
func (policy NeverRetryPolicy) Delay(target string, r *http.Response, err error, numRetries int) time.Duration {
return time.Duration(0)
}
// shouldRetry determines if we should retry the request.
//
// See http://docs.aws.amazon.com/general/latest/gr/api-retries.html.
func shouldRetry(r *http.Response, err error, numRetries int, maxRetries int) bool {
// Once we've exceeded the max retry attempts, game over.
if numRetries >= maxRetries {
return false
}
// Always retry temporary network errors.
if err, ok := err.(net.Error); ok && err.Temporary() {
return true
}
// Always retry 5xx responses.
if r != nil && r.StatusCode >= 500 {
return true
}
// Always retry throttling exceptions.
if err, ok := err.(ServiceError); ok && isThrottlingException(err) {
return true
}
// Other classes of failures indicate a problem with the request. Retrying
// won't help.
return false
}
func exponentialBackoff(numRetries int, scale time.Duration) time.Duration {
if numRetries < 0 {
return time.Duration(0)
}
delay := (1 << uint(numRetries)) * scale
if delay > maxDelay {
return maxDelay
}
return delay
}
func isThrottlingException(err ServiceError) bool {
switch err.ErrorCode() {
case "Throttling", "ThrottlingException", "ProvisionedThroughputExceededException":
return true
default:
return false
}
}

View file

@ -1,303 +0,0 @@
package aws
import (
"math/rand"
"net"
"net/http"
"testing"
"time"
)
type testInput struct {
res *http.Response
err error
numRetries int
}
type testResult struct {
shouldRetry bool
delay time.Duration
}
type testCase struct {
input testInput
defaultResult testResult
dynamoDBResult testResult
}
var testCases = []testCase{
// Test nil fields
testCase{
input: testInput{
err: nil,
res: nil,
numRetries: 0,
},
defaultResult: testResult{
shouldRetry: false,
delay: 300 * time.Millisecond,
},
dynamoDBResult: testResult{
shouldRetry: false,
delay: 25 * time.Millisecond,
},
},
// Test 3 different throttling exceptions
testCase{
input: testInput{
err: &Error{
Code: "Throttling",
},
numRetries: 0,
},
defaultResult: testResult{
shouldRetry: true,
delay: 617165505 * time.Nanosecond, // account for randomness with known seed
},
dynamoDBResult: testResult{
shouldRetry: true,
delay: 25 * time.Millisecond,
},
},
testCase{
input: testInput{
err: &Error{
Code: "ThrottlingException",
},
numRetries: 0,
},
defaultResult: testResult{
shouldRetry: true,
delay: 579393152 * time.Nanosecond, // account for randomness with known seed
},
dynamoDBResult: testResult{
shouldRetry: true,
delay: 25 * time.Millisecond,
},
},
testCase{
input: testInput{
err: &Error{
Code: "ProvisionedThroughputExceededException",
},
numRetries: 1,
},
defaultResult: testResult{
shouldRetry: true,
delay: 1105991654 * time.Nanosecond, // account for randomness with known seed
},
dynamoDBResult: testResult{
shouldRetry: true,
delay: 50 * time.Millisecond,
},
},
// Test a fake throttling exception
testCase{
input: testInput{
err: &Error{
Code: "MyMadeUpThrottlingCode",
},
numRetries: 0,
},
defaultResult: testResult{
shouldRetry: false,
delay: 300 * time.Millisecond,
},
dynamoDBResult: testResult{
shouldRetry: false,
delay: 25 * time.Millisecond,
},
},
// Test 5xx errors
testCase{
input: testInput{
res: &http.Response{
StatusCode: http.StatusInternalServerError,
},
numRetries: 1,
},
defaultResult: testResult{
shouldRetry: true,
delay: 600 * time.Millisecond,
},
dynamoDBResult: testResult{
shouldRetry: true,
delay: 50 * time.Millisecond,
},
},
testCase{
input: testInput{
res: &http.Response{
StatusCode: http.StatusServiceUnavailable,
},
numRetries: 1,
},
defaultResult: testResult{
shouldRetry: true,
delay: 600 * time.Millisecond,
},
dynamoDBResult: testResult{
shouldRetry: true,
delay: 50 * time.Millisecond,
},
},
// Test a random 400 error
testCase{
input: testInput{
res: &http.Response{
StatusCode: http.StatusNotFound,
},
numRetries: 1,
},
defaultResult: testResult{
shouldRetry: false,
delay: 600 * time.Millisecond,
},
dynamoDBResult: testResult{
shouldRetry: false,
delay: 50 * time.Millisecond,
},
},
// Test a temporary net.Error
testCase{
input: testInput{
res: &http.Response{},
err: &net.DNSError{
IsTimeout: true,
},
numRetries: 2,
},
defaultResult: testResult{
shouldRetry: true,
delay: 1200 * time.Millisecond,
},
dynamoDBResult: testResult{
shouldRetry: true,
delay: 100 * time.Millisecond,
},
},
// Test a non-temporary net.Error
testCase{
input: testInput{
res: &http.Response{},
err: &net.DNSError{
IsTimeout: false,
},
numRetries: 3,
},
defaultResult: testResult{
shouldRetry: false,
delay: 2400 * time.Millisecond,
},
dynamoDBResult: testResult{
shouldRetry: false,
delay: 200 * time.Millisecond,
},
},
// Assert failure after hitting max default retries
testCase{
input: testInput{
err: &Error{
Code: "ProvisionedThroughputExceededException",
},
numRetries: defaultMaxRetries,
},
defaultResult: testResult{
shouldRetry: false,
delay: 4313582352 * time.Nanosecond, // account for randomness with known seed
},
dynamoDBResult: testResult{
shouldRetry: true,
delay: 200 * time.Millisecond,
},
},
// Assert failure after hitting max DynamoDB retries
testCase{
input: testInput{
err: &Error{
Code: "ProvisionedThroughputExceededException",
},
numRetries: dynamoDBMaxRetries,
},
defaultResult: testResult{
shouldRetry: false,
delay: maxDelay,
},
dynamoDBResult: testResult{
shouldRetry: false,
delay: maxDelay,
},
},
// Assert we never go over the maxDelay value
testCase{
input: testInput{
numRetries: 25,
},
defaultResult: testResult{
shouldRetry: false,
delay: maxDelay,
},
dynamoDBResult: testResult{
shouldRetry: false,
delay: maxDelay,
},
},
}
func TestDefaultRetryPolicy(t *testing.T) {
rand.Seed(0)
var policy RetryPolicy
policy = &DefaultRetryPolicy{}
for _, test := range testCases {
res := test.input.res
err := test.input.err
numRetries := test.input.numRetries
shouldRetry := policy.ShouldRetry("", res, err, numRetries)
if shouldRetry != test.defaultResult.shouldRetry {
t.Errorf("ShouldRetry returned %v, expected %v res=%#v err=%#v numRetries=%d", shouldRetry, test.defaultResult.shouldRetry, res, err, numRetries)
}
delay := policy.Delay("", res, err, numRetries)
if delay != test.defaultResult.delay {
t.Errorf("Delay returned %v, expected %v res=%#v err=%#v numRetries=%d", delay, test.defaultResult.delay, res, err, numRetries)
}
}
}
func TestDynamoDBRetryPolicy(t *testing.T) {
var policy RetryPolicy
policy = &DynamoDBRetryPolicy{}
for _, test := range testCases {
res := test.input.res
err := test.input.err
numRetries := test.input.numRetries
shouldRetry := policy.ShouldRetry("", res, err, numRetries)
if shouldRetry != test.dynamoDBResult.shouldRetry {
t.Errorf("ShouldRetry returned %v, expected %v res=%#v err=%#v numRetries=%d", shouldRetry, test.dynamoDBResult.shouldRetry, res, err, numRetries)
}
delay := policy.Delay("", res, err, numRetries)
if delay != test.dynamoDBResult.delay {
t.Errorf("Delay returned %v, expected %v res=%#v err=%#v numRetries=%d", delay, test.dynamoDBResult.delay, res, err, numRetries)
}
}
}
func TestNeverRetryPolicy(t *testing.T) {
var policy RetryPolicy
policy = &NeverRetryPolicy{}
for _, test := range testCases {
res := test.input.res
err := test.input.err
numRetries := test.input.numRetries
shouldRetry := policy.ShouldRetry("", res, err, numRetries)
if shouldRetry {
t.Errorf("ShouldRetry returned %v, expected %v res=%#v err=%#v numRetries=%d", shouldRetry, false, res, err, numRetries)
}
delay := policy.Delay("", res, err, numRetries)
if delay != time.Duration(0) {
t.Errorf("Delay returned %v, expected %v res=%#v err=%#v numRetries=%d", delay, time.Duration(0), res, err, numRetries)
}
}
}

View file

@ -1,413 +0,0 @@
package aws
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"path"
"sort"
"strings"
"time"
)
type V2Signer struct {
auth Auth
service ServiceInfo
host string
}
var b64 = base64.StdEncoding
func NewV2Signer(auth Auth, service ServiceInfo) (*V2Signer, error) {
u, err := url.Parse(service.Endpoint)
if err != nil {
return nil, err
}
return &V2Signer{auth: auth, service: service, host: u.Host}, nil
}
func (s *V2Signer) Sign(method, path string, params map[string]string) {
params["AWSAccessKeyId"] = s.auth.AccessKey
params["SignatureVersion"] = "2"
params["SignatureMethod"] = "HmacSHA256"
if s.auth.Token() != "" {
params["SecurityToken"] = s.auth.Token()
}
// AWS specifies that the parameters in a signed request must
// be provided in the natural order of the keys. This is distinct
// from the natural order of the encoded value of key=value.
// Percent and gocheck.Equals affect the sorting order.
var keys, sarray []string
for k, _ := range params {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
sarray = append(sarray, Encode(k)+"="+Encode(params[k]))
}
joined := strings.Join(sarray, "&")
payload := method + "\n" + s.host + "\n" + path + "\n" + joined
hash := hmac.New(sha256.New, []byte(s.auth.SecretKey))
hash.Write([]byte(payload))
signature := make([]byte, b64.EncodedLen(hash.Size()))
b64.Encode(signature, hash.Sum(nil))
params["Signature"] = string(signature)
}
// Common date formats for signing requests
const (
ISO8601BasicFormat = "20060102T150405Z"
ISO8601BasicFormatShort = "20060102"
)
type Route53Signer struct {
auth Auth
}
func NewRoute53Signer(auth Auth) *Route53Signer {
return &Route53Signer{auth: auth}
}
// getCurrentDate fetches the date stamp from the aws servers to
// ensure the auth headers are within 5 minutes of the server time
func (s *Route53Signer) getCurrentDate() string {
response, err := http.Get("https://route53.amazonaws.com/date")
if err != nil {
fmt.Print("Unable to get date from amazon: ", err)
return ""
}
response.Body.Close()
return response.Header.Get("Date")
}
// Creates the authorize signature based on the date stamp and secret key
func (s *Route53Signer) getHeaderAuthorize(message string) string {
hmacSha256 := hmac.New(sha256.New, []byte(s.auth.SecretKey))
hmacSha256.Write([]byte(message))
cryptedString := hmacSha256.Sum(nil)
return base64.StdEncoding.EncodeToString(cryptedString)
}
// Adds all the required headers for AWS Route53 API to the request
// including the authorization
func (s *Route53Signer) Sign(req *http.Request) {
date := s.getCurrentDate()
authHeader := fmt.Sprintf("AWS3-HTTPS AWSAccessKeyId=%s,Algorithm=%s,Signature=%s",
s.auth.AccessKey, "HmacSHA256", s.getHeaderAuthorize(date))
req.Header.Set("Host", req.Host)
req.Header.Set("X-Amzn-Authorization", authHeader)
req.Header.Set("X-Amz-Date", date)
req.Header.Set("Content-Type", "application/xml")
if s.auth.Token() != "" {
req.Header.Set("X-Amzn-Security-Token", s.auth.Token())
}
}
/*
The V4Signer encapsulates all of the functionality to sign a request with the AWS
Signature Version 4 Signing Process. (http://goo.gl/u1OWZz)
*/
type V4Signer struct {
auth Auth
serviceName string
region Region
// Add the x-amz-content-sha256 header
IncludeXAmzContentSha256 bool
}
/*
Return a new instance of a V4Signer capable of signing AWS requests.
*/
func NewV4Signer(auth Auth, serviceName string, region Region) *V4Signer {
return &V4Signer{
auth: auth,
serviceName: serviceName,
region: region,
IncludeXAmzContentSha256: false,
}
}
/*
Sign a request according to the AWS Signature Version 4 Signing Process. (http://goo.gl/u1OWZz)
The signed request will include an "x-amz-date" header with a current timestamp if a valid "x-amz-date"
or "date" header was not available in the original request. In addition, AWS Signature Version 4 requires
the "host" header to be a signed header, therefor the Sign method will manually set a "host" header from
the request.Host.
The signed request will include a new "Authorization" header indicating that the request has been signed.
Any changes to the request after signing the request will invalidate the signature.
*/
func (s *V4Signer) Sign(req *http.Request) {
req.Header.Set("host", req.Host) // host header must be included as a signed header
t := s.requestTime(req) // Get request time
payloadHash := ""
if _, ok := req.Form["X-Amz-Expires"]; ok {
// We are authenticating the the request by using query params
// (also known as pre-signing a url, http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html)
payloadHash = "UNSIGNED-PAYLOAD"
req.Header.Del("x-amz-date")
req.Form["X-Amz-SignedHeaders"] = []string{s.signedHeaders(req.Header)}
req.Form["X-Amz-Algorithm"] = []string{"AWS4-HMAC-SHA256"}
req.Form["X-Amz-Credential"] = []string{s.auth.AccessKey + "/" + s.credentialScope(t)}
req.Form["X-Amz-Date"] = []string{t.Format(ISO8601BasicFormat)}
req.URL.RawQuery = req.Form.Encode()
} else {
payloadHash = s.payloadHash(req)
if s.IncludeXAmzContentSha256 {
req.Header.Set("x-amz-content-sha256", payloadHash) // x-amz-content-sha256 contains the payload hash
}
}
creq := s.canonicalRequest(req, payloadHash) // Build canonical request
sts := s.stringToSign(t, creq) // Build string to sign
signature := s.signature(t, sts) // Calculate the AWS Signature Version 4
auth := s.authorization(req.Header, t, signature) // Create Authorization header value
if _, ok := req.Form["X-Amz-Expires"]; ok {
req.Form["X-Amz-Signature"] = []string{signature}
} else {
req.Header.Set("Authorization", auth) // Add Authorization header to request
}
return
}
/*
requestTime method will parse the time from the request "x-amz-date" or "date" headers.
If the "x-amz-date" header is present, that will take priority over the "date" header.
If neither header is defined or we are unable to parse either header as a valid date
then we will create a new "x-amz-date" header with the current time.
*/
func (s *V4Signer) requestTime(req *http.Request) time.Time {
// Get "x-amz-date" header
date := req.Header.Get("x-amz-date")
// Attempt to parse as ISO8601BasicFormat
t, err := time.Parse(ISO8601BasicFormat, date)
if err == nil {
return t
}
// Attempt to parse as http.TimeFormat
t, err = time.Parse(http.TimeFormat, date)
if err == nil {
req.Header.Set("x-amz-date", t.Format(ISO8601BasicFormat))
return t
}
// Get "date" header
date = req.Header.Get("date")
// Attempt to parse as http.TimeFormat
t, err = time.Parse(http.TimeFormat, date)
if err == nil {
return t
}
// Create a current time header to be used
t = time.Now().UTC()
req.Header.Set("x-amz-date", t.Format(ISO8601BasicFormat))
return t
}
/*
canonicalRequest method creates the canonical request according to Task 1 of the AWS Signature Version 4 Signing Process. (http://goo.gl/eUUZ3S)
CanonicalRequest =
HTTPRequestMethod + '\n' +
CanonicalURI + '\n' +
CanonicalQueryString + '\n' +
CanonicalHeaders + '\n' +
SignedHeaders + '\n' +
HexEncode(Hash(Payload))
payloadHash is optional; use the empty string and it will be calculated from the request
*/
func (s *V4Signer) canonicalRequest(req *http.Request, payloadHash string) string {
if payloadHash == "" {
payloadHash = s.payloadHash(req)
}
c := new(bytes.Buffer)
fmt.Fprintf(c, "%s\n", req.Method)
fmt.Fprintf(c, "%s\n", s.canonicalURI(req.URL))
fmt.Fprintf(c, "%s\n", s.canonicalQueryString(req.URL))
fmt.Fprintf(c, "%s\n\n", s.canonicalHeaders(req.Header))
fmt.Fprintf(c, "%s\n", s.signedHeaders(req.Header))
fmt.Fprintf(c, "%s", payloadHash)
return c.String()
}
func (s *V4Signer) canonicalURI(u *url.URL) string {
u = &url.URL{Path: u.Path}
canonicalPath := u.String()
slash := strings.HasSuffix(canonicalPath, "/")
canonicalPath = path.Clean(canonicalPath)
if canonicalPath == "" || canonicalPath == "." {
canonicalPath = "/"
}
if canonicalPath != "/" && slash {
canonicalPath += "/"
}
return canonicalPath
}
func (s *V4Signer) canonicalQueryString(u *url.URL) string {
var a []string
for k, vs := range u.Query() {
k = url.QueryEscape(k)
for _, v := range vs {
if v == "" {
a = append(a, k+"=")
} else {
v = url.QueryEscape(v)
a = append(a, k+"="+v)
}
}
}
sort.Strings(a)
return strings.Join(a, "&")
}
func (s *V4Signer) canonicalHeaders(h http.Header) string {
i, a, lowerCase := 0, make([]string, len(h)), make(map[string][]string)
for k, v := range h {
lowerCase[strings.ToLower(k)] = v
}
var keys []string
for k := range lowerCase {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
v := lowerCase[k]
for j, w := range v {
v[j] = strings.Trim(w, " ")
}
sort.Strings(v)
a[i] = strings.ToLower(k) + ":" + strings.Join(v, ",")
i++
}
return strings.Join(a, "\n")
}
func (s *V4Signer) signedHeaders(h http.Header) string {
i, a := 0, make([]string, len(h))
for k, _ := range h {
a[i] = strings.ToLower(k)
i++
}
sort.Strings(a)
return strings.Join(a, ";")
}
func (s *V4Signer) payloadHash(req *http.Request) string {
var b []byte
if req.Body == nil {
b = []byte("")
} else {
var err error
b, err = ioutil.ReadAll(req.Body)
if err != nil {
// TODO: I REALLY DON'T LIKE THIS PANIC!!!!
panic(err)
}
}
req.Body = ioutil.NopCloser(bytes.NewBuffer(b))
return s.hash(string(b))
}
/*
stringToSign method creates the string to sign accorting to Task 2 of the AWS Signature Version 4 Signing Process. (http://goo.gl/es1PAu)
StringToSign =
Algorithm + '\n' +
RequestDate + '\n' +
CredentialScope + '\n' +
HexEncode(Hash(CanonicalRequest))
*/
func (s *V4Signer) stringToSign(t time.Time, creq string) string {
w := new(bytes.Buffer)
fmt.Fprint(w, "AWS4-HMAC-SHA256\n")
fmt.Fprintf(w, "%s\n", t.Format(ISO8601BasicFormat))
fmt.Fprintf(w, "%s\n", s.credentialScope(t))
fmt.Fprintf(w, "%s", s.hash(creq))
return w.String()
}
func (s *V4Signer) credentialScope(t time.Time) string {
return fmt.Sprintf("%s/%s/%s/aws4_request", t.Format(ISO8601BasicFormatShort), s.region.Name, s.serviceName)
}
/*
signature method calculates the AWS Signature Version 4 according to Task 3 of the AWS Signature Version 4 Signing Process. (http://goo.gl/j0Yqe1)
signature = HexEncode(HMAC(derived-signing-key, string-to-sign))
*/
func (s *V4Signer) signature(t time.Time, sts string) string {
h := s.hmac(s.derivedKey(t), []byte(sts))
return fmt.Sprintf("%x", h)
}
/*
derivedKey method derives a signing key to be used for signing a request.
kSecret = Your AWS Secret Access Key
kDate = HMAC("AWS4" + kSecret, Date)
kRegion = HMAC(kDate, Region)
kService = HMAC(kRegion, Service)
kSigning = HMAC(kService, "aws4_request")
*/
func (s *V4Signer) derivedKey(t time.Time) []byte {
h := s.hmac([]byte("AWS4"+s.auth.SecretKey), []byte(t.Format(ISO8601BasicFormatShort)))
h = s.hmac(h, []byte(s.region.Name))
h = s.hmac(h, []byte(s.serviceName))
h = s.hmac(h, []byte("aws4_request"))
return h
}
/*
authorization method generates the authorization header value.
*/
func (s *V4Signer) authorization(header http.Header, t time.Time, signature string) string {
w := new(bytes.Buffer)
fmt.Fprint(w, "AWS4-HMAC-SHA256 ")
fmt.Fprintf(w, "Credential=%s/%s, ", s.auth.AccessKey, s.credentialScope(t))
fmt.Fprintf(w, "SignedHeaders=%s, ", s.signedHeaders(header))
fmt.Fprintf(w, "Signature=%s", signature)
return w.String()
}
// hash method calculates the sha256 hash for a given string
func (s *V4Signer) hash(in string) string {
h := sha256.New()
fmt.Fprintf(h, "%s", in)
return fmt.Sprintf("%x", h.Sum(nil))
}
// hmac method calculates the sha256 hmac for a given slice of bytes
func (s *V4Signer) hmac(key, data []byte) []byte {
h := hmac.New(sha256.New, key)
h.Write(data)
return h.Sum(nil)
}

View file

@ -1,569 +0,0 @@
package aws_test
import (
"fmt"
"github.com/AdRoll/goamz/aws"
"gopkg.in/check.v1"
"net/http"
"strings"
"time"
)
var _ = check.Suite(&V4SignerSuite{})
type V4SignerSuite struct {
auth aws.Auth
region aws.Region
cases []V4SignerSuiteCase
}
type V4SignerSuiteCase struct {
label string
request V4SignerSuiteCaseRequest
canonicalRequest string
stringToSign string
signature string
authorization string
}
type V4SignerSuiteCaseRequest struct {
method string
host string
url string
headers []string
body string
}
func (s *V4SignerSuite) SetUpSuite(c *check.C) {
s.auth = aws.Auth{AccessKey: "AKIDEXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"}
s.region = aws.USEast
// Test cases from the Signature Version 4 Test Suite (http://goo.gl/nguvs0)
s.cases = append(s.cases,
// get-header-key-duplicate
V4SignerSuiteCase{
label: "get-header-key-duplicate",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"DATE:Mon, 09 Sep 2011 23:36:00 GMT", "ZOO:zoobar", "zoo:foobar", "zoo:zoobar"},
},
canonicalRequest: "POST\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\nzoo:foobar,zoobar,zoobar\n\ndate;host;zoo\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n3c52f0eaae2b61329c0a332e3fa15842a37bc5812cf4d80eb64784308850e313",
signature: "54afcaaf45b331f81cd2edb974f7b824ff4dd594cbbaa945ed636b48477368ed",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host;zoo, Signature=54afcaaf45b331f81cd2edb974f7b824ff4dd594cbbaa945ed636b48477368ed",
},
// get-header-value-order
V4SignerSuiteCase{
label: "get-header-value-order",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"DATE:Mon, 09 Sep 2011 23:36:00 GMT", "p:z", "p:a", "p:p", "p:a"},
},
canonicalRequest: "POST\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\np:a,a,p,z\n\ndate;host;p\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n94c0389fefe0988cbbedc8606f0ca0b485b48da010d09fc844b45b697c8924fe",
signature: "d2973954263943b11624a11d1c963ca81fb274169c7868b2858c04f083199e3d",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host;p, Signature=d2973954263943b11624a11d1c963ca81fb274169c7868b2858c04f083199e3d",
},
// get-header-value-trim
V4SignerSuiteCase{
label: "get-header-value-trim",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"DATE:Mon, 09 Sep 2011 23:36:00 GMT", "p: phfft "},
},
canonicalRequest: "POST\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\np:phfft\n\ndate;host;p\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\ndddd1902add08da1ac94782b05f9278c08dc7468db178a84f8950d93b30b1f35",
signature: "debf546796015d6f6ded8626f5ce98597c33b47b9164cf6b17b4642036fcb592",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host;p, Signature=debf546796015d6f6ded8626f5ce98597c33b47b9164cf6b17b4642036fcb592",
},
// get-empty
V4SignerSuiteCase{
label: "get-relative-relative",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-single-relative
V4SignerSuiteCase{
label: "get-relative-relative",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/.",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-multiple-relative
V4SignerSuiteCase{
label: "get-relative-relative",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/./././",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-relative-relative
V4SignerSuiteCase{
label: "get-relative-relative",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/foo/bar/../..",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-relative
V4SignerSuiteCase{
label: "get-relative",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/foo/..",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-slash-dot-slash
V4SignerSuiteCase{
label: "get-slash-dot-slash",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/./",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-slash-pointless-dot
V4SignerSuiteCase{
label: "get-slash-pointless-dot",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/./foo",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/foo\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n8021a97572ee460f87ca67f4e8c0db763216d84715f5424a843a5312a3321e2d",
signature: "910e4d6c9abafaf87898e1eb4c929135782ea25bb0279703146455745391e63a",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=910e4d6c9abafaf87898e1eb4c929135782ea25bb0279703146455745391e63a",
},
// get-slash
V4SignerSuiteCase{
label: "get-slash",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "//",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-slashes
V4SignerSuiteCase{
label: "get-slashes",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "//foo//",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/foo/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n6bb4476ee8745730c9cb79f33a0c70baa6d8af29c0077fa12e4e8f1dd17e7098",
signature: "b00392262853cfe3201e47ccf945601079e9b8a7f51ee4c3d9ee4f187aa9bf19",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b00392262853cfe3201e47ccf945601079e9b8a7f51ee4c3d9ee4f187aa9bf19",
},
// get-space
V4SignerSuiteCase{
label: "get-space",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/%20/foo",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/%20/foo\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n69c45fb9fe3fd76442b5086e50b2e9fec8298358da957b293ef26e506fdfb54b",
signature: "f309cfbd10197a230c42dd17dbf5cca8a0722564cb40a872d25623cfa758e374",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=f309cfbd10197a230c42dd17dbf5cca8a0722564cb40a872d25623cfa758e374",
},
// get-unreserved
V4SignerSuiteCase{
label: "get-unreserved",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/-._~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/-._~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\ndf63ee3247c0356c696a3b21f8d8490b01fa9cd5bc6550ef5ef5f4636b7b8901",
signature: "830cc36d03f0f84e6ee4953fbe701c1c8b71a0372c63af9255aa364dd183281e",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=830cc36d03f0f84e6ee4953fbe701c1c8b71a0372c63af9255aa364dd183281e",
},
// get-utf8
V4SignerSuiteCase{
label: "get-utf8",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/%E1%88%B4",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/%E1%88%B4\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n27ba31df5dbc6e063d8f87d62eb07143f7f271c5330a917840586ac1c85b6f6b",
signature: "8d6634c189aa8c75c2e51e106b6b5121bed103fdb351f7d7d4381c738823af74",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=8d6634c189aa8c75c2e51e106b6b5121bed103fdb351f7d7d4381c738823af74",
},
// get-vanilla-empty-query-key
V4SignerSuiteCase{
label: "get-vanilla-empty-query-key",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/?foo=bar",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\nfoo=bar\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n0846c2945b0832deb7a463c66af5c4f8bd54ec28c438e67a214445b157c9ddf8",
signature: "56c054473fd260c13e4e7393eb203662195f5d4a1fada5314b8b52b23f985e9f",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=56c054473fd260c13e4e7393eb203662195f5d4a1fada5314b8b52b23f985e9f",
},
// get-vanilla-query-order-key-case
V4SignerSuiteCase{
label: "get-vanilla-query-order-key-case",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/?foo=Zoo&foo=aha",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\nfoo=Zoo&foo=aha\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\ne25f777ba161a0f1baf778a87faf057187cf5987f17953320e3ca399feb5f00d",
signature: "be7148d34ebccdc6423b19085378aa0bee970bdc61d144bd1a8c48c33079ab09",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=be7148d34ebccdc6423b19085378aa0bee970bdc61d144bd1a8c48c33079ab09",
},
// get-vanilla-query-order-key
V4SignerSuiteCase{
label: "get-vanilla-query-order-key",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/?a=foo&b=foo",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\na=foo&b=foo\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n2f23d14fe13caebf6dfda346285c6d9c14f49eaca8f5ec55c627dd7404f7a727",
signature: "0dc122f3b28b831ab48ba65cb47300de53fbe91b577fe113edac383730254a3b",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=0dc122f3b28b831ab48ba65cb47300de53fbe91b577fe113edac383730254a3b",
},
// get-vanilla-query-order-value
V4SignerSuiteCase{
label: "get-vanilla-query-order-value",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/?foo=b&foo=a",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\nfoo=a&foo=b\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n33dffc220e89131f8f6157a35c40903daa658608d9129ff9489e5cf5bbd9b11b",
signature: "feb926e49e382bec75c9d7dcb2a1b6dc8aa50ca43c25d2bc51143768c0875acc",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=feb926e49e382bec75c9d7dcb2a1b6dc8aa50ca43c25d2bc51143768c0875acc",
},
// get-vanilla-query-unreserved
V4SignerSuiteCase{
label: "get-vanilla-query-unreserved",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/?-._~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz=-._~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n-._~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz=-._~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\nd2578f3156d4c9d180713d1ff20601d8a3eed0dd35447d24603d7d67414bd6b5",
signature: "f1498ddb4d6dae767d97c466fb92f1b59a2c71ca29ac954692663f9db03426fb",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=f1498ddb4d6dae767d97c466fb92f1b59a2c71ca29ac954692663f9db03426fb",
},
// get-vanilla-query
V4SignerSuiteCase{
label: "get-vanilla-query",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// get-vanilla-ut8-query
V4SignerSuiteCase{
label: "get-vanilla-ut8-query",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/?ሴ=bar",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n%E1%88%B4=bar\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\nde5065ff39c131e6c2e2bd19cd9345a794bf3b561eab20b8d97b2093fc2a979e",
signature: "6fb359e9a05394cc7074e0feb42573a2601abc0c869a953e8c5c12e4e01f1a8c",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=6fb359e9a05394cc7074e0feb42573a2601abc0c869a953e8c5c12e4e01f1a8c",
},
// get-vanilla
V4SignerSuiteCase{
label: "get-vanilla",
request: V4SignerSuiteCaseRequest{
method: "GET",
host: "host.foo.com",
url: "/",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "GET\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n366b91fb121d72a00f46bbe8d395f53a102b06dfb7e79636515208ed3fa606b1",
signature: "b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b27ccfbfa7df52a200ff74193ca6e32d4b48b8856fab7ebf1c595d0670a7e470",
},
// post-header-key-case
V4SignerSuiteCase{
label: "post-header-key-case",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"DATE:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "POST\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n05da62cee468d24ae84faff3c39f1b85540de60243c1bcaace39c0a2acc7b2c4",
signature: "22902d79e148b64e7571c3565769328423fe276eae4b26f83afceda9e767f726",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=22902d79e148b64e7571c3565769328423fe276eae4b26f83afceda9e767f726",
},
// post-header-key-sort
V4SignerSuiteCase{
label: "post-header-key-sort",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"DATE:Mon, 09 Sep 2011 23:36:00 GMT", "ZOO:zoobar"},
},
canonicalRequest: "POST\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\nzoo:zoobar\n\ndate;host;zoo\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n34e1bddeb99e76ee01d63b5e28656111e210529efeec6cdfd46a48e4c734545d",
signature: "b7a95a52518abbca0964a999a880429ab734f35ebbf1235bd79a5de87756dc4a",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host;zoo, Signature=b7a95a52518abbca0964a999a880429ab734f35ebbf1235bd79a5de87756dc4a",
},
// post-header-value-case
V4SignerSuiteCase{
label: "post-header-value-case",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"DATE:Mon, 09 Sep 2011 23:36:00 GMT", "zoo:ZOOBAR"},
},
canonicalRequest: "POST\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\nzoo:ZOOBAR\n\ndate;host;zoo\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n3aae6d8274b8c03e2cc96fc7d6bda4b9bd7a0a184309344470b2c96953e124aa",
signature: "273313af9d0c265c531e11db70bbd653f3ba074c1009239e8559d3987039cad7",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host;zoo, Signature=273313af9d0c265c531e11db70bbd653f3ba074c1009239e8559d3987039cad7",
},
// post-vanilla-empty-query-value
V4SignerSuiteCase{
label: "post-vanilla-empty-query-value",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/?foo=bar",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "POST\n/\nfoo=bar\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\ncd4f39132d8e60bb388831d734230460872b564871c47f5de62e62d1a68dbe1e",
signature: "b6e3b79003ce0743a491606ba1035a804593b0efb1e20a11cba83f8c25a57a92",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b6e3b79003ce0743a491606ba1035a804593b0efb1e20a11cba83f8c25a57a92",
},
// post-vanilla-query
V4SignerSuiteCase{
label: "post-vanilla-query",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/?foo=bar",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "POST\n/\nfoo=bar\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\ncd4f39132d8e60bb388831d734230460872b564871c47f5de62e62d1a68dbe1e",
signature: "b6e3b79003ce0743a491606ba1035a804593b0efb1e20a11cba83f8c25a57a92",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=b6e3b79003ce0743a491606ba1035a804593b0efb1e20a11cba83f8c25a57a92",
},
// post-vanilla
V4SignerSuiteCase{
label: "post-vanilla",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"Date:Mon, 09 Sep 2011 23:36:00 GMT"},
},
canonicalRequest: "POST\n/\n\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ndate;host\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n05da62cee468d24ae84faff3c39f1b85540de60243c1bcaace39c0a2acc7b2c4",
signature: "22902d79e148b64e7571c3565769328423fe276eae4b26f83afceda9e767f726",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=date;host, Signature=22902d79e148b64e7571c3565769328423fe276eae4b26f83afceda9e767f726",
},
// post-x-www-form-urlencoded-parameters
V4SignerSuiteCase{
label: "post-x-www-form-urlencoded-parameters",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"Content-Type:application/x-www-form-urlencoded; charset=utf8", "Date:Mon, 09 Sep 2011 23:36:00 GMT"},
body: "foo=bar",
},
canonicalRequest: "POST\n/\n\ncontent-type:application/x-www-form-urlencoded; charset=utf8\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ncontent-type;date;host\n3ba8907e7a252327488df390ed517c45b96dead033600219bdca7107d1d3f88a",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\nc4115f9e54b5cecf192b1eaa23b8e88ed8dc5391bd4fde7b3fff3d9c9fe0af1f",
signature: "b105eb10c6d318d2294de9d49dd8b031b55e3c3fe139f2e637da70511e9e7b71",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=content-type;date;host, Signature=b105eb10c6d318d2294de9d49dd8b031b55e3c3fe139f2e637da70511e9e7b71",
},
// post-x-www-form-urlencoded
V4SignerSuiteCase{
label: "post-x-www-form-urlencoded",
request: V4SignerSuiteCaseRequest{
method: "POST",
host: "host.foo.com",
url: "/",
headers: []string{"Content-Type:application/x-www-form-urlencoded", "Date:Mon, 09 Sep 2011 23:36:00 GMT"},
body: "foo=bar",
},
canonicalRequest: "POST\n/\n\ncontent-type:application/x-www-form-urlencoded\ndate:Mon, 09 Sep 2011 23:36:00 GMT\nhost:host.foo.com\n\ncontent-type;date;host\n3ba8907e7a252327488df390ed517c45b96dead033600219bdca7107d1d3f88a",
stringToSign: "AWS4-HMAC-SHA256\n20110909T233600Z\n20110909/us-east-1/host/aws4_request\n4c5c6e4b52fb5fb947a8733982a8a5a61b14f04345cbfe6e739236c76dd48f74",
signature: "5a15b22cf462f047318703b92e6f4f38884e4a7ab7b1d6426ca46a8bd1c26cbc",
authorization: "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20110909/us-east-1/host/aws4_request, SignedHeaders=content-type;date;host, Signature=5a15b22cf462f047318703b92e6f4f38884e4a7ab7b1d6426ca46a8bd1c26cbc",
},
)
}
func (s *V4SignerSuite) TestCases(c *check.C) {
signer := aws.NewV4Signer(s.auth, "host", s.region)
for _, testCase := range s.cases {
req, err := http.NewRequest(testCase.request.method, "http://"+testCase.request.host+testCase.request.url, strings.NewReader(testCase.request.body))
c.Assert(err, check.IsNil, check.Commentf("Testcase: %s", testCase.label))
for _, v := range testCase.request.headers {
h := strings.SplitN(v, ":", 2)
req.Header.Add(h[0], h[1])
}
req.Header.Set("host", req.Host)
t := signer.RequestTime(req)
canonicalRequest := signer.CanonicalRequest(req)
c.Check(canonicalRequest, check.Equals, testCase.canonicalRequest, check.Commentf("Testcase: %s", testCase.label))
stringToSign := signer.StringToSign(t, canonicalRequest)
c.Check(stringToSign, check.Equals, testCase.stringToSign, check.Commentf("Testcase: %s", testCase.label))
signature := signer.Signature(t, stringToSign)
c.Check(signature, check.Equals, testCase.signature, check.Commentf("Testcase: %s", testCase.label))
authorization := signer.Authorization(req.Header, t, signature)
c.Check(authorization, check.Equals, testCase.authorization, check.Commentf("Testcase: %s", testCase.label))
signer.Sign(req)
c.Check(req.Header.Get("Authorization"), check.Equals, testCase.authorization, check.Commentf("Testcase: %s", testCase.label))
}
}
func ExampleV4Signer() {
// Get auth from env vars
auth, err := aws.EnvAuth()
if err != nil {
fmt.Println(err)
}
// Create a signer with the auth, name of the service, and aws region
signer := aws.NewV4Signer(auth, "dynamodb", aws.USEast)
// Create a request
req, err := http.NewRequest("POST", aws.USEast.DynamoDBEndpoint, strings.NewReader("sample_request"))
if err != nil {
fmt.Println(err)
}
// Date or x-amz-date header is required to sign a request
req.Header.Add("Date", time.Now().UTC().Format(http.TimeFormat))
// Sign the request
signer.Sign(req)
// Issue signed request
http.DefaultClient.Do(req)
}

View file

@ -1,143 +0,0 @@
package cloudfront
import (
"crypto"
"crypto/rsa"
"crypto/sha1"
"encoding/base64"
"encoding/json"
"fmt"
"github.com/AdRoll/goamz/aws"
"net/url"
"strconv"
"strings"
"time"
)
type CloudFront struct {
BaseURL string
keyPairId string
key *rsa.PrivateKey
}
var base64Replacer = strings.NewReplacer("=", "_", "+", "-", "/", "~")
func NewKeyLess(auth aws.Auth, baseurl string) *CloudFront {
return &CloudFront{keyPairId: auth.AccessKey, BaseURL: baseurl}
}
func New(baseurl string, key *rsa.PrivateKey, keyPairId string) *CloudFront {
return &CloudFront{
BaseURL: baseurl,
keyPairId: keyPairId,
key: key,
}
}
type epochTime struct {
EpochTime int64 `json:"AWS:EpochTime"`
}
type condition struct {
DateLessThan epochTime
}
type statement struct {
Resource string
Condition condition
}
type policy struct {
Statement []statement
}
func buildPolicy(resource string, expireTime time.Time) ([]byte, error) {
p := &policy{
Statement: []statement{
statement{
Resource: resource,
Condition: condition{
DateLessThan: epochTime{
EpochTime: expireTime.Truncate(time.Millisecond).Unix(),
},
},
},
},
}
return json.Marshal(p)
}
func (cf *CloudFront) generateSignature(policy []byte) (string, error) {
hash := sha1.New()
_, err := hash.Write(policy)
if err != nil {
return "", err
}
hashed := hash.Sum(nil)
var signed []byte
if cf.key.Validate() == nil {
signed, err = rsa.SignPKCS1v15(nil, cf.key, crypto.SHA1, hashed)
if err != nil {
return "", err
}
} else {
signed = hashed
}
encoded := base64Replacer.Replace(base64.StdEncoding.EncodeToString(signed))
return encoded, nil
}
// Creates a signed url using RSAwithSHA1 as specified by
// http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-canned-policy.html#private-content-canned-policy-creating-signature
func (cf *CloudFront) CannedSignedURL(path, queryString string, expires time.Time) (string, error) {
resource := cf.BaseURL + path
if queryString != "" {
resource = path + "?" + queryString
}
policy, err := buildPolicy(resource, expires)
if err != nil {
return "", err
}
signature, err := cf.generateSignature(policy)
if err != nil {
return "", err
}
// TOOD: Do this once
uri, err := url.Parse(cf.BaseURL)
if err != nil {
return "", err
}
uri.RawQuery = queryString
if queryString != "" {
uri.RawQuery += "&"
}
expireTime := expires.Truncate(time.Millisecond).Unix()
uri.Path = path
uri.RawQuery += fmt.Sprintf("Expires=%d&Signature=%s&Key-Pair-Id=%s", expireTime, signature, cf.keyPairId)
return uri.String(), nil
}
func (cloudfront *CloudFront) SignedURL(path, querystrings string, expires time.Time) string {
policy := `{"Statement":[{"Resource":"` + path + "?" + querystrings + `,"Condition":{"DateLessThan":{"AWS:EpochTime":` + strconv.FormatInt(expires.Truncate(time.Millisecond).Unix(), 10) + `}}}]}`
hash := sha1.New()
hash.Write([]byte(policy))
b := hash.Sum(nil)
he := base64.StdEncoding.EncodeToString(b)
policySha1 := he
url := cloudfront.BaseURL + path + "?" + querystrings + "&Expires=" + strconv.FormatInt(expires.Unix(), 10) + "&Signature=" + policySha1 + "&Key-Pair-Id=" + cloudfront.keyPairId
return url
}

View file

@ -1,52 +0,0 @@
package cloudfront
import (
"crypto/x509"
"encoding/pem"
"io/ioutil"
"net/url"
"testing"
"time"
)
func TestSignedCannedURL(t *testing.T) {
rawKey, err := ioutil.ReadFile("testdata/key.pem")
if err != nil {
t.Fatal(err)
}
pemKey, _ := pem.Decode(rawKey)
privateKey, err := x509.ParsePKCS1PrivateKey(pemKey.Bytes)
if err != nil {
t.Fatal(err)
}
cf := &CloudFront{
key: privateKey,
keyPairId: "test-key-pair-1231245",
BaseURL: "https://cloudfront.com",
}
expireTime, err := time.Parse(time.RFC3339, "2014-03-28T14:00:21Z")
if err != nil {
t.Fatal(err)
}
query := make(url.Values)
query.Add("test", "value")
uri, err := cf.CannedSignedURL("test", "test=value", expireTime)
if err != nil {
t.Fatal(err)
}
parsed, err := url.Parse(uri)
if err != nil {
t.Fatal(err)
}
signature := parsed.Query().Get("Signature")
if signature == "" {
t.Fatal("Encoded signature is empty")
}
}

View file

@ -1,6 +0,0 @@
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC0yMzp9DkPAE99DhsEaGkqougL
vtmDKri4bZj0fFjmGmjyyjz9hlrsr87LHVWzH/7igK7040HG1UqypX3ijtJa9+6B
KHwBBctboU3y4GfwFwVAOumY9UytFpyPlgUFrffZLQAywKkT24OgcfEj0G5kiQn7
60wFnmSUtOuITo708QIDAQAB
-----END PUBLIC KEY-----

View file

@ -1,202 +0,0 @@
package s3
import (
"crypto/md5"
"encoding/base64"
"encoding/xml"
"net/url"
"strconv"
"time"
)
// Implements an interface for s3 bucket lifecycle configuration
// See goo.gl/d0bbDf for details.
const (
LifecycleRuleStatusEnabled = "Enabled"
LifecycleRuleStatusDisabled = "Disabled"
LifecycleRuleDateFormat = "2006-01-02"
StorageClassGlacier = "GLACIER"
)
type Expiration struct {
Days *uint `xml:"Days,omitempty"`
Date string `xml:"Date,omitempty"`
}
// Returns Date as a time.Time.
func (r *Expiration) ParseDate() (time.Time, error) {
return time.Parse(LifecycleRuleDateFormat, r.Date)
}
type Transition struct {
Days *uint `xml:"Days,omitempty"`
Date string `xml:"Date,omitempty"`
StorageClass string `xml:"StorageClass"`
}
// Returns Date as a time.Time.
func (r *Transition) ParseDate() (time.Time, error) {
return time.Parse(LifecycleRuleDateFormat, r.Date)
}
type NoncurrentVersionExpiration struct {
Days *uint `xml:"NoncurrentDays,omitempty"`
}
type NoncurrentVersionTransition struct {
Days *uint `xml:"NoncurrentDays,omitempty"`
StorageClass string `xml:"StorageClass"`
}
type LifecycleRule struct {
ID string `xml:"ID"`
Prefix string `xml:"Prefix"`
Status string `xml:"Status"`
NoncurrentVersionTransition *NoncurrentVersionTransition `xml:"NoncurrentVersionTransition,omitempty"`
NoncurrentVersionExpiration *NoncurrentVersionExpiration `xml:"NoncurrentVersionExpiration,omitempty"`
Transition *Transition `xml:"Transition,omitempty"`
Expiration *Expiration `xml:"Expiration,omitempty"`
}
// Create a lifecycle rule with arbitrary identifier id and object name prefix
// for which the rules should apply.
func NewLifecycleRule(id, prefix string) *LifecycleRule {
rule := &LifecycleRule{
ID: id,
Prefix: prefix,
Status: LifecycleRuleStatusEnabled,
}
return rule
}
// Adds a transition rule in days. Overwrites any previous transition rule.
func (r *LifecycleRule) SetTransitionDays(days uint) {
r.Transition = &Transition{
Days: &days,
StorageClass: StorageClassGlacier,
}
}
// Adds a transition rule as a date. Overwrites any previous transition rule.
func (r *LifecycleRule) SetTransitionDate(date time.Time) {
r.Transition = &Transition{
Date: date.Format(LifecycleRuleDateFormat),
StorageClass: StorageClassGlacier,
}
}
// Adds an expiration rule in days. Overwrites any previous expiration rule.
// Days must be > 0.
func (r *LifecycleRule) SetExpirationDays(days uint) {
r.Expiration = &Expiration{
Days: &days,
}
}
// Adds an expiration rule as a date. Overwrites any previous expiration rule.
func (r *LifecycleRule) SetExpirationDate(date time.Time) {
r.Expiration = &Expiration{
Date: date.Format(LifecycleRuleDateFormat),
}
}
// Adds a noncurrent version transition rule. Overwrites any previous
// noncurrent version transition rule.
func (r *LifecycleRule) SetNoncurrentVersionTransitionDays(days uint) {
r.NoncurrentVersionTransition = &NoncurrentVersionTransition{
Days: &days,
StorageClass: StorageClassGlacier,
}
}
// Adds a noncurrent version expiration rule. Days must be > 0. Overwrites
// any previous noncurrent version expiration rule.
func (r *LifecycleRule) SetNoncurrentVersionExpirationDays(days uint) {
r.NoncurrentVersionExpiration = &NoncurrentVersionExpiration{
Days: &days,
}
}
// Marks the rule as disabled.
func (r *LifecycleRule) Disable() {
r.Status = LifecycleRuleStatusDisabled
}
// Marks the rule as enabled (default).
func (r *LifecycleRule) Enable() {
r.Status = LifecycleRuleStatusEnabled
}
type LifecycleConfiguration struct {
XMLName xml.Name `xml:"LifecycleConfiguration"`
Rules *[]*LifecycleRule `xml:"Rule,omitempty"`
}
// Adds a LifecycleRule to the configuration.
func (c *LifecycleConfiguration) AddRule(r *LifecycleRule) {
var rules []*LifecycleRule
if c.Rules != nil {
rules = *c.Rules
}
rules = append(rules, r)
c.Rules = &rules
}
// Sets the bucket's lifecycle configuration.
func (b *Bucket) PutLifecycleConfiguration(c *LifecycleConfiguration) error {
doc, err := xml.Marshal(c)
if err != nil {
return err
}
buf := makeXmlBuffer(doc)
digest := md5.New()
size, err := digest.Write(buf.Bytes())
if err != nil {
return err
}
headers := map[string][]string{
"Content-Length": {strconv.FormatInt(int64(size), 10)},
"Content-MD5": {base64.StdEncoding.EncodeToString(digest.Sum(nil))},
}
req := &request{
path: "/",
method: "PUT",
bucket: b.Name,
headers: headers,
payload: buf,
params: url.Values{"lifecycle": {""}},
}
return b.S3.queryV4Sign(req, nil)
}
// Retrieves the lifecycle configuration for the bucket. AWS returns an error
// if no lifecycle found.
func (b *Bucket) GetLifecycleConfiguration() (*LifecycleConfiguration, error) {
req := &request{
method: "GET",
bucket: b.Name,
path: "/",
params: url.Values{"lifecycle": {""}},
}
conf := &LifecycleConfiguration{}
err := b.S3.queryV4Sign(req, conf)
return conf, err
}
// Delete the bucket's lifecycle configuration.
func (b *Bucket) DeleteLifecycleConfiguration() error {
req := &request{
method: "DELETE",
bucket: b.Name,
path: "/",
params: url.Values{"lifecycle": {""}},
}
return b.S3.queryV4Sign(req, nil)
}

View file

@ -1,205 +0,0 @@
package s3_test
import (
"encoding/xml"
"github.com/AdRoll/goamz/s3"
"gopkg.in/check.v1"
"io/ioutil"
"net/http"
"strings"
"time"
)
func (s *S) TestLifecycleConfiguration(c *check.C) {
date, err := time.Parse(s3.LifecycleRuleDateFormat, "2014-09-10")
c.Check(err, check.IsNil)
conf := &s3.LifecycleConfiguration{}
rule := s3.NewLifecycleRule("transition-days", "/")
rule.SetTransitionDays(7)
conf.AddRule(rule)
rule = s3.NewLifecycleRule("transition-date", "/")
rule.SetTransitionDate(date)
conf.AddRule(rule)
rule = s3.NewLifecycleRule("expiration-days", "")
rule.SetExpirationDays(1)
conf.AddRule(rule)
rule = s3.NewLifecycleRule("expiration-date", "")
rule.SetExpirationDate(date)
conf.AddRule(rule)
rule = s3.NewLifecycleRule("noncurrent-transition", "")
rule.SetNoncurrentVersionTransitionDays(11)
conf.AddRule(rule)
rule = s3.NewLifecycleRule("noncurrent-expiration", "")
rule.SetNoncurrentVersionExpirationDays(1011)
// Test Disable() and Enable() toggling
c.Check(rule.Status, check.Equals, s3.LifecycleRuleStatusEnabled)
rule.Disable()
c.Check(rule.Status, check.Equals, s3.LifecycleRuleStatusDisabled)
rule.Enable()
c.Check(rule.Status, check.Equals, s3.LifecycleRuleStatusEnabled)
rule.Disable()
c.Check(rule.Status, check.Equals, s3.LifecycleRuleStatusDisabled)
conf.AddRule(rule)
doc, err := xml.MarshalIndent(conf, "", " ")
c.Check(err, check.IsNil)
expectedDoc := `<LifecycleConfiguration>
<Rule>
<ID>transition-days</ID>
<Prefix>/</Prefix>
<Status>Enabled</Status>
<Transition>
<Days>7</Days>
<StorageClass>GLACIER</StorageClass>
</Transition>
</Rule>
<Rule>
<ID>transition-date</ID>
<Prefix>/</Prefix>
<Status>Enabled</Status>
<Transition>
<Date>2014-09-10</Date>
<StorageClass>GLACIER</StorageClass>
</Transition>
</Rule>
<Rule>
<ID>expiration-days</ID>
<Prefix></Prefix>
<Status>Enabled</Status>
<Expiration>
<Days>1</Days>
</Expiration>
</Rule>
<Rule>
<ID>expiration-date</ID>
<Prefix></Prefix>
<Status>Enabled</Status>
<Expiration>
<Date>2014-09-10</Date>
</Expiration>
</Rule>
<Rule>
<ID>noncurrent-transition</ID>
<Prefix></Prefix>
<Status>Enabled</Status>
<NoncurrentVersionTransition>
<NoncurrentDays>11</NoncurrentDays>
<StorageClass>GLACIER</StorageClass>
</NoncurrentVersionTransition>
</Rule>
<Rule>
<ID>noncurrent-expiration</ID>
<Prefix></Prefix>
<Status>Disabled</Status>
<NoncurrentVersionExpiration>
<NoncurrentDays>1011</NoncurrentDays>
</NoncurrentVersionExpiration>
</Rule>
</LifecycleConfiguration>`
c.Check(string(doc), check.Equals, expectedDoc)
// Unmarshalling test
conf2 := &s3.LifecycleConfiguration{}
err = xml.Unmarshal(doc, conf2)
c.Check(err, check.IsNil)
s.checkLifecycleConfigurationEqual(c, conf, conf2)
}
func (s *S) checkLifecycleConfigurationEqual(c *check.C, conf, conf2 *s3.LifecycleConfiguration) {
c.Check(len(*conf2.Rules), check.Equals, len(*conf.Rules))
for i, rule := range *conf2.Rules {
confRules := *conf.Rules
c.Check(rule, check.DeepEquals, confRules[i])
}
}
func (s *S) checkLifecycleRequest(c *check.C, req *http.Request) {
// ?lifecycle= is the only query param
v, ok := req.Form["lifecycle"]
c.Assert(ok, check.Equals, true)
c.Assert(v, check.HasLen, 1)
c.Assert(v[0], check.Equals, "")
c.Assert(req.Header["X-Amz-Date"], check.HasLen, 1)
c.Assert(req.Header["X-Amz-Date"][0], check.Not(check.Equals), "")
// Lifecycle methods require V4 auth
usesV4 := strings.HasPrefix(req.Header["Authorization"][0], "AWS4-HMAC-SHA256")
c.Assert(usesV4, check.Equals, true)
}
func (s *S) TestPutLifecycleConfiguration(c *check.C) {
testServer.Response(200, nil, "")
conf := &s3.LifecycleConfiguration{}
rule := s3.NewLifecycleRule("id", "")
rule.SetTransitionDays(7)
conf.AddRule(rule)
doc, err := xml.Marshal(conf)
c.Check(err, check.IsNil)
b := s.s3.Bucket("bucket")
err = b.PutLifecycleConfiguration(conf)
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/bucket/")
c.Assert(req.Header["Content-Md5"], check.HasLen, 1)
c.Assert(req.Header["Content-Md5"][0], check.Not(check.Equals), "")
s.checkLifecycleRequest(c, req)
// Check we sent the correct xml serialization
data, err := ioutil.ReadAll(req.Body)
req.Body.Close()
c.Assert(err, check.IsNil)
header := "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
c.Assert(string(data), check.Equals, header+string(doc))
}
func (s *S) TestGetLifecycleConfiguration(c *check.C) {
conf := &s3.LifecycleConfiguration{}
rule := s3.NewLifecycleRule("id", "")
rule.SetTransitionDays(7)
conf.AddRule(rule)
doc, err := xml.Marshal(conf)
c.Check(err, check.IsNil)
testServer.Response(200, nil, string(doc))
b := s.s3.Bucket("bucket")
conf2, err := b.GetLifecycleConfiguration()
c.Check(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/bucket/")
s.checkLifecycleRequest(c, req)
s.checkLifecycleConfigurationEqual(c, conf, conf2)
}
func (s *S) TestDeleteLifecycleConfiguration(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
err := b.DeleteLifecycleConfiguration()
c.Check(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "DELETE")
c.Assert(req.URL.Path, check.Equals, "/bucket/")
s.checkLifecycleRequest(c, req)
}

View file

@ -1,425 +0,0 @@
package s3_test
import (
"encoding/xml"
"github.com/AdRoll/goamz/s3"
"gopkg.in/check.v1"
"io"
"io/ioutil"
"strings"
)
func (s *S) TestInitMulti(c *check.C) {
testServer.Response(200, nil, InitMultiResultDump)
b := s.s3.Bucket("sample")
metadata := make(map[string][]string)
metadata["key1"] = []string{"value1"}
metadata["key2"] = []string{"value2"}
options := s3.Options{
SSE: true,
Meta: metadata,
ContentEncoding: "text/utf8",
CacheControl: "no-cache",
RedirectLocation: "http://github.com/AdRoll/goamz",
ContentMD5: "0000000000000000",
}
multi, err := b.InitMulti("multi", "text/plain", s3.Private, options)
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "POST")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Header["Content-Type"], check.DeepEquals, []string{"text/plain"})
c.Assert(req.Header["X-Amz-Acl"], check.DeepEquals, []string{"private"})
c.Assert(req.Form["uploads"], check.DeepEquals, []string{""})
c.Assert(req.Header["X-Amz-Server-Side-Encryption"], check.DeepEquals, []string{"AES256"})
c.Assert(req.Header["Content-Encoding"], check.DeepEquals, []string{"text/utf8"})
c.Assert(req.Header["Cache-Control"], check.DeepEquals, []string{"no-cache"})
c.Assert(req.Header["Content-Md5"], check.DeepEquals, []string{"0000000000000000"})
c.Assert(req.Header["X-Amz-Website-Redirect-Location"], check.DeepEquals, []string{"http://github.com/AdRoll/goamz"})
c.Assert(req.Header["X-Amz-Meta-Key1"], check.DeepEquals, []string{"value1"})
c.Assert(req.Header["X-Amz-Meta-Key2"], check.DeepEquals, []string{"value2"})
c.Assert(multi.UploadId, check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
}
func (s *S) TestMultiNoPreviousUpload(c *check.C) {
// Don't retry the NoSuchUpload error.
s.DisableRetries()
testServer.Response(404, nil, NoSuchUploadErrorDump)
testServer.Response(200, nil, InitMultiResultDump)
b := s.s3.Bucket("sample")
multi, err := b.Multi("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/")
c.Assert(req.Form["uploads"], check.DeepEquals, []string{""})
c.Assert(req.Form["prefix"], check.DeepEquals, []string{"multi"})
req = testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "POST")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form["uploads"], check.DeepEquals, []string{""})
c.Assert(multi.UploadId, check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
}
func (s *S) TestMultiReturnOld(c *check.C) {
testServer.Response(200, nil, ListMultiResultDump)
b := s.s3.Bucket("sample")
multi, err := b.Multi("multi1", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
c.Assert(multi.Key, check.Equals, "multi1")
c.Assert(multi.UploadId, check.Equals, "iUVug89pPvSswrikD")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/")
c.Assert(req.Form["uploads"], check.DeepEquals, []string{""})
c.Assert(req.Form["prefix"], check.DeepEquals, []string{"multi1"})
}
func (s *S) TestListParts(c *check.C) {
testServer.Response(200, nil, InitMultiResultDump)
testServer.Response(200, nil, ListPartsResultDump1)
testServer.Response(404, nil, NoSuchUploadErrorDump) // :-(
testServer.Response(200, nil, ListPartsResultDump2)
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
parts, err := multi.ListParts()
c.Assert(err, check.IsNil)
c.Assert(parts, check.HasLen, 3)
c.Assert(parts[0].N, check.Equals, 1)
c.Assert(parts[0].Size, check.Equals, int64(5))
c.Assert(parts[0].ETag, check.Equals, `"ffc88b4ca90a355f8ddba6b2c3b2af5c"`)
c.Assert(parts[1].N, check.Equals, 2)
c.Assert(parts[1].Size, check.Equals, int64(5))
c.Assert(parts[1].ETag, check.Equals, `"d067a0fa9dc61a6e7195ca99696b5a89"`)
c.Assert(parts[2].N, check.Equals, 3)
c.Assert(parts[2].Size, check.Equals, int64(5))
c.Assert(parts[2].ETag, check.Equals, `"49dcd91231f801159e893fb5c6674985"`)
testServer.WaitRequest()
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form.Get("uploadId"), check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
c.Assert(req.Form["max-parts"], check.DeepEquals, []string{"1000"})
testServer.WaitRequest() // The internal error.
req = testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form.Get("uploadId"), check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
c.Assert(req.Form["max-parts"], check.DeepEquals, []string{"1000"})
c.Assert(req.Form["part-number-marker"], check.DeepEquals, []string{"2"})
}
func (s *S) TestPutPart(c *check.C) {
headers := map[string]string{
"ETag": `"26f90efd10d614f100252ff56d88dad8"`,
}
testServer.Response(200, nil, InitMultiResultDump)
testServer.Response(200, headers, "")
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
part, err := multi.PutPart(1, strings.NewReader("<part 1>"))
c.Assert(err, check.IsNil)
c.Assert(part.N, check.Equals, 1)
c.Assert(part.Size, check.Equals, int64(8))
c.Assert(part.ETag, check.Equals, headers["ETag"])
testServer.WaitRequest()
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form.Get("uploadId"), check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
c.Assert(req.Form["partNumber"], check.DeepEquals, []string{"1"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"8"})
c.Assert(req.Header["Content-Md5"], check.DeepEquals, []string{"JvkO/RDWFPEAJS/1bYja2A=="})
}
func (s *S) TestPutPartCopy(c *check.C) {
testServer.Response(200, nil, InitMultiResultDump)
// PutPartCopy makes a Head request internally to verify access to the source object
// and obtain its size
testServer.Response(200, nil, "content")
testServer.Response(200, nil, PutCopyResultDump)
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
res, part, err := multi.PutPartCopy(1, s3.CopyOptions{}, "source-bucket/\u00FCber-fil\u00E9.jpg")
c.Assert(err, check.IsNil)
c.Assert(part.N, check.Equals, 1)
c.Assert(part.Size, check.Equals, int64(7))
c.Assert(res, check.DeepEquals, &s3.CopyObjectResult{
ETag: `"9b2cf535f27731c974343645a3985328"`,
LastModified: `2009-10-28T22:32:00`})
// Verify the Head request
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "POST")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
c.Assert(err, check.IsNil)
testServer.WaitRequest()
req = testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form.Get("uploadId"), check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
c.Assert(req.Form["partNumber"], check.DeepEquals, []string{"1"})
c.Assert(req.Header["X-Amz-Copy-Source"], check.DeepEquals, []string{`source-bucket%2F%C3%BCber-fil%C3%A9.jpg`})
}
func readAll(r io.Reader) string {
data, err := ioutil.ReadAll(r)
if err != nil {
panic(err)
}
return string(data)
}
func (s *S) TestPutAllNoPreviousUpload(c *check.C) {
// Don't retry the NoSuchUpload error.
s.DisableRetries()
etag1 := map[string]string{"ETag": `"etag1"`}
etag2 := map[string]string{"ETag": `"etag2"`}
etag3 := map[string]string{"ETag": `"etag3"`}
testServer.Response(200, nil, InitMultiResultDump)
testServer.Response(404, nil, NoSuchUploadErrorDump)
testServer.Response(200, etag1, "")
testServer.Response(200, etag2, "")
testServer.Response(200, etag3, "")
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
parts, err := multi.PutAll(strings.NewReader("part1part2last"), 5)
c.Assert(parts, check.HasLen, 3)
c.Assert(parts[0].ETag, check.Equals, `"etag1"`)
c.Assert(parts[1].ETag, check.Equals, `"etag2"`)
c.Assert(parts[2].ETag, check.Equals, `"etag3"`)
c.Assert(err, check.IsNil)
// Init
testServer.WaitRequest()
// List old parts. Won't find anything.
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
// Send part 1.
req = testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form["partNumber"], check.DeepEquals, []string{"1"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"5"})
c.Assert(readAll(req.Body), check.Equals, "part1")
// Send part 2.
req = testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form["partNumber"], check.DeepEquals, []string{"2"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"5"})
c.Assert(readAll(req.Body), check.Equals, "part2")
// Send part 3 with shorter body.
req = testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form["partNumber"], check.DeepEquals, []string{"3"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"4"})
c.Assert(readAll(req.Body), check.Equals, "last")
}
func (s *S) TestPutAllZeroSizeFile(c *check.C) {
// Don't retry the NoSuchUpload error.
s.DisableRetries()
etag1 := map[string]string{"ETag": `"etag1"`}
testServer.Response(200, nil, InitMultiResultDump)
testServer.Response(404, nil, NoSuchUploadErrorDump)
testServer.Response(200, etag1, "")
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
// Must send at least one part, so that completing it will work.
parts, err := multi.PutAll(strings.NewReader(""), 5)
c.Assert(parts, check.HasLen, 1)
c.Assert(parts[0].ETag, check.Equals, `"etag1"`)
c.Assert(err, check.IsNil)
// Init
testServer.WaitRequest()
// List old parts. Won't find anything.
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
// Send empty part.
req = testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form["partNumber"], check.DeepEquals, []string{"1"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"0"})
c.Assert(readAll(req.Body), check.Equals, "")
}
func (s *S) TestPutAllResume(c *check.C) {
etag2 := map[string]string{"ETag": `"etag2"`}
testServer.Response(200, nil, InitMultiResultDump)
testServer.Response(200, nil, ListPartsResultDump1)
testServer.Response(200, nil, ListPartsResultDump2)
testServer.Response(200, etag2, "")
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
// "part1" and "part3" match the checksums in ResultDump1.
// The middle one is a mismatch (it refers to "part2").
parts, err := multi.PutAll(strings.NewReader("part1partXpart3"), 5)
c.Assert(parts, check.HasLen, 3)
c.Assert(parts[0].N, check.Equals, 1)
c.Assert(parts[0].Size, check.Equals, int64(5))
c.Assert(parts[0].ETag, check.Equals, `"ffc88b4ca90a355f8ddba6b2c3b2af5c"`)
c.Assert(parts[1].N, check.Equals, 2)
c.Assert(parts[1].Size, check.Equals, int64(5))
c.Assert(parts[1].ETag, check.Equals, `"etag2"`)
c.Assert(parts[2].N, check.Equals, 3)
c.Assert(parts[2].Size, check.Equals, int64(5))
c.Assert(parts[2].ETag, check.Equals, `"49dcd91231f801159e893fb5c6674985"`)
c.Assert(err, check.IsNil)
// Init
testServer.WaitRequest()
// List old parts, broken in two requests.
for i := 0; i < 2; i++ {
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
}
// Send part 2, as it didn't match the checksum.
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form["partNumber"], check.DeepEquals, []string{"2"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"5"})
c.Assert(readAll(req.Body), check.Equals, "partX")
}
func (s *S) TestMultiComplete(c *check.C) {
testServer.Response(200, nil, InitMultiResultDump)
// Note the 200 response. Completing will hold the connection on some
// kind of long poll, and may return a late error even after a 200.
testServer.Response(200, nil, InternalErrorDump)
testServer.Response(200, nil, "")
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
err = multi.Complete([]s3.Part{{2, `"ETag2"`, 32}, {1, `"ETag1"`, 64}})
c.Assert(err, check.IsNil)
testServer.WaitRequest()
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "POST")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form.Get("uploadId"), check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
var payload struct {
XMLName xml.Name
Part []struct {
PartNumber int
ETag string
}
}
dec := xml.NewDecoder(req.Body)
err = dec.Decode(&payload)
c.Assert(err, check.IsNil)
c.Assert(payload.XMLName.Local, check.Equals, "CompleteMultipartUpload")
c.Assert(len(payload.Part), check.Equals, 2)
c.Assert(payload.Part[0].PartNumber, check.Equals, 1)
c.Assert(payload.Part[0].ETag, check.Equals, `"ETag1"`)
c.Assert(payload.Part[1].PartNumber, check.Equals, 2)
c.Assert(payload.Part[1].ETag, check.Equals, `"ETag2"`)
}
func (s *S) TestMultiAbort(c *check.C) {
testServer.Response(200, nil, InitMultiResultDump)
testServer.Response(200, nil, "")
b := s.s3.Bucket("sample")
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
err = multi.Abort()
c.Assert(err, check.IsNil)
testServer.WaitRequest()
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "DELETE")
c.Assert(req.URL.Path, check.Equals, "/sample/multi")
c.Assert(req.Form.Get("uploadId"), check.Matches, "JNbR_[A-Za-z0-9.]+QQ--")
}
func (s *S) TestListMulti(c *check.C) {
testServer.Response(200, nil, ListMultiResultDump)
b := s.s3.Bucket("sample")
multis, prefixes, err := b.ListMulti("", "/")
c.Assert(err, check.IsNil)
c.Assert(prefixes, check.DeepEquals, []string{"a/", "b/"})
c.Assert(multis, check.HasLen, 2)
c.Assert(multis[0].Key, check.Equals, "multi1")
c.Assert(multis[0].UploadId, check.Equals, "iUVug89pPvSswrikD")
c.Assert(multis[1].Key, check.Equals, "multi2")
c.Assert(multis[1].UploadId, check.Equals, "DkirwsSvPp98guVUi")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/sample/")
c.Assert(req.Form["uploads"], check.DeepEquals, []string{""})
c.Assert(req.Form["prefix"], check.DeepEquals, []string{""})
c.Assert(req.Form["delimiter"], check.DeepEquals, []string{"/"})
c.Assert(req.Form["max-uploads"], check.DeepEquals, []string{"1000"})
}

View file

@ -1,239 +0,0 @@
package s3_test
var PutCopyResultDump = `
<?xml version="1.0" encoding="UTF-8"?>
<CopyObjectResult>
<LastModified>2009-10-28T22:32:00</LastModified>
<ETag>&quot;9b2cf535f27731c974343645a3985328&quot;</ETag>
</CopyObjectResult>
`
var GetObjectErrorDump = `
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message>
<BucketName>non-existent-bucket</BucketName><RequestId>3F1B667FAD71C3D8</RequestId>
<HostId>L4ee/zrm1irFXY5F45fKXIRdOf9ktsKY/8TDVawuMK2jWRb1RF84i1uBzkdNqS5D</HostId></Error>
`
var GetListResultDump1 = `
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01">
<Name>quotes</Name>
<Prefix>N</Prefix>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>Nelson</Key>
<LastModified>2006-01-01T12:00:00.000Z</LastModified>
<ETag>&quot;828ef3fdfa96f00ad9f27c383fc9ac7f&quot;</ETag>
<Size>5</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>bcaf161ca5fb16fd081034f</ID>
<DisplayName>webfile</DisplayName>
</Owner>
</Contents>
<Contents>
<Key>Neo</Key>
<LastModified>2006-01-01T12:00:00.000Z</LastModified>
<ETag>&quot;828ef3fdfa96f00ad9f27c383fc9ac7f&quot;</ETag>
<Size>4</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>bcaf1ffd86a5fb16fd081034f</ID>
<DisplayName>webfile</DisplayName>
</Owner>
</Contents>
</ListBucketResult>
`
var GetListResultDump2 = `
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>example-bucket</Name>
<Prefix>photos/2006/</Prefix>
<Marker>some-marker</Marker>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>photos/2006/feb/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>photos/2006/jan/</Prefix>
</CommonPrefixes>
</ListBucketResult>
`
var InitMultiResultDump = `
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>sample</Bucket>
<Key>multi</Key>
<UploadId>JNbR_cMdwnGiD12jKAd6WK2PUkfj2VxA7i4nCwjE6t71nI9Tl3eVDPFlU0nOixhftH7I17ZPGkV3QA.l7ZD.QQ--</UploadId>
</InitiateMultipartUploadResult>
`
var ListPartsResultDump1 = `
<?xml version="1.0" encoding="UTF-8"?>
<ListPartsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>sample</Bucket>
<Key>multi</Key>
<UploadId>JNbR_cMdwnGiD12jKAd6WK2PUkfj2VxA7i4nCwjE6t71nI9Tl3eVDPFlU0nOixhftH7I17ZPGkV3QA.l7ZD.QQ--</UploadId>
<Initiator>
<ID>bb5c0f63b0b25f2d099c</ID>
<DisplayName>joe</DisplayName>
</Initiator>
<Owner>
<ID>bb5c0f63b0b25f2d099c</ID>
<DisplayName>joe</DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
<PartNumberMarker>0</PartNumberMarker>
<NextPartNumberMarker>2</NextPartNumberMarker>
<MaxParts>2</MaxParts>
<IsTruncated>true</IsTruncated>
<Part>
<PartNumber>1</PartNumber>
<LastModified>2013-01-30T13:45:51.000Z</LastModified>
<ETag>&quot;ffc88b4ca90a355f8ddba6b2c3b2af5c&quot;</ETag>
<Size>5</Size>
</Part>
<Part>
<PartNumber>2</PartNumber>
<LastModified>2013-01-30T13:45:52.000Z</LastModified>
<ETag>&quot;d067a0fa9dc61a6e7195ca99696b5a89&quot;</ETag>
<Size>5</Size>
</Part>
</ListPartsResult>
`
var ListPartsResultDump2 = `
<?xml version="1.0" encoding="UTF-8"?>
<ListPartsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>sample</Bucket>
<Key>multi</Key>
<UploadId>JNbR_cMdwnGiD12jKAd6WK2PUkfj2VxA7i4nCwjE6t71nI9Tl3eVDPFlU0nOixhftH7I17ZPGkV3QA.l7ZD.QQ--</UploadId>
<Initiator>
<ID>bb5c0f63b0b25f2d099c</ID>
<DisplayName>joe</DisplayName>
</Initiator>
<Owner>
<ID>bb5c0f63b0b25f2d099c</ID>
<DisplayName>joe</DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
<PartNumberMarker>2</PartNumberMarker>
<NextPartNumberMarker>3</NextPartNumberMarker>
<MaxParts>2</MaxParts>
<IsTruncated>false</IsTruncated>
<Part>
<PartNumber>3</PartNumber>
<LastModified>2013-01-30T13:46:50.000Z</LastModified>
<ETag>&quot;49dcd91231f801159e893fb5c6674985&quot;</ETag>
<Size>5</Size>
</Part>
</ListPartsResult>
`
var ListMultiResultDump = `
<?xml version="1.0"?>
<ListMultipartUploadsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>goamz-test-bucket-us-east-1-akiajk3wyewhctyqbf7a</Bucket>
<KeyMarker/>
<UploadIdMarker/>
<NextKeyMarker>multi1</NextKeyMarker>
<NextUploadIdMarker>iUVug89pPvSswrikD72p8uO62EzhNtpDxRmwC5WSiWDdK9SfzmDqe3xpP1kMWimyimSnz4uzFc3waVM5ufrKYQ--</NextUploadIdMarker>
<Delimiter>/</Delimiter>
<MaxUploads>1000</MaxUploads>
<IsTruncated>false</IsTruncated>
<Upload>
<Key>multi1</Key>
<UploadId>iUVug89pPvSswrikD</UploadId>
<Initiator>
<ID>bb5c0f63b0b25f2d0</ID>
<DisplayName>gustavoniemeyer</DisplayName>
</Initiator>
<Owner>
<ID>bb5c0f63b0b25f2d0</ID>
<DisplayName>gustavoniemeyer</DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
<Initiated>2013-01-30T18:15:47.000Z</Initiated>
</Upload>
<Upload>
<Key>multi2</Key>
<UploadId>DkirwsSvPp98guVUi</UploadId>
<Initiator>
<ID>bb5c0f63b0b25f2d0</ID>
<DisplayName>joe</DisplayName>
</Initiator>
<Owner>
<ID>bb5c0f63b0b25f2d0</ID>
<DisplayName>joe</DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
<Initiated>2013-01-30T18:15:47.000Z</Initiated>
</Upload>
<CommonPrefixes>
<Prefix>a/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>b/</Prefix>
</CommonPrefixes>
</ListMultipartUploadsResult>
`
var NoSuchUploadErrorDump = `
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>NoSuchUpload</Code>
<Message>Not relevant</Message>
<BucketName>sample</BucketName>
<RequestId>3F1B667FAD71C3D8</RequestId>
<HostId>kjhwqk</HostId>
</Error>
`
var InternalErrorDump = `
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InternalError</Code>
<Message>Not relevant</Message>
<BucketName>sample</BucketName>
<RequestId>3F1B667FAD71C3D8</RequestId>
<HostId>kjhwqk</HostId>
</Error>
`
var GetServiceDump = `
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01">
<Owner>
<ID>bcaf1ffd86f461ca5fb16fd081034f</ID>
<DisplayName>webfile</DisplayName>
</Owner>
<Buckets>
<Bucket>
<Name>quotes</Name>
<CreationDate>2006-02-03T16:45:09.000Z</CreationDate>
</Bucket>
<Bucket>
<Name>samples</Name>
<CreationDate>2006-02-03T16:41:58.000Z</CreationDate>
</Bucket>
</Buckets>
</ListAllMyBucketsResult>
`
var GetLocationUsStandard = `
<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>
`
var GetLocationUsWest1 = `
<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">us-west-1</LocationConstraint>
`
var BucketWebsiteConfigurationDump = `<?xml version="1.0" encoding="UTF-8"?>
<WebsiteConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><RedirectAllRequestsTo><HostName>example.com</HostName></RedirectAllRequestsTo></WebsiteConfiguration>`

View file

@ -1,502 +0,0 @@
package s3_test
import (
"bytes"
"io/ioutil"
"net/http"
"testing"
"time"
"github.com/AdRoll/goamz/aws"
"github.com/AdRoll/goamz/s3"
"github.com/AdRoll/goamz/testutil"
"gopkg.in/check.v1"
)
func Test(t *testing.T) {
check.TestingT(t)
}
type S struct {
s3 *s3.S3
}
var _ = check.Suite(&S{})
var testServer = testutil.NewHTTPServer()
func (s *S) SetUpSuite(c *check.C) {
testServer.Start()
auth := aws.Auth{AccessKey: "abc", SecretKey: "123"}
s.s3 = s3.New(auth, aws.Region{Name: "faux-region-1", S3Endpoint: testServer.URL})
}
func (s *S) TearDownSuite(c *check.C) {
s3.SetAttemptStrategy(nil)
}
func (s *S) SetUpTest(c *check.C) {
attempts := aws.AttemptStrategy{
Total: 300 * time.Millisecond,
Delay: 100 * time.Millisecond,
}
s3.SetAttemptStrategy(&attempts)
}
func (s *S) TearDownTest(c *check.C) {
testServer.Flush()
}
func (s *S) DisableRetries() {
s3.SetAttemptStrategy(&aws.AttemptStrategy{})
}
// PutBucket docs: http://goo.gl/kBTCu
func (s *S) TestPutBucket(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
err := b.PutBucket(s3.Private)
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/bucket/")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
}
// PutBucketWebsite docs: http://goo.gl/TpRlUy
func (s *S) TestPutBucketWebsite(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
config := s3.WebsiteConfiguration{
RedirectAllRequestsTo: &s3.RedirectAllRequestsTo{HostName: "example.com"},
}
err := b.PutBucketWebsite(config)
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
body, err := ioutil.ReadAll(req.Body)
req.Body.Close()
c.Assert(err, check.IsNil)
c.Assert(string(body), check.Equals, BucketWebsiteConfigurationDump)
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/bucket/")
c.Assert(req.URL.RawQuery, check.Equals, "website=")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
}
// Head docs: http://bit.ly/17K1ylI
func (s *S) TestHead(c *check.C) {
testServer.Response(200, nil, "content")
b := s.s3.Bucket("bucket")
resp, err := b.Head("name", nil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "HEAD")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
c.Assert(err, check.IsNil)
c.Assert(resp.ContentLength, check.FitsTypeOf, int64(0))
c.Assert(resp, check.FitsTypeOf, &http.Response{})
}
// DeleteBucket docs: http://goo.gl/GoBrY
func (s *S) TestDelBucket(c *check.C) {
testServer.Response(204, nil, "")
b := s.s3.Bucket("bucket")
err := b.DelBucket()
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "DELETE")
c.Assert(req.URL.Path, check.Equals, "/bucket/")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
}
// GetObject docs: http://goo.gl/isCO7
func (s *S) TestGet(c *check.C) {
testServer.Response(200, nil, "content")
b := s.s3.Bucket("bucket")
data, err := b.Get("name")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
c.Assert(err, check.IsNil)
c.Assert(string(data), check.Equals, "content")
}
func (s *S) TestGetWithPlus(c *check.C) {
testServer.Response(200, nil, "content")
b := s.s3.Bucket("bucket")
_, err := b.Get("has+plus")
req := testServer.WaitRequest()
c.Assert(err, check.IsNil)
c.Assert(req.RequestURI, check.Equals, "http://localhost:4444/bucket/has%2Bplus")
}
func (s *S) TestURL(c *check.C) {
testServer.Response(200, nil, "content")
b := s.s3.Bucket("bucket")
url := b.URL("name")
r, err := http.Get(url)
c.Assert(err, check.IsNil)
data, err := ioutil.ReadAll(r.Body)
r.Body.Close()
c.Assert(err, check.IsNil)
c.Assert(string(data), check.Equals, "content")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
}
func (s *S) TestGetReader(c *check.C) {
testServer.Response(200, nil, "content")
b := s.s3.Bucket("bucket")
rc, err := b.GetReader("name")
c.Assert(err, check.IsNil)
data, err := ioutil.ReadAll(rc)
rc.Close()
c.Assert(err, check.IsNil)
c.Assert(string(data), check.Equals, "content")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
}
func (s *S) TestGetNotFound(c *check.C) {
for i := 0; i < 10; i++ {
testServer.Response(404, nil, GetObjectErrorDump)
}
b := s.s3.Bucket("non-existent-bucket")
data, err := b.Get("non-existent")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/non-existent-bucket/non-existent")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
s3err, _ := err.(*s3.Error)
c.Assert(s3err, check.NotNil)
c.Assert(s3err.StatusCode, check.Equals, 404)
c.Assert(s3err.BucketName, check.Equals, "non-existent-bucket")
c.Assert(s3err.RequestId, check.Equals, "3F1B667FAD71C3D8")
c.Assert(s3err.HostId, check.Equals, "L4ee/zrm1irFXY5F45fKXIRdOf9ktsKY/8TDVawuMK2jWRb1RF84i1uBzkdNqS5D")
c.Assert(s3err.Code, check.Equals, "NoSuchBucket")
c.Assert(s3err.Message, check.Equals, "The specified bucket does not exist")
c.Assert(s3err.Error(), check.Equals, "The specified bucket does not exist")
c.Assert(data, check.IsNil)
}
// PutObject docs: http://goo.gl/FEBPD
func (s *S) TestPutObject(c *check.C) {
testServer.Response(200, nil, "")
const DISPOSITION = "attachment; filename=\"0x1a2b3c.jpg\""
b := s.s3.Bucket("bucket")
err := b.Put("name", []byte("content"), "content-type", s3.Private, s3.Options{ContentDisposition: DISPOSITION})
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.DeepEquals), []string{""})
c.Assert(req.Header["Content-Type"], check.DeepEquals, []string{"content-type"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"7"})
c.Assert(req.Header["Content-Disposition"], check.DeepEquals, []string{DISPOSITION})
//c.Assert(req.Header["Content-MD5"], gocheck.DeepEquals, "...")
c.Assert(req.Header["X-Amz-Acl"], check.DeepEquals, []string{"private"})
}
func (s *S) TestPutObjectReducedRedundancy(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
err := b.Put("name", []byte("content"), "content-type", s3.Private, s3.Options{StorageClass: s3.ReducedRedundancy})
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.DeepEquals), []string{""})
c.Assert(req.Header["Content-Type"], check.DeepEquals, []string{"content-type"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"7"})
c.Assert(req.Header["X-Amz-Storage-Class"], check.DeepEquals, []string{"REDUCED_REDUNDANCY"})
}
// PutCopy docs: http://goo.gl/mhEHtA
func (s *S) TestPutCopy(c *check.C) {
testServer.Response(200, nil, PutCopyResultDump)
b := s.s3.Bucket("bucket")
res, err := b.PutCopy("name", s3.Private, s3.CopyOptions{},
// 0xFC is &uuml; - 0xE9 is &eacute;
"source-bucket/\u00FCber-fil\u00E9.jpg")
c.Assert(err, check.IsNil)
c.Assert(res, check.DeepEquals, &s3.CopyObjectResult{
ETag: `"9b2cf535f27731c974343645a3985328"`,
LastModified: `2009-10-28T22:32:00`})
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.DeepEquals), []string{""})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"0"})
c.Assert(req.Header["X-Amz-Copy-Source"], check.DeepEquals, []string{`source-bucket%2F%C3%BCber-fil%C3%A9.jpg`})
c.Assert(req.Header["X-Amz-Acl"], check.DeepEquals, []string{"private"})
}
func (s *S) TestPutObjectReadTimeout(c *check.C) {
s.s3.ReadTimeout = 50 * time.Millisecond
defer func() {
s.s3.ReadTimeout = 0
}()
b := s.s3.Bucket("bucket")
err := b.Put("name", []byte("content"), "content-type", s3.Private, s3.Options{})
// Make sure that we get a timeout error.
c.Assert(err, check.NotNil)
// Set the response after the request times out so that the next request will work.
testServer.Response(200, nil, "")
// This time set the response within our timeout period so that we expect the call
// to return successfully.
go func() {
time.Sleep(25 * time.Millisecond)
testServer.Response(200, nil, "")
}()
err = b.Put("name", []byte("content"), "content-type", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
}
func (s *S) TestPutReader(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
buf := bytes.NewBufferString("content")
err := b.PutReader("name", buf, int64(buf.Len()), "content-type", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "PUT")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.DeepEquals), []string{""})
c.Assert(req.Header["Content-Type"], check.DeepEquals, []string{"content-type"})
c.Assert(req.Header["Content-Length"], check.DeepEquals, []string{"7"})
//c.Assert(req.Header["Content-MD5"], gocheck.Equals, "...")
c.Assert(req.Header["X-Amz-Acl"], check.DeepEquals, []string{"private"})
}
// DelObject docs: http://goo.gl/APeTt
func (s *S) TestDelObject(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
err := b.Del("name")
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "DELETE")
c.Assert(req.URL.Path, check.Equals, "/bucket/name")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
}
func (s *S) TestDelMultiObjects(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
objects := []s3.Object{s3.Object{Key: "test"}}
err := b.DelMulti(s3.Delete{
Quiet: false,
Objects: objects,
})
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "POST")
c.Assert(req.URL.RawQuery, check.Equals, "delete=")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
c.Assert(req.Header["Content-MD5"], check.Not(check.Equals), "")
c.Assert(req.Header["Content-Type"], check.Not(check.Equals), "")
c.Assert(req.ContentLength, check.Not(check.Equals), "")
}
// Bucket List Objects docs: http://goo.gl/YjQTc
func (s *S) TestList(c *check.C) {
testServer.Response(200, nil, GetListResultDump1)
b := s.s3.Bucket("quotes")
data, err := b.List("N", "", "", 0)
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/quotes/")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
c.Assert(req.Form["prefix"], check.DeepEquals, []string{"N"})
c.Assert(req.Form["delimiter"], check.DeepEquals, []string{""})
c.Assert(req.Form["marker"], check.DeepEquals, []string{""})
c.Assert(req.Form["max-keys"], check.DeepEquals, []string(nil))
c.Assert(data.Name, check.Equals, "quotes")
c.Assert(data.Prefix, check.Equals, "N")
c.Assert(data.IsTruncated, check.Equals, false)
c.Assert(len(data.Contents), check.Equals, 2)
c.Assert(data.Contents[0].Key, check.Equals, "Nelson")
c.Assert(data.Contents[0].LastModified, check.Equals, "2006-01-01T12:00:00.000Z")
c.Assert(data.Contents[0].ETag, check.Equals, `"828ef3fdfa96f00ad9f27c383fc9ac7f"`)
c.Assert(data.Contents[0].Size, check.Equals, int64(5))
c.Assert(data.Contents[0].StorageClass, check.Equals, "STANDARD")
c.Assert(data.Contents[0].Owner.ID, check.Equals, "bcaf161ca5fb16fd081034f")
c.Assert(data.Contents[0].Owner.DisplayName, check.Equals, "webfile")
c.Assert(data.Contents[1].Key, check.Equals, "Neo")
c.Assert(data.Contents[1].LastModified, check.Equals, "2006-01-01T12:00:00.000Z")
c.Assert(data.Contents[1].ETag, check.Equals, `"828ef3fdfa96f00ad9f27c383fc9ac7f"`)
c.Assert(data.Contents[1].Size, check.Equals, int64(4))
c.Assert(data.Contents[1].StorageClass, check.Equals, "STANDARD")
c.Assert(data.Contents[1].Owner.ID, check.Equals, "bcaf1ffd86a5fb16fd081034f")
c.Assert(data.Contents[1].Owner.DisplayName, check.Equals, "webfile")
}
func (s *S) TestListWithDelimiter(c *check.C) {
testServer.Response(200, nil, GetListResultDump2)
b := s.s3.Bucket("quotes")
data, err := b.List("photos/2006/", "/", "some-marker", 1000)
c.Assert(err, check.IsNil)
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "GET")
c.Assert(req.URL.Path, check.Equals, "/quotes/")
c.Assert(req.Header["Date"], check.Not(check.Equals), "")
c.Assert(req.Form["prefix"], check.DeepEquals, []string{"photos/2006/"})
c.Assert(req.Form["delimiter"], check.DeepEquals, []string{"/"})
c.Assert(req.Form["marker"], check.DeepEquals, []string{"some-marker"})
c.Assert(req.Form["max-keys"], check.DeepEquals, []string{"1000"})
c.Assert(data.Name, check.Equals, "example-bucket")
c.Assert(data.Prefix, check.Equals, "photos/2006/")
c.Assert(data.Delimiter, check.Equals, "/")
c.Assert(data.Marker, check.Equals, "some-marker")
c.Assert(data.IsTruncated, check.Equals, false)
c.Assert(len(data.Contents), check.Equals, 0)
c.Assert(data.CommonPrefixes, check.DeepEquals, []string{"photos/2006/feb/", "photos/2006/jan/"})
}
func (s *S) TestExists(c *check.C) {
testServer.Response(200, nil, "")
b := s.s3.Bucket("bucket")
result, err := b.Exists("name")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "HEAD")
c.Assert(err, check.IsNil)
c.Assert(result, check.Equals, true)
}
func (s *S) TestExistsNotFound404(c *check.C) {
testServer.Response(404, nil, "")
b := s.s3.Bucket("bucket")
result, err := b.Exists("name")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "HEAD")
c.Assert(err, check.IsNil)
c.Assert(result, check.Equals, false)
}
func (s *S) TestExistsNotFound403(c *check.C) {
testServer.Response(403, nil, "")
b := s.s3.Bucket("bucket")
result, err := b.Exists("name")
req := testServer.WaitRequest()
c.Assert(req.Method, check.Equals, "HEAD")
c.Assert(err, check.IsNil)
c.Assert(result, check.Equals, false)
}
func (s *S) TestGetService(c *check.C) {
testServer.Response(200, nil, GetServiceDump)
expected := s3.GetServiceResp{
Owner: s3.Owner{
ID: "bcaf1ffd86f461ca5fb16fd081034f",
DisplayName: "webfile",
},
Buckets: []s3.BucketInfo{
s3.BucketInfo{
Name: "quotes",
CreationDate: "2006-02-03T16:45:09.000Z",
},
s3.BucketInfo{
Name: "samples",
CreationDate: "2006-02-03T16:41:58.000Z",
},
},
}
received, err := s.s3.GetService()
c.Assert(err, check.IsNil)
c.Assert(*received, check.DeepEquals, expected)
}
func (s *S) TestLocation(c *check.C) {
testServer.Response(200, nil, GetLocationUsStandard)
expectedUsStandard := "us-east-1"
bucketUsStandard := s.s3.Bucket("us-east-1")
resultUsStandard, err := bucketUsStandard.Location()
c.Assert(err, check.IsNil)
c.Assert(resultUsStandard, check.Equals, expectedUsStandard)
testServer.Response(200, nil, GetLocationUsWest1)
expectedUsWest1 := "us-west-1"
bucketUsWest1 := s.s3.Bucket("us-west-1")
resultUsWest1, err := bucketUsWest1.Location()
c.Assert(err, check.IsNil)
c.Assert(resultUsWest1, check.Equals, expectedUsWest1)
}

View file

@ -1,589 +0,0 @@
package s3_test
import (
"bytes"
"crypto/md5"
"fmt"
"github.com/AdRoll/goamz/aws"
"github.com/AdRoll/goamz/s3"
"github.com/AdRoll/goamz/testutil"
"gopkg.in/check.v1"
"io/ioutil"
"net"
"net/http"
"sort"
"strings"
"time"
)
// AmazonServer represents an Amazon S3 server.
type AmazonServer struct {
auth aws.Auth
}
func (s *AmazonServer) SetUp(c *check.C) {
auth, err := aws.EnvAuth()
if err != nil {
c.Fatal(err.Error())
}
s.auth = auth
}
var _ = check.Suite(&AmazonClientSuite{Region: aws.USEast})
var _ = check.Suite(&AmazonClientSuite{Region: aws.EUWest})
var _ = check.Suite(&AmazonDomainClientSuite{Region: aws.USEast})
// AmazonClientSuite tests the client against a live S3 server.
type AmazonClientSuite struct {
aws.Region
srv AmazonServer
ClientTests
}
func (s *AmazonClientSuite) SetUpSuite(c *check.C) {
if !testutil.Amazon {
c.Skip("live tests against AWS disabled (no -amazon)")
}
s.srv.SetUp(c)
s.s3 = s3.New(s.srv.auth, s.Region)
// In case tests were interrupted in the middle before.
s.ClientTests.Cleanup()
}
func (s *AmazonClientSuite) TearDownTest(c *check.C) {
s.ClientTests.Cleanup()
}
// AmazonDomainClientSuite tests the client against a live S3
// server using bucket names in the endpoint domain name rather
// than the request path.
type AmazonDomainClientSuite struct {
aws.Region
srv AmazonServer
ClientTests
}
func (s *AmazonDomainClientSuite) SetUpSuite(c *check.C) {
if !testutil.Amazon {
c.Skip("live tests against AWS disabled (no -amazon)")
}
s.srv.SetUp(c)
region := s.Region
region.S3BucketEndpoint = "https://${bucket}.s3.amazonaws.com"
s.s3 = s3.New(s.srv.auth, region)
s.ClientTests.Cleanup()
}
func (s *AmazonDomainClientSuite) TearDownTest(c *check.C) {
s.ClientTests.Cleanup()
}
// ClientTests defines integration tests designed to test the client.
// It is not used as a test suite in itself, but embedded within
// another type.
type ClientTests struct {
s3 *s3.S3
authIsBroken bool
}
func (s *ClientTests) Cleanup() {
killBucket(testBucket(s.s3))
}
func testBucket(s *s3.S3) *s3.Bucket {
// Watch out! If this function is corrupted and made to match with something
// people own, killBucket will happily remove *everything* inside the bucket.
key := s.Auth.AccessKey
if len(key) >= 8 {
key = s.Auth.AccessKey[:8]
}
return s.Bucket(fmt.Sprintf("goamz-%s-%s", s.Region.Name, key))
}
var attempts = aws.AttemptStrategy{
Min: 5,
Total: 20 * time.Second,
Delay: 100 * time.Millisecond,
}
func killBucket(b *s3.Bucket) {
var err error
for attempt := attempts.Start(); attempt.Next(); {
err = b.DelBucket()
if err == nil {
return
}
if _, ok := err.(*net.DNSError); ok {
return
}
e, ok := err.(*s3.Error)
if ok && e.Code == "NoSuchBucket" {
return
}
if ok && e.Code == "BucketNotEmpty" {
// Errors are ignored here. Just retry.
resp, err := b.List("", "", "", 1000)
if err == nil {
for _, key := range resp.Contents {
_ = b.Del(key.Key)
}
}
multis, _, _ := b.ListMulti("", "")
for _, m := range multis {
_ = m.Abort()
}
}
}
message := "cannot delete test bucket"
if err != nil {
message += ": " + err.Error()
}
panic(message)
}
func get(url string) ([]byte, error) {
for attempt := attempts.Start(); attempt.Next(); {
resp, err := http.Get(url)
if err != nil {
if attempt.HasNext() {
continue
}
return nil, err
}
data, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
if attempt.HasNext() {
continue
}
return nil, err
}
return data, err
}
panic("unreachable")
}
func (s *ClientTests) TestBasicFunctionality(c *check.C) {
b := testBucket(s.s3)
err := b.PutBucket(s3.PublicRead)
c.Assert(err, check.IsNil)
err = b.Put("name", []byte("yo!"), "text/plain", s3.PublicRead, s3.Options{})
c.Assert(err, check.IsNil)
defer b.Del("name")
data, err := b.Get("name")
c.Assert(err, check.IsNil)
c.Assert(string(data), check.Equals, "yo!")
data, err = get(b.URL("name"))
c.Assert(err, check.IsNil)
c.Assert(string(data), check.Equals, "yo!")
buf := bytes.NewBufferString("hey!")
err = b.PutReader("name2", buf, int64(buf.Len()), "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
defer b.Del("name2")
rc, err := b.GetReader("name2")
c.Assert(err, check.IsNil)
data, err = ioutil.ReadAll(rc)
c.Check(err, check.IsNil)
c.Check(string(data), check.Equals, "hey!")
rc.Close()
data, err = get(b.SignedURL("name2", time.Now().Add(time.Hour)))
c.Assert(err, check.IsNil)
c.Assert(string(data), check.Equals, "hey!")
if !s.authIsBroken {
data, err = get(b.SignedURL("name2", time.Now().Add(-time.Hour)))
c.Assert(err, check.IsNil)
c.Assert(string(data), check.Matches, "(?s).*AccessDenied.*")
}
err = b.DelBucket()
c.Assert(err, check.NotNil)
s3err, ok := err.(*s3.Error)
c.Assert(ok, check.Equals, true)
c.Assert(s3err.Code, check.Equals, "BucketNotEmpty")
c.Assert(s3err.BucketName, check.Equals, b.Name)
c.Assert(s3err.Message, check.Equals, "The bucket you tried to delete is not empty")
err = b.Del("name")
c.Assert(err, check.IsNil)
err = b.Del("name2")
c.Assert(err, check.IsNil)
err = b.DelBucket()
c.Assert(err, check.IsNil)
}
func (s *ClientTests) TestGetNotFound(c *check.C) {
b := s.s3.Bucket("goamz-" + s.s3.Auth.AccessKey)
data, err := b.Get("non-existent")
s3err, _ := err.(*s3.Error)
c.Assert(s3err, check.NotNil)
c.Assert(s3err.StatusCode, check.Equals, 404)
c.Assert(s3err.Code, check.Equals, "NoSuchBucket")
c.Assert(s3err.Message, check.Equals, "The specified bucket does not exist")
c.Assert(data, check.IsNil)
}
// Communicate with all endpoints to see if they are alive.
func (s *ClientTests) TestRegions(c *check.C) {
errs := make(chan error, len(aws.Regions))
for _, region := range aws.Regions {
go func(r aws.Region) {
s := s3.New(s.s3.Auth, r)
b := s.Bucket("goamz-" + s.Auth.AccessKey)
_, err := b.Get("non-existent")
errs <- err
}(region)
}
for _ = range aws.Regions {
err := <-errs
if err != nil {
s3_err, ok := err.(*s3.Error)
if ok {
c.Check(s3_err.Code, check.Matches, "NoSuchBucket")
} else if _, ok = err.(*net.DNSError); ok {
// Okay as well.
} else {
c.Errorf("Non-S3 error: %s", err)
}
} else {
c.Errorf("Test should have errored but it seems to have succeeded")
}
}
}
var objectNames = []string{
"index.html",
"index2.html",
"photos/2006/February/sample2.jpg",
"photos/2006/February/sample3.jpg",
"photos/2006/February/sample4.jpg",
"photos/2006/January/sample.jpg",
"test/bar",
"test/foo",
}
func keys(names ...string) []s3.Key {
ks := make([]s3.Key, len(names))
for i, name := range names {
ks[i].Key = name
}
return ks
}
// As the ListResp specifies all the parameters to the
// request too, we use it to specify request parameters
// and expected results. The Contents field is
// used only for the key names inside it.
var listTests = []s3.ListResp{
// normal list.
{
Contents: keys(objectNames...),
}, {
Marker: objectNames[0],
Contents: keys(objectNames[1:]...),
}, {
Marker: objectNames[0] + "a",
Contents: keys(objectNames[1:]...),
}, {
Marker: "z",
},
// limited results.
{
MaxKeys: 2,
Contents: keys(objectNames[0:2]...),
IsTruncated: true,
}, {
MaxKeys: 2,
Marker: objectNames[0],
Contents: keys(objectNames[1:3]...),
IsTruncated: true,
}, {
MaxKeys: 2,
Marker: objectNames[len(objectNames)-2],
Contents: keys(objectNames[len(objectNames)-1:]...),
},
// with delimiter
{
Delimiter: "/",
CommonPrefixes: []string{"photos/", "test/"},
Contents: keys("index.html", "index2.html"),
}, {
Delimiter: "/",
Prefix: "photos/2006/",
CommonPrefixes: []string{"photos/2006/February/", "photos/2006/January/"},
}, {
Delimiter: "/",
Prefix: "t",
CommonPrefixes: []string{"test/"},
}, {
Delimiter: "/",
MaxKeys: 1,
Contents: keys("index.html"),
IsTruncated: true,
}, {
Delimiter: "/",
MaxKeys: 1,
Marker: "index2.html",
CommonPrefixes: []string{"photos/"},
IsTruncated: true,
}, {
Delimiter: "/",
MaxKeys: 1,
Marker: "photos/",
CommonPrefixes: []string{"test/"},
IsTruncated: false,
}, {
Delimiter: "Feb",
CommonPrefixes: []string{"photos/2006/Feb"},
Contents: keys("index.html", "index2.html", "photos/2006/January/sample.jpg", "test/bar", "test/foo"),
},
}
func (s *ClientTests) TestDoublePutBucket(c *check.C) {
b := testBucket(s.s3)
err := b.PutBucket(s3.PublicRead)
c.Assert(err, check.IsNil)
err = b.PutBucket(s3.PublicRead)
if err != nil {
c.Assert(err, check.FitsTypeOf, new(s3.Error))
c.Assert(err.(*s3.Error).Code, check.Equals, "BucketAlreadyOwnedByYou")
}
}
func (s *ClientTests) TestBucketList(c *check.C) {
b := testBucket(s.s3)
err := b.PutBucket(s3.Private)
c.Assert(err, check.IsNil)
objData := make(map[string][]byte)
for i, path := range objectNames {
data := []byte(strings.Repeat("a", i))
err := b.Put(path, data, "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
defer b.Del(path)
objData[path] = data
}
for i, t := range listTests {
c.Logf("test %d", i)
resp, err := b.List(t.Prefix, t.Delimiter, t.Marker, t.MaxKeys)
c.Assert(err, check.IsNil)
c.Check(resp.Name, check.Equals, b.Name)
c.Check(resp.Delimiter, check.Equals, t.Delimiter)
c.Check(resp.IsTruncated, check.Equals, t.IsTruncated)
c.Check(resp.CommonPrefixes, check.DeepEquals, t.CommonPrefixes)
checkContents(c, resp.Contents, objData, t.Contents)
}
}
func etag(data []byte) string {
sum := md5.New()
sum.Write(data)
return fmt.Sprintf(`"%x"`, sum.Sum(nil))
}
func checkContents(c *check.C, contents []s3.Key, data map[string][]byte, expected []s3.Key) {
c.Assert(contents, check.HasLen, len(expected))
for i, k := range contents {
c.Check(k.Key, check.Equals, expected[i].Key)
// TODO mtime
c.Check(k.Size, check.Equals, int64(len(data[k.Key])))
c.Check(k.ETag, check.Equals, etag(data[k.Key]))
}
}
func (s *ClientTests) TestMultiInitPutList(c *check.C) {
b := testBucket(s.s3)
err := b.PutBucket(s3.Private)
c.Assert(err, check.IsNil)
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
c.Assert(multi.UploadId, check.Matches, ".+")
defer multi.Abort()
var sent []s3.Part
for i := 0; i < 5; i++ {
p, err := multi.PutPart(i+1, strings.NewReader(fmt.Sprintf("<part %d>", i+1)))
c.Assert(err, check.IsNil)
c.Assert(p.N, check.Equals, i+1)
c.Assert(p.Size, check.Equals, int64(8))
c.Assert(p.ETag, check.Matches, ".+")
sent = append(sent, p)
}
s3.SetListPartsMax(2)
parts, err := multi.ListParts()
c.Assert(err, check.IsNil)
c.Assert(parts, check.HasLen, len(sent))
for i := range parts {
c.Assert(parts[i].N, check.Equals, sent[i].N)
c.Assert(parts[i].Size, check.Equals, sent[i].Size)
c.Assert(parts[i].ETag, check.Equals, sent[i].ETag)
}
err = multi.Complete(parts)
s3err, failed := err.(*s3.Error)
c.Assert(failed, check.Equals, true)
c.Assert(s3err.Code, check.Equals, "EntityTooSmall")
err = multi.Abort()
c.Assert(err, check.IsNil)
_, err = multi.ListParts()
s3err, ok := err.(*s3.Error)
c.Assert(ok, check.Equals, true)
c.Assert(s3err.Code, check.Equals, "NoSuchUpload")
}
// This may take a minute or more due to the minimum size accepted S3
// on multipart upload parts.
func (s *ClientTests) TestMultiComplete(c *check.C) {
b := testBucket(s.s3)
err := b.PutBucket(s3.Private)
c.Assert(err, check.IsNil)
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
c.Assert(multi.UploadId, check.Matches, ".+")
defer multi.Abort()
// Minimum size S3 accepts for all but the last part is 5MB.
data1 := make([]byte, 5*1024*1024)
data2 := []byte("<part 2>")
part1, err := multi.PutPart(1, bytes.NewReader(data1))
c.Assert(err, check.IsNil)
part2, err := multi.PutPart(2, bytes.NewReader(data2))
c.Assert(err, check.IsNil)
// Purposefully reversed. The order requirement must be handled.
err = multi.Complete([]s3.Part{part2, part1})
c.Assert(err, check.IsNil)
data, err := b.Get("multi")
c.Assert(err, check.IsNil)
c.Assert(len(data), check.Equals, len(data1)+len(data2))
for i := range data1 {
if data[i] != data1[i] {
c.Fatalf("uploaded object at byte %d: want %d, got %d", data1[i], data[i])
}
}
c.Assert(string(data[len(data1):]), check.Equals, string(data2))
}
type multiList []*s3.Multi
func (l multiList) Len() int { return len(l) }
func (l multiList) Less(i, j int) bool { return l[i].Key < l[j].Key }
func (l multiList) Swap(i, j int) { l[i], l[j] = l[j], l[i] }
func (s *ClientTests) TestListMulti(c *check.C) {
b := testBucket(s.s3)
err := b.PutBucket(s3.Private)
c.Assert(err, check.IsNil)
// Ensure an empty state before testing its behavior.
multis, _, err := b.ListMulti("", "")
for _, m := range multis {
err := m.Abort()
c.Assert(err, check.IsNil)
}
keys := []string{
"a/multi2",
"a/multi3",
"b/multi4",
"multi1",
}
for _, key := range keys {
m, err := b.InitMulti(key, "", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
defer m.Abort()
}
// Amazon's implementation of the multiple-request listing for
// multipart uploads in progress seems broken in multiple ways.
// (next tokens are not provided, etc).
//s3.SetListMultiMax(2)
multis, prefixes, err := b.ListMulti("", "")
c.Assert(err, check.IsNil)
for attempt := attempts.Start(); attempt.Next() && len(multis) < len(keys); {
multis, prefixes, err = b.ListMulti("", "")
c.Assert(err, check.IsNil)
}
sort.Sort(multiList(multis))
c.Assert(prefixes, check.IsNil)
var gotKeys []string
for _, m := range multis {
gotKeys = append(gotKeys, m.Key)
}
c.Assert(gotKeys, check.DeepEquals, keys)
for _, m := range multis {
c.Assert(m.Bucket, check.Equals, b)
c.Assert(m.UploadId, check.Matches, ".+")
}
multis, prefixes, err = b.ListMulti("", "/")
for attempt := attempts.Start(); attempt.Next() && len(prefixes) < 2; {
multis, prefixes, err = b.ListMulti("", "")
c.Assert(err, check.IsNil)
}
c.Assert(err, check.IsNil)
c.Assert(prefixes, check.DeepEquals, []string{"a/", "b/"})
c.Assert(multis, check.HasLen, 1)
c.Assert(multis[0].Bucket, check.Equals, b)
c.Assert(multis[0].Key, check.Equals, "multi1")
c.Assert(multis[0].UploadId, check.Matches, ".+")
for attempt := attempts.Start(); attempt.Next() && len(multis) < 2; {
multis, prefixes, err = b.ListMulti("", "")
c.Assert(err, check.IsNil)
}
multis, prefixes, err = b.ListMulti("a/", "/")
c.Assert(err, check.IsNil)
c.Assert(prefixes, check.IsNil)
c.Assert(multis, check.HasLen, 2)
c.Assert(multis[0].Bucket, check.Equals, b)
c.Assert(multis[0].Key, check.Equals, "a/multi2")
c.Assert(multis[0].UploadId, check.Matches, ".+")
c.Assert(multis[1].Bucket, check.Equals, b)
c.Assert(multis[1].Key, check.Equals, "a/multi3")
c.Assert(multis[1].UploadId, check.Matches, ".+")
}
func (s *ClientTests) TestMultiPutAllZeroLength(c *check.C) {
b := testBucket(s.s3)
err := b.PutBucket(s3.Private)
c.Assert(err, check.IsNil)
multi, err := b.InitMulti("multi", "text/plain", s3.Private, s3.Options{})
c.Assert(err, check.IsNil)
defer multi.Abort()
// This tests an edge case. Amazon requires at least one
// part for multiprat uploads to work, even the part is empty.
parts, err := multi.PutAll(strings.NewReader(""), 5*1024*1024)
c.Assert(err, check.IsNil)
c.Assert(parts, check.HasLen, 1)
c.Assert(parts[0].Size, check.Equals, int64(0))
c.Assert(parts[0].ETag, check.Equals, `"d41d8cd98f00b204e9800998ecf8427e"`)
err = multi.Complete(parts)
c.Assert(err, check.IsNil)
}

View file

@ -1,79 +0,0 @@
package s3_test
import (
"github.com/AdRoll/goamz/aws"
"github.com/AdRoll/goamz/s3"
"github.com/AdRoll/goamz/s3/s3test"
"gopkg.in/check.v1"
)
type LocalServer struct {
auth aws.Auth
region aws.Region
srv *s3test.Server
config *s3test.Config
}
func (s *LocalServer) SetUp(c *check.C) {
srv, err := s3test.NewServer(s.config)
c.Assert(err, check.IsNil)
c.Assert(srv, check.NotNil)
s.srv = srv
s.region = aws.Region{
Name: "faux-region-1",
S3Endpoint: srv.URL(),
S3LocationConstraint: true, // s3test server requires a LocationConstraint
}
}
// LocalServerSuite defines tests that will run
// against the local s3test server. It includes
// selected tests from ClientTests;
// when the s3test functionality is sufficient, it should
// include all of them, and ClientTests can be simply embedded.
type LocalServerSuite struct {
srv LocalServer
clientTests ClientTests
}
var (
// run tests twice, once in us-east-1 mode, once not.
_ = check.Suite(&LocalServerSuite{})
_ = check.Suite(&LocalServerSuite{
srv: LocalServer{
config: &s3test.Config{
Send409Conflict: true,
},
},
})
)
func (s *LocalServerSuite) SetUpSuite(c *check.C) {
s.srv.SetUp(c)
s.clientTests.s3 = s3.New(s.srv.auth, s.srv.region)
// TODO Sadly the fake server ignores auth completely right now. :-(
s.clientTests.authIsBroken = true
s.clientTests.Cleanup()
}
func (s *LocalServerSuite) TearDownTest(c *check.C) {
s.clientTests.Cleanup()
}
func (s *LocalServerSuite) TestBasicFunctionality(c *check.C) {
s.clientTests.TestBasicFunctionality(c)
}
func (s *LocalServerSuite) TestGetNotFound(c *check.C) {
s.clientTests.TestGetNotFound(c)
}
func (s *LocalServerSuite) TestBucketList(c *check.C) {
s.clientTests.TestBucketList(c)
}
func (s *LocalServerSuite) TestDoublePutBucket(c *check.C) {
s.clientTests.TestDoublePutBucket(c)
}

View file

@ -1,870 +0,0 @@
package s3test
import (
"bytes"
"crypto/md5"
"encoding/base64"
"encoding/hex"
"encoding/xml"
"fmt"
"github.com/AdRoll/goamz/s3"
"io"
"io/ioutil"
"log"
"math/rand"
"net"
"net/http"
"net/url"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
)
const debug = false
type s3Error struct {
statusCode int
XMLName struct{} `xml:"Error"`
Code string
Message string
BucketName string
RequestId string
HostId string
}
type action struct {
srv *Server
w http.ResponseWriter
req *http.Request
reqId string
}
// Config controls the internal behaviour of the Server. A nil config is the default
// and behaves as if all configurations assume their default behaviour. Once passed
// to NewServer, the configuration must not be modified.
type Config struct {
// Send409Conflict controls how the Server will respond to calls to PUT on a
// previously existing bucket. The default is false, and corresponds to the
// us-east-1 s3 enpoint. Setting this value to true emulates the behaviour of
// all other regions.
// http://docs.amazonwebservices.com/AmazonS3/latest/API/ErrorResponses.html
Send409Conflict bool
// Address on which to listen. By default, a random port is assigned by the
// operating system and the server listens on localhost.
ListenAddress string
}
func (c *Config) send409Conflict() bool {
if c != nil {
return c.Send409Conflict
}
return false
}
// Server is a fake S3 server for testing purposes.
// All of the data for the server is kept in memory.
type Server struct {
url string
reqId int
listener net.Listener
mu sync.Mutex
buckets map[string]*bucket
config *Config
}
type bucket struct {
name string
acl s3.ACL
ctime time.Time
objects map[string]*object
multipartUploads map[string][]*multipartUploadPart
}
type object struct {
name string
mtime time.Time
meta http.Header // metadata to return with requests.
checksum []byte // also held as Content-MD5 in meta.
data []byte
}
type multipartUploadPart struct {
data []byte
etag string
lastModified time.Time
}
// A resource encapsulates the subject of an HTTP request.
// The resource referred to may or may not exist
// when the request is made.
type resource interface {
put(a *action) interface{}
get(a *action) interface{}
post(a *action) interface{}
delete(a *action) interface{}
}
func NewServer(config *Config) (*Server, error) {
listenAddress := "localhost:0"
if config != nil && config.ListenAddress != "" {
listenAddress = config.ListenAddress
}
l, err := net.Listen("tcp", listenAddress)
if err != nil {
return nil, fmt.Errorf("cannot listen on localhost: %v", err)
}
srv := &Server{
listener: l,
url: "http://" + l.Addr().String(),
buckets: make(map[string]*bucket),
config: config,
}
go http.Serve(l, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
srv.serveHTTP(w, req)
}))
return srv, nil
}
// Quit closes down the server.
func (srv *Server) Quit() {
srv.listener.Close()
}
// URL returns a URL for the server.
func (srv *Server) URL() string {
return srv.url
}
func fatalf(code int, codeStr string, errf string, a ...interface{}) {
panic(&s3Error{
statusCode: code,
Code: codeStr,
Message: fmt.Sprintf(errf, a...),
})
}
// serveHTTP serves the S3 protocol.
func (srv *Server) serveHTTP(w http.ResponseWriter, req *http.Request) {
// ignore error from ParseForm as it's usually spurious.
req.ParseForm()
srv.mu.Lock()
defer srv.mu.Unlock()
if debug {
log.Printf("s3test %q %q", req.Method, req.URL)
}
a := &action{
srv: srv,
w: w,
req: req,
reqId: fmt.Sprintf("%09X", srv.reqId),
}
srv.reqId++
var r resource
defer func() {
switch err := recover().(type) {
case *s3Error:
switch r := r.(type) {
case objectResource:
err.BucketName = r.bucket.name
case bucketResource:
err.BucketName = r.name
}
err.RequestId = a.reqId
// TODO HostId
w.Header().Set("Content-Type", `xml version="1.0" encoding="UTF-8"`)
w.WriteHeader(err.statusCode)
xmlMarshal(w, err)
case nil:
default:
panic(err)
}
}()
r = srv.resourceForURL(req.URL)
var resp interface{}
switch req.Method {
case "PUT":
resp = r.put(a)
case "GET", "HEAD":
resp = r.get(a)
case "DELETE":
resp = r.delete(a)
case "POST":
resp = r.post(a)
default:
fatalf(400, "MethodNotAllowed", "unknown http request method %q", req.Method)
}
if resp != nil && req.Method != "HEAD" {
xmlMarshal(w, resp)
}
}
// xmlMarshal is the same as xml.Marshal except that
// it panics on error. The marshalling should not fail,
// but we want to know if it does.
func xmlMarshal(w io.Writer, x interface{}) {
if err := xml.NewEncoder(w).Encode(x); err != nil {
panic(fmt.Errorf("error marshalling %#v: %v", x, err))
}
}
// In a fully implemented test server, each of these would have
// its own resource type.
var unimplementedBucketResourceNames = map[string]bool{
"acl": true,
"lifecycle": true,
"policy": true,
"location": true,
"logging": true,
"notification": true,
"versions": true,
"requestPayment": true,
"versioning": true,
"website": true,
"uploads": true,
}
var unimplementedObjectResourceNames = map[string]bool{
"acl": true,
"torrent": true,
}
var pathRegexp = regexp.MustCompile("/(([^/]+)(/(.*))?)?")
// resourceForURL returns a resource object for the given URL.
func (srv *Server) resourceForURL(u *url.URL) (r resource) {
m := pathRegexp.FindStringSubmatch(u.Path)
if m == nil {
fatalf(404, "InvalidURI", "Couldn't parse the specified URI")
}
bucketName := m[2]
objectName := m[4]
if bucketName == "" {
return nullResource{} // root
}
b := bucketResource{
name: bucketName,
bucket: srv.buckets[bucketName],
}
q := u.Query()
if objectName == "" {
for name := range q {
if unimplementedBucketResourceNames[name] {
return nullResource{}
}
}
return b
}
if b.bucket == nil {
fatalf(404, "NoSuchBucket", "The specified bucket does not exist")
}
objr := objectResource{
name: objectName,
version: q.Get("versionId"),
bucket: b.bucket,
}
for name := range q {
if unimplementedObjectResourceNames[name] {
return nullResource{}
}
}
if obj := objr.bucket.objects[objr.name]; obj != nil {
objr.object = obj
}
return objr
}
// nullResource has error stubs for all resource methods.
type nullResource struct{}
func notAllowed() interface{} {
fatalf(400, "MethodNotAllowed", "The specified method is not allowed against this resource")
return nil
}
func (nullResource) put(a *action) interface{} { return notAllowed() }
func (nullResource) get(a *action) interface{} { return notAllowed() }
func (nullResource) post(a *action) interface{} { return notAllowed() }
func (nullResource) delete(a *action) interface{} { return notAllowed() }
const timeFormat = "2006-01-02T15:04:05.000Z07:00"
type bucketResource struct {
name string
bucket *bucket // non-nil if the bucket already exists.
}
// GET on a bucket lists the objects in the bucket.
// http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGET.html
func (r bucketResource) get(a *action) interface{} {
if r.bucket == nil {
fatalf(404, "NoSuchBucket", "The specified bucket does not exist")
}
delimiter := a.req.Form.Get("delimiter")
marker := a.req.Form.Get("marker")
maxKeys := -1
if s := a.req.Form.Get("max-keys"); s != "" {
i, err := strconv.Atoi(s)
if err != nil || i < 0 {
fatalf(400, "invalid value for max-keys: %q", s)
}
maxKeys = i
}
prefix := a.req.Form.Get("prefix")
a.w.Header().Set("Content-Type", "application/xml")
if a.req.Method == "HEAD" {
return nil
}
var objs orderedObjects
// first get all matching objects and arrange them in alphabetical order.
for name, obj := range r.bucket.objects {
if strings.HasPrefix(name, prefix) {
objs = append(objs, obj)
}
}
sort.Sort(objs)
if maxKeys <= 0 {
maxKeys = 1000
}
resp := &s3.ListResp{
Name: r.bucket.name,
Prefix: prefix,
Delimiter: delimiter,
Marker: marker,
MaxKeys: maxKeys,
}
var prefixes []string
for _, obj := range objs {
if !strings.HasPrefix(obj.name, prefix) {
continue
}
name := obj.name
isPrefix := false
if delimiter != "" {
if i := strings.Index(obj.name[len(prefix):], delimiter); i >= 0 {
name = obj.name[:len(prefix)+i+len(delimiter)]
if prefixes != nil && prefixes[len(prefixes)-1] == name {
continue
}
isPrefix = true
}
}
if name <= marker {
continue
}
if len(resp.Contents)+len(prefixes) >= maxKeys {
resp.IsTruncated = true
break
}
if isPrefix {
prefixes = append(prefixes, name)
} else {
// Contents contains only keys not found in CommonPrefixes
resp.Contents = append(resp.Contents, obj.s3Key())
}
}
resp.CommonPrefixes = prefixes
return resp
}
// orderedObjects holds a slice of objects that can be sorted
// by name.
type orderedObjects []*object
func (s orderedObjects) Len() int {
return len(s)
}
func (s orderedObjects) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s orderedObjects) Less(i, j int) bool {
return s[i].name < s[j].name
}
func (obj *object) s3Key() s3.Key {
return s3.Key{
Key: obj.name,
LastModified: obj.mtime.Format(timeFormat),
Size: int64(len(obj.data)),
ETag: fmt.Sprintf(`"%x"`, obj.checksum),
// TODO StorageClass
// TODO Owner
}
}
// DELETE on a bucket deletes the bucket if it's not empty.
func (r bucketResource) delete(a *action) interface{} {
b := r.bucket
if b == nil {
fatalf(404, "NoSuchBucket", "The specified bucket does not exist")
}
if len(b.objects) > 0 {
fatalf(400, "BucketNotEmpty", "The bucket you tried to delete is not empty")
}
delete(a.srv.buckets, b.name)
return nil
}
// PUT on a bucket creates the bucket.
// http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUT.html
func (r bucketResource) put(a *action) interface{} {
var created bool
if r.bucket == nil {
if !validBucketName(r.name) {
fatalf(400, "InvalidBucketName", "The specified bucket is not valid")
}
if loc := locationConstraint(a); loc == "" {
fatalf(400, "InvalidRequets", "The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.")
}
// TODO validate acl
r.bucket = &bucket{
name: r.name,
// TODO default acl
objects: make(map[string]*object),
multipartUploads: make(map[string][]*multipartUploadPart),
}
a.srv.buckets[r.name] = r.bucket
created = true
}
if !created && a.srv.config.send409Conflict() {
fatalf(409, "BucketAlreadyOwnedByYou", "Your previous request to create the named bucket succeeded and you already own it.")
}
r.bucket.acl = s3.ACL(a.req.Header.Get("x-amz-acl"))
return nil
}
func (r bucketResource) post(a *action) interface{} {
if _, multiDel := a.req.URL.Query()["delete"]; multiDel {
return r.multiDel(a)
}
fatalf(400, "Method", "bucket operation not supported")
return nil
}
func (b bucketResource) multiDel(a *action) interface{} {
type multiDelRequestObject struct {
Key string
VersionId string
}
type multiDelRequest struct {
Quiet bool
Object []*multiDelRequestObject
}
type multiDelDelete struct {
XMLName struct{} `xml:"Deleted"`
Key string
}
type multiDelError struct {
XMLName struct{} `xml:"Error"`
Key string
Code string
Message string
}
type multiDelResult struct {
XMLName struct{} `xml:"DeleteResult"`
Deleted []*multiDelDelete
Error []*multiDelError
}
req := &multiDelRequest{}
if err := xml.NewDecoder(a.req.Body).Decode(req); err != nil {
fatalf(400, "InvalidRequest", err.Error())
}
res := &multiDelResult{
Deleted: []*multiDelDelete{},
Error: []*multiDelError{},
}
for _, o := range req.Object {
if _, exists := b.bucket.objects[o.Key]; exists {
delete(b.bucket.objects, o.Key)
res.Deleted = append(res.Deleted, &multiDelDelete{
Key: o.Key,
})
} else {
res.Error = append(res.Error, &multiDelError{
Key: o.Key,
Code: "AccessDenied",
Message: "Access Denied",
})
}
}
return res
}
// validBucketName returns whether name is a valid bucket name.
// Here are the rules, from:
// http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/BucketRestrictions.html
//
// Can contain lowercase letters, numbers, periods (.), underscores (_),
// and dashes (-). You can use uppercase letters for buckets only in the
// US Standard region.
//
// Must start with a number or letter
//
// Must be between 3 and 255 characters long
//
// There's one extra rule (Must not be formatted as an IP address (e.g., 192.168.5.4)
// but the real S3 server does not seem to check that rule, so we will not
// check it either.
//
func validBucketName(name string) bool {
if len(name) < 3 || len(name) > 255 {
return false
}
r := name[0]
if !(r >= '0' && r <= '9' || r >= 'a' && r <= 'z') {
return false
}
for _, r := range name {
switch {
case r >= '0' && r <= '9':
case r >= 'a' && r <= 'z':
case r == '_' || r == '-':
case r == '.':
default:
return false
}
}
return true
}
var responseParams = map[string]bool{
"content-type": true,
"content-language": true,
"expires": true,
"cache-control": true,
"content-disposition": true,
"content-encoding": true,
}
type objectResource struct {
name string
version string
bucket *bucket // always non-nil.
object *object // may be nil.
}
// GET on an object gets the contents of the object.
// http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html
func (objr objectResource) get(a *action) interface{} {
obj := objr.object
if obj == nil {
fatalf(404, "NoSuchKey", "The specified key does not exist.")
}
h := a.w.Header()
// add metadata
for name, d := range obj.meta {
h[name] = d
}
// override header values in response to request parameters.
for name, vals := range a.req.Form {
if strings.HasPrefix(name, "response-") {
name = name[len("response-"):]
if !responseParams[name] {
continue
}
h.Set(name, vals[0])
}
}
if r := a.req.Header.Get("Range"); r != "" {
fatalf(400, "NotImplemented", "range unimplemented")
}
// TODO Last-Modified-Since
// TODO If-Modified-Since
// TODO If-Unmodified-Since
// TODO If-Match
// TODO If-None-Match
// TODO Connection: close ??
// TODO x-amz-request-id
h.Set("Content-Length", fmt.Sprint(len(obj.data)))
h.Set("ETag", hex.EncodeToString(obj.checksum))
h.Set("Last-Modified", obj.mtime.Format(time.RFC1123))
if a.req.Method == "HEAD" {
return nil
}
// TODO avoid holding the lock when writing data.
_, err := a.w.Write(obj.data)
if err != nil {
// we can't do much except just log the fact.
log.Printf("error writing data: %v", err)
}
return nil
}
var metaHeaders = map[string]bool{
"Content-MD5": true,
"x-amz-acl": true,
"Content-Type": true,
"Content-Encoding": true,
"Content-Disposition": true,
}
// PUT on an object creates the object.
func (objr objectResource) put(a *action) interface{} {
// TODO Cache-Control header
// TODO Expires header
// TODO x-amz-server-side-encryption
// TODO x-amz-storage-class
uploadId := a.req.URL.Query().Get("uploadId")
// Check that the upload ID is valid if this is a multipart upload
if uploadId != "" {
if _, ok := objr.bucket.multipartUploads[uploadId]; !ok {
fatalf(404, "NoSuchUpload", "The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")
}
partNumberStr := a.req.URL.Query().Get("partNumber")
if partNumberStr == "" {
fatalf(400, "InvalidRequest", "Missing partNumber parameter")
}
partNumber, err := strconv.ParseUint(partNumberStr, 10, 32)
if err != nil {
fatalf(400, "InvalidRequest", "partNumber is not a number")
}
// Parts are 1-indexed for multipart uploads
if uint(partNumber)-1 != uint(len(objr.bucket.multipartUploads[uploadId])) {
fatalf(400, "InvalidRequest", "Invalid part number")
}
}
var expectHash []byte
if c := a.req.Header.Get("Content-MD5"); c != "" {
var err error
expectHash, err = base64.StdEncoding.DecodeString(c)
if err != nil || len(expectHash) != md5.Size {
fatalf(400, "InvalidDigest", "The Content-MD5 you specified was invalid")
}
}
sum := md5.New()
// TODO avoid holding lock while reading data.
data, err := ioutil.ReadAll(io.TeeReader(a.req.Body, sum))
if err != nil {
fatalf(400, "TODO", "read error")
}
gotHash := sum.Sum(nil)
if expectHash != nil && bytes.Compare(gotHash, expectHash) != 0 {
fatalf(400, "BadDigest", "The Content-MD5 you specified did not match what we received")
}
if a.req.ContentLength >= 0 && int64(len(data)) != a.req.ContentLength {
fatalf(400, "IncompleteBody", "You did not provide the number of bytes specified by the Content-Length HTTP header")
}
etag := fmt.Sprintf("\"%x\"", gotHash)
a.w.Header().Add("ETag", etag)
if uploadId == "" {
// For traditional uploads
// TODO is this correct, or should we erase all previous metadata?
obj := objr.object
if obj == nil {
obj = &object{
name: objr.name,
meta: make(http.Header),
}
}
// PUT request has been successful - save data and metadata
for key, values := range a.req.Header {
key = http.CanonicalHeaderKey(key)
if metaHeaders[key] || strings.HasPrefix(key, "X-Amz-Meta-") {
obj.meta[key] = values
}
}
obj.data = data
obj.checksum = gotHash
obj.mtime = time.Now()
objr.bucket.objects[objr.name] = obj
} else {
// For multipart commit
parts := objr.bucket.multipartUploads[uploadId]
part := &multipartUploadPart{
data,
etag,
time.Now(),
}
objr.bucket.multipartUploads[uploadId] = append(parts, part)
}
return nil
}
func (objr objectResource) delete(a *action) interface{} {
uploadId := a.req.URL.Query().Get("uploadId")
if uploadId == "" {
// Traditional object delete
delete(objr.bucket.objects, objr.name)
} else {
// Multipart commit abort
_, ok := objr.bucket.multipartUploads[uploadId]
if !ok {
fatalf(404, "NoSuchUpload", "The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")
}
delete(objr.bucket.multipartUploads, uploadId)
}
return nil
}
func (objr objectResource) post(a *action) interface{} {
// Check if we're initializing a multipart upload
if _, ok := a.req.URL.Query()["uploads"]; ok {
type multipartInitResponse struct {
XMLName struct{} `xml:"InitiateMultipartUploadResult"`
Bucket string
Key string
UploadId string
}
uploadId := strconv.FormatInt(rand.Int63(), 16)
objr.bucket.multipartUploads[uploadId] = []*multipartUploadPart{}
return &multipartInitResponse{
Bucket: objr.bucket.name,
Key: objr.name,
UploadId: uploadId,
}
}
// Check if we're completing a multipart upload
if uploadId := a.req.URL.Query().Get("uploadId"); uploadId != "" {
type multipartCompleteRequestPart struct {
XMLName struct{} `xml:"Part"`
PartNumber uint
ETag string
}
type multipartCompleteRequest struct {
XMLName struct{} `xml:"CompleteMultipartUpload"`
Part []multipartCompleteRequestPart
}
type multipartCompleteResponse struct {
XMLName struct{} `xml:"CompleteMultipartUploadResult"`
Location string
Bucket string
Key string
ETag string
}
parts, ok := objr.bucket.multipartUploads[uploadId]
if !ok {
fatalf(404, "NoSuchUpload", "The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.")
}
req := &multipartCompleteRequest{}
if err := xml.NewDecoder(a.req.Body).Decode(req); err != nil {
fatalf(400, "InvalidRequest", err.Error())
}
if len(req.Part) != len(parts) {
fatalf(400, "InvalidRequest", fmt.Sprintf("Number of parts does not match: expected %d, received %d", len(parts), len(req.Part)))
}
sum := md5.New()
data := &bytes.Buffer{}
w := io.MultiWriter(sum, data)
for i, p := range parts {
reqPart := req.Part[i]
if reqPart.PartNumber != uint(1+i) {
fatalf(400, "InvalidRequest", "Bad part number")
}
if reqPart.ETag != p.etag {
fatalf(400, "InvalidRequest", fmt.Sprintf("Invalid etag for part %d", reqPart.PartNumber))
}
w.Write(p.data)
}
delete(objr.bucket.multipartUploads, uploadId)
obj := objr.object
if obj == nil {
obj = &object{
name: objr.name,
meta: make(http.Header),
}
}
obj.data = data.Bytes()
obj.checksum = sum.Sum(nil)
obj.mtime = time.Now()
objr.bucket.objects[objr.name] = obj
objectLocation := fmt.Sprintf("http://%s/%s/%s", a.srv.listener.Addr().String(), objr.bucket.name, objr.name)
return &multipartCompleteResponse{
Location: objectLocation,
Bucket: objr.bucket.name,
Key: objr.name,
ETag: uploadId,
}
}
fatalf(400, "MethodNotAllowed", "The specified method is not allowed against this resource")
return nil
}
type CreateBucketConfiguration struct {
LocationConstraint string
}
// locationConstraint parses the <CreateBucketConfiguration /> request body (if present).
// If there is no body, an empty string will be returned.
func locationConstraint(a *action) string {
var body bytes.Buffer
if _, err := io.Copy(&body, a.req.Body); err != nil {
fatalf(400, "InvalidRequest", err.Error())
}
if body.Len() == 0 {
return ""
}
var loc CreateBucketConfiguration
if err := xml.NewDecoder(&body).Decode(&loc); err != nil {
fatalf(400, "InvalidRequest", err.Error())
}
return loc.LocationConstraint
}

View file

@ -1,120 +0,0 @@
package s3
import (
"crypto/hmac"
"crypto/sha1"
"encoding/base64"
"github.com/AdRoll/goamz/aws"
"log"
"sort"
"strings"
)
var b64 = base64.StdEncoding
// ----------------------------------------------------------------------------
// S3 signing (http://goo.gl/G1LrK)
var s3ParamsToSign = map[string]bool{
"acl": true,
"location": true,
"logging": true,
"notification": true,
"partNumber": true,
"policy": true,
"requestPayment": true,
"torrent": true,
"uploadId": true,
"uploads": true,
"versionId": true,
"versioning": true,
"versions": true,
"response-content-type": true,
"response-content-language": true,
"response-expires": true,
"response-cache-control": true,
"response-content-disposition": true,
"response-content-encoding": true,
"website": true,
"delete": true,
}
func sign(auth aws.Auth, method, canonicalPath string, params, headers map[string][]string) {
var md5, ctype, date, xamz string
var xamzDate bool
var keys, sarray []string
xheaders := make(map[string]string)
for k, v := range headers {
k = strings.ToLower(k)
switch k {
case "content-md5":
md5 = v[0]
case "content-type":
ctype = v[0]
case "date":
if !xamzDate {
date = v[0]
}
default:
if strings.HasPrefix(k, "x-amz-") {
keys = append(keys, k)
xheaders[k] = strings.Join(v, ",")
if k == "x-amz-date" {
xamzDate = true
date = ""
}
}
}
}
if len(keys) > 0 {
sort.StringSlice(keys).Sort()
for i := range keys {
key := keys[i]
value := xheaders[key]
sarray = append(sarray, key+":"+value)
}
xamz = strings.Join(sarray, "\n") + "\n"
}
expires := false
if v, ok := params["Expires"]; ok {
// Query string request authentication alternative.
expires = true
date = v[0]
params["AWSAccessKeyId"] = []string{auth.AccessKey}
}
sarray = sarray[0:0]
for k, v := range params {
if s3ParamsToSign[k] {
for _, vi := range v {
if vi == "" {
sarray = append(sarray, k)
} else {
// "When signing you do not encode these values."
sarray = append(sarray, k+"="+vi)
}
}
}
}
if len(sarray) > 0 {
sort.StringSlice(sarray).Sort()
canonicalPath = canonicalPath + "?" + strings.Join(sarray, "&")
}
payload := method + "\n" + md5 + "\n" + ctype + "\n" + date + "\n" + xamz + canonicalPath
hash := hmac.New(sha1.New, []byte(auth.SecretKey))
hash.Write([]byte(payload))
signature := make([]byte, b64.EncodedLen(hash.Size()))
b64.Encode(signature, hash.Sum(nil))
if expires {
params["Signature"] = []string{string(signature)}
} else {
headers["Authorization"] = []string{"AWS " + auth.AccessKey + ":" + string(signature)}
}
if debug {
log.Printf("Signature payload: %q", payload)
log.Printf("Signature: %q", signature)
}
}

View file

@ -1,148 +0,0 @@
package s3_test
import (
"github.com/AdRoll/goamz/aws"
"github.com/AdRoll/goamz/s3"
"gopkg.in/check.v1"
)
// S3 ReST authentication docs: http://goo.gl/G1LrK
var testAuth = aws.Auth{AccessKey: "0PN5J17HBGZHT7JJ3X82", SecretKey: "uV3F3YluFJax1cknvbcGwgjvx4QpvB+leU8dUj2o"}
func (s *S) TestSignExampleObjectGet(c *check.C) {
method := "GET"
path := "/johnsmith/photos/puppy.jpg"
headers := map[string][]string{
"Host": {"johnsmith.s3.amazonaws.com"},
"Date": {"Tue, 27 Mar 2007 19:36:42 +0000"},
}
s3.Sign(testAuth, method, path, nil, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:xXjDGYUmKxnwqr5KXNPGldn5LbA="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleObjectPut(c *check.C) {
method := "PUT"
path := "/johnsmith/photos/puppy.jpg"
headers := map[string][]string{
"Host": {"johnsmith.s3.amazonaws.com"},
"Date": {"Tue, 27 Mar 2007 21:15:45 +0000"},
"Content-Type": {"image/jpeg"},
"Content-Length": {"94328"},
}
s3.Sign(testAuth, method, path, nil, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:hcicpDDvL9SsO6AkvxqmIWkmOuQ="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleList(c *check.C) {
method := "GET"
path := "/johnsmith/"
params := map[string][]string{
"prefix": {"photos"},
"max-keys": {"50"},
"marker": {"puppy"},
}
headers := map[string][]string{
"Host": {"johnsmith.s3.amazonaws.com"},
"Date": {"Tue, 27 Mar 2007 19:42:41 +0000"},
"User-Agent": {"Mozilla/5.0"},
}
s3.Sign(testAuth, method, path, params, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:jsRt/rhG+Vtp88HrYL706QhE4w4="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleFetch(c *check.C) {
method := "GET"
path := "/johnsmith/"
params := map[string][]string{
"acl": {""},
}
headers := map[string][]string{
"Host": {"johnsmith.s3.amazonaws.com"},
"Date": {"Tue, 27 Mar 2007 19:44:46 +0000"},
}
s3.Sign(testAuth, method, path, params, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:thdUi9VAkzhkniLj96JIrOPGi0g="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleDelete(c *check.C) {
method := "DELETE"
path := "/johnsmith/photos/puppy.jpg"
params := map[string][]string{}
headers := map[string][]string{
"Host": {"s3.amazonaws.com"},
"Date": {"Tue, 27 Mar 2007 21:20:27 +0000"},
"User-Agent": {"dotnet"},
"x-amz-date": {"Tue, 27 Mar 2007 21:20:26 +0000"},
}
s3.Sign(testAuth, method, path, params, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:k3nL7gH3+PadhTEVn5Ip83xlYzk="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleUpload(c *check.C) {
method := "PUT"
path := "/static.johnsmith.net/db-backup.dat.gz"
params := map[string][]string{}
headers := map[string][]string{
"Host": {"static.johnsmith.net:8080"},
"Date": {"Tue, 27 Mar 2007 21:06:08 +0000"},
"User-Agent": {"curl/7.15.5"},
"x-amz-acl": {"public-read"},
"content-type": {"application/x-download"},
"Content-MD5": {"4gJE4saaMU4BqNR0kLY+lw=="},
"X-Amz-Meta-ReviewedBy": {"joe@johnsmith.net,jane@johnsmith.net"},
"X-Amz-Meta-FileChecksum": {"0x02661779"},
"X-Amz-Meta-ChecksumAlgorithm": {"crc32"},
"Content-Disposition": {"attachment; filename=database.dat"},
"Content-Encoding": {"gzip"},
"Content-Length": {"5913339"},
}
s3.Sign(testAuth, method, path, params, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:C0FlOtU8Ylb9KDTpZqYkZPX91iI="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleListAllMyBuckets(c *check.C) {
method := "GET"
path := "/"
headers := map[string][]string{
"Host": {"s3.amazonaws.com"},
"Date": {"Wed, 28 Mar 2007 01:29:59 +0000"},
}
s3.Sign(testAuth, method, path, nil, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:Db+gepJSUbZKwpx1FR0DLtEYoZA="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleUnicodeKeys(c *check.C) {
method := "GET"
path := "/dictionary/fran%C3%A7ais/pr%c3%a9f%c3%a8re"
headers := map[string][]string{
"Host": {"s3.amazonaws.com"},
"Date": {"Wed, 28 Mar 2007 01:49:49 +0000"},
}
s3.Sign(testAuth, method, path, nil, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:dxhSBHoI6eVSPcXJqEghlUzZMnY="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}
func (s *S) TestSignExampleCustomSSE(c *check.C) {
method := "GET"
path := "/secret/config"
params := map[string][]string{}
headers := map[string][]string{
"Host": {"secret.johnsmith.net:8080"},
"Date": {"Tue, 27 Mar 2007 21:06:08 +0000"},
"x-amz-server-side-encryption-customer-key": {"MWJhakVna1dQT1B0SDFMeGtVVnRQRTFGaU1ldFJrU0I="},
"x-amz-server-side-encryption-customer-key-MD5": {"glIqxpqQ4a9aoK/iLttKzQ=="},
"x-amz-server-side-encryption-customer-algorithm": {"AES256"},
}
s3.Sign(testAuth, method, path, params, headers)
expected := "AWS 0PN5J17HBGZHT7JJ3X82:Xq6PWmIo0aOWq+LDjCEiCGgbmHE="
c.Assert(headers["Authorization"], check.DeepEquals, []string{expected})
}

View file

@ -1,19 +0,0 @@
sudo: false
language: go
before_script:
- go get -u golang.org/x/tools/cmd/vet
- go get -u github.com/golang/lint/golint
go: tip
script:
- test -z "$(gofmt -s -l -w management | tee /dev/stderr)"
- test -z "$(gofmt -s -l -w storage | tee /dev/stderr)"
- go build -v ./...
- go test -v ./storage/... -check.v
- test -z "$(golint ./storage/... | tee /dev/stderr)"
- go vet ./storage/...
- go test -v ./management/...
- test -z "$(golint ./management/... | grep -v 'should have comment' | grep -v 'stutters' | tee /dev/stderr)"
- go vet ./management/...

View file

@ -1,88 +0,0 @@
# Microsoft Azure SDK for Go
This project provides various Go packages to perform operations
on Microsoft Azure REST APIs.
[![GoDoc](https://godoc.org/github.com/Azure/azure-sdk-for-go?status.svg)](https://godoc.org/github.com/Azure/azure-sdk-for-go) [![Build Status](https://travis-ci.org/Azure/azure-sdk-for-go.svg?branch=master)](https://travis-ci.org/Azure/azure-sdk-for-go)
See list of implemented API clients [here](http://godoc.org/github.com/Azure/azure-sdk-for-go).
> **NOTE:** This repository is under heavy ongoing development and
is likely to break over time. We currently do not have any releases
yet. If you are planning to use the repository, please consider vendoring
the packages in your project and update them when a stable tag is out.
# Installation
go get -d github.com/Azure/azure-sdk-for-go/management
# Usage
Read Godoc of the repository at: http://godoc.org/github.com/Azure/azure-sdk-for-go/
The client currently supports authentication to the Service Management
API with certificates or Azure `.publishSettings` file. You can
download the `.publishSettings` file for your subscriptions
[here](https://manage.windowsazure.com/publishsettings).
### Example: Creating a Linux Virtual Machine
```go
package main
import (
"encoding/base64"
"fmt"
"github.com/Azure/azure-sdk-for-go/management"
"github.com/Azure/azure-sdk-for-go/management/hostedservice"
"github.com/Azure/azure-sdk-for-go/management/virtualmachine"
"github.com/Azure/azure-sdk-for-go/management/vmutils"
)
func main() {
dnsName := "test-vm-from-go"
storageAccount := "mystorageaccount"
location := "West US"
vmSize := "Small"
vmImage := "b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140724-en-us-30GB"
userName := "testuser"
userPassword := "Test123"
client, err := management.ClientFromPublishSettingsFile("path/to/downloaded.publishsettings", "")
if err != nil {
panic(err)
}
// create hosted service
if err := hostedservice.NewClient(client).CreateHostedService(hostedservice.CreateHostedServiceParameters{
ServiceName: dnsName,
Location: location,
Label: base64.StdEncoding.EncodeToString([]byte(dnsName))}); err != nil {
panic(err)
}
// create virtual machine
role := vmutils.NewVMConfiguration(dnsName, vmSize)
vmutils.ConfigureDeploymentFromPlatformImage(
&role,
vmImage,
fmt.Sprintf("http://%s.blob.core.windows.net/sdktest/%s.vhd", storageAccount, dnsName),
"")
vmutils.ConfigureForLinux(&role, dnsName, userName, userPassword)
vmutils.ConfigureWithPublicSSH(&role)
operationID, err := virtualmachine.NewClient(client).
CreateDeployment(role, dnsName, virtualmachine.CreateDeploymentOptions{})
if err != nil {
panic(err)
}
if err := client.WaitForOperation(operationID, nil); err != nil {
panic(err)
}
}
```
# License
This project is published under [Apache 2.0 License](LICENSE).

View file

@ -1,858 +0,0 @@
package storage
import (
"bytes"
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
)
// BlobStorageClient contains operations for Microsoft Azure Blob Storage
// Service.
type BlobStorageClient struct {
client Client
}
// A Container is an entry in ContainerListResponse.
type Container struct {
Name string `xml:"Name"`
Properties ContainerProperties `xml:"Properties"`
// TODO (ahmetalpbalkan) Metadata
}
// ContainerProperties contains various properties of a container returned from
// various endpoints like ListContainers.
type ContainerProperties struct {
LastModified string `xml:"Last-Modified"`
Etag string `xml:"Etag"`
LeaseStatus string `xml:"LeaseStatus"`
LeaseState string `xml:"LeaseState"`
LeaseDuration string `xml:"LeaseDuration"`
// TODO (ahmetalpbalkan) remaining fields
}
// ContainerListResponse contains the response fields from
// ListContainers call.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179352.aspx
type ContainerListResponse struct {
XMLName xml.Name `xml:"EnumerationResults"`
Xmlns string `xml:"xmlns,attr"`
Prefix string `xml:"Prefix"`
Marker string `xml:"Marker"`
NextMarker string `xml:"NextMarker"`
MaxResults int64 `xml:"MaxResults"`
Containers []Container `xml:"Containers>Container"`
}
// A Blob is an entry in BlobListResponse.
type Blob struct {
Name string `xml:"Name"`
Properties BlobProperties `xml:"Properties"`
// TODO (ahmetalpbalkan) Metadata
}
// BlobProperties contains various properties of a blob
// returned in various endpoints like ListBlobs or GetBlobProperties.
type BlobProperties struct {
LastModified string `xml:"Last-Modified"`
Etag string `xml:"Etag"`
ContentMD5 string `xml:"Content-MD5"`
ContentLength int64 `xml:"Content-Length"`
ContentType string `xml:"Content-Type"`
ContentEncoding string `xml:"Content-Encoding"`
BlobType BlobType `xml:"x-ms-blob-blob-type"`
SequenceNumber int64 `xml:"x-ms-blob-sequence-number"`
CopyID string `xml:"CopyId"`
CopyStatus string `xml:"CopyStatus"`
CopySource string `xml:"CopySource"`
CopyProgress string `xml:"CopyProgress"`
CopyCompletionTime string `xml:"CopyCompletionTime"`
CopyStatusDescription string `xml:"CopyStatusDescription"`
}
// BlobListResponse contains the response fields from ListBlobs call.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd135734.aspx
type BlobListResponse struct {
XMLName xml.Name `xml:"EnumerationResults"`
Xmlns string `xml:"xmlns,attr"`
Prefix string `xml:"Prefix"`
Marker string `xml:"Marker"`
NextMarker string `xml:"NextMarker"`
MaxResults int64 `xml:"MaxResults"`
Blobs []Blob `xml:"Blobs>Blob"`
}
// ListContainersParameters defines the set of customizable parameters to make a
// List Containers call.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179352.aspx
type ListContainersParameters struct {
Prefix string
Marker string
Include string
MaxResults uint
Timeout uint
}
func (p ListContainersParameters) getParameters() url.Values {
out := url.Values{}
if p.Prefix != "" {
out.Set("prefix", p.Prefix)
}
if p.Marker != "" {
out.Set("marker", p.Marker)
}
if p.Include != "" {
out.Set("include", p.Include)
}
if p.MaxResults != 0 {
out.Set("maxresults", fmt.Sprintf("%v", p.MaxResults))
}
if p.Timeout != 0 {
out.Set("timeout", fmt.Sprintf("%v", p.Timeout))
}
return out
}
// ListBlobsParameters defines the set of customizable
// parameters to make a List Blobs call.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd135734.aspx
type ListBlobsParameters struct {
Prefix string
Delimiter string
Marker string
Include string
MaxResults uint
Timeout uint
}
func (p ListBlobsParameters) getParameters() url.Values {
out := url.Values{}
if p.Prefix != "" {
out.Set("prefix", p.Prefix)
}
if p.Delimiter != "" {
out.Set("delimiter", p.Delimiter)
}
if p.Marker != "" {
out.Set("marker", p.Marker)
}
if p.Include != "" {
out.Set("include", p.Include)
}
if p.MaxResults != 0 {
out.Set("maxresults", fmt.Sprintf("%v", p.MaxResults))
}
if p.Timeout != 0 {
out.Set("timeout", fmt.Sprintf("%v", p.Timeout))
}
return out
}
// BlobType defines the type of the Azure Blob.
type BlobType string
// Types of page blobs
const (
BlobTypeBlock BlobType = "BlockBlob"
BlobTypePage BlobType = "PageBlob"
)
// PageWriteType defines the type updates that are going to be
// done on the page blob.
type PageWriteType string
// Types of operations on page blobs
const (
PageWriteTypeUpdate PageWriteType = "update"
PageWriteTypeClear PageWriteType = "clear"
)
const (
blobCopyStatusPending = "pending"
blobCopyStatusSuccess = "success"
blobCopyStatusAborted = "aborted"
blobCopyStatusFailed = "failed"
)
// BlockListType is used to filter out types of blocks in a Get Blocks List call
// for a block blob.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179400.aspx for all
// block types.
type BlockListType string
// Filters for listing blocks in block blobs
const (
BlockListTypeAll BlockListType = "all"
BlockListTypeCommitted BlockListType = "committed"
BlockListTypeUncommitted BlockListType = "uncommitted"
)
// ContainerAccessType defines the access level to the container from a public
// request.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179468.aspx and "x-ms-
// blob-public-access" header.
type ContainerAccessType string
// Access options for containers
const (
ContainerAccessTypePrivate ContainerAccessType = ""
ContainerAccessTypeBlob ContainerAccessType = "blob"
ContainerAccessTypeContainer ContainerAccessType = "container"
)
// Maximum sizes (per REST API) for various concepts
const (
MaxBlobBlockSize = 4 * 1024 * 1024
MaxBlobPageSize = 4 * 1024 * 1024
)
// BlockStatus defines states a block for a block blob can
// be in.
type BlockStatus string
// List of statuses that can be used to refer to a block in a block list
const (
BlockStatusUncommitted BlockStatus = "Uncommitted"
BlockStatusCommitted BlockStatus = "Committed"
BlockStatusLatest BlockStatus = "Latest"
)
// Block is used to create Block entities for Put Block List
// call.
type Block struct {
ID string
Status BlockStatus
}
// BlockListResponse contains the response fields from Get Block List call.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179400.aspx
type BlockListResponse struct {
XMLName xml.Name `xml:"BlockList"`
CommittedBlocks []BlockResponse `xml:"CommittedBlocks>Block"`
UncommittedBlocks []BlockResponse `xml:"UncommittedBlocks>Block"`
}
// BlockResponse contains the block information returned
// in the GetBlockListCall.
type BlockResponse struct {
Name string `xml:"Name"`
Size int64 `xml:"Size"`
}
// GetPageRangesResponse contains the reponse fields from
// Get Page Ranges call.
//
// See https://msdn.microsoft.com/en-us/library/azure/ee691973.aspx
type GetPageRangesResponse struct {
XMLName xml.Name `xml:"PageList"`
PageList []PageRange `xml:"PageRange"`
}
// PageRange contains information about a page of a page blob from
// Get Pages Range call.
//
// See https://msdn.microsoft.com/en-us/library/azure/ee691973.aspx
type PageRange struct {
Start int64 `xml:"Start"`
End int64 `xml:"End"`
}
var (
errBlobCopyAborted = errors.New("storage: blob copy is aborted")
errBlobCopyIDMismatch = errors.New("storage: blob copy id is a mismatch")
)
// ListContainers returns the list of containers in a storage account along with
// pagination token and other response details.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179352.aspx
func (b BlobStorageClient) ListContainers(params ListContainersParameters) (ContainerListResponse, error) {
q := mergeParams(params.getParameters(), url.Values{"comp": {"list"}})
uri := b.client.getEndpoint(blobServiceName, "", q)
headers := b.client.getStandardHeaders()
var out ContainerListResponse
resp, err := b.client.exec("GET", uri, headers, nil)
if err != nil {
return out, err
}
defer resp.body.Close()
err = xmlUnmarshal(resp.body, &out)
return out, err
}
// CreateContainer creates a blob container within the storage account
// with given name and access level. Returns error if container already exists.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179468.aspx
func (b BlobStorageClient) CreateContainer(name string, access ContainerAccessType) error {
resp, err := b.createContainer(name, access)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// CreateContainerIfNotExists creates a blob container if it does not exist. Returns
// true if container is newly created or false if container already exists.
func (b BlobStorageClient) CreateContainerIfNotExists(name string, access ContainerAccessType) (bool, error) {
resp, err := b.createContainer(name, access)
if resp != nil {
defer resp.body.Close()
if resp.statusCode == http.StatusCreated || resp.statusCode == http.StatusConflict {
return resp.statusCode == http.StatusCreated, nil
}
}
return false, err
}
func (b BlobStorageClient) createContainer(name string, access ContainerAccessType) (*storageResponse, error) {
verb := "PUT"
uri := b.client.getEndpoint(blobServiceName, pathForContainer(name), url.Values{"restype": {"container"}})
headers := b.client.getStandardHeaders()
headers["Content-Length"] = "0"
if access != "" {
headers["x-ms-blob-public-access"] = string(access)
}
return b.client.exec(verb, uri, headers, nil)
}
// ContainerExists returns true if a container with given name exists
// on the storage account, otherwise returns false.
func (b BlobStorageClient) ContainerExists(name string) (bool, error) {
verb := "HEAD"
uri := b.client.getEndpoint(blobServiceName, pathForContainer(name), url.Values{"restype": {"container"}})
headers := b.client.getStandardHeaders()
resp, err := b.client.exec(verb, uri, headers, nil)
if resp != nil {
defer resp.body.Close()
if resp.statusCode == http.StatusOK || resp.statusCode == http.StatusNotFound {
return resp.statusCode == http.StatusOK, nil
}
}
return false, err
}
// DeleteContainer deletes the container with given name on the storage
// account. If the container does not exist returns error.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179408.aspx
func (b BlobStorageClient) DeleteContainer(name string) error {
resp, err := b.deleteContainer(name)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusAccepted})
}
// DeleteContainerIfExists deletes the container with given name on the storage
// account if it exists. Returns true if container is deleted with this call, or
// false if the container did not exist at the time of the Delete Container
// operation.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179408.aspx
func (b BlobStorageClient) DeleteContainerIfExists(name string) (bool, error) {
resp, err := b.deleteContainer(name)
if resp != nil {
defer resp.body.Close()
if resp.statusCode == http.StatusAccepted || resp.statusCode == http.StatusNotFound {
return resp.statusCode == http.StatusAccepted, nil
}
}
return false, err
}
func (b BlobStorageClient) deleteContainer(name string) (*storageResponse, error) {
verb := "DELETE"
uri := b.client.getEndpoint(blobServiceName, pathForContainer(name), url.Values{"restype": {"container"}})
headers := b.client.getStandardHeaders()
return b.client.exec(verb, uri, headers, nil)
}
// ListBlobs returns an object that contains list of blobs in the container,
// pagination token and other information in the response of List Blobs call.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd135734.aspx
func (b BlobStorageClient) ListBlobs(container string, params ListBlobsParameters) (BlobListResponse, error) {
q := mergeParams(params.getParameters(), url.Values{
"restype": {"container"},
"comp": {"list"}})
uri := b.client.getEndpoint(blobServiceName, pathForContainer(container), q)
headers := b.client.getStandardHeaders()
var out BlobListResponse
resp, err := b.client.exec("GET", uri, headers, nil)
if err != nil {
return out, err
}
defer resp.body.Close()
err = xmlUnmarshal(resp.body, &out)
return out, err
}
// BlobExists returns true if a blob with given name exists on the specified
// container of the storage account.
func (b BlobStorageClient) BlobExists(container, name string) (bool, error) {
verb := "HEAD"
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{})
headers := b.client.getStandardHeaders()
resp, err := b.client.exec(verb, uri, headers, nil)
if resp != nil {
defer resp.body.Close()
if resp.statusCode == http.StatusOK || resp.statusCode == http.StatusNotFound {
return resp.statusCode == http.StatusOK, nil
}
}
return false, err
}
// GetBlobURL gets the canonical URL to the blob with the specified name in the
// specified container. This method does not create a publicly accessible URL if
// the blob or container is private and this method does not check if the blob
// exists.
func (b BlobStorageClient) GetBlobURL(container, name string) string {
if container == "" {
container = "$root"
}
return b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{})
}
// GetBlob returns a stream to read the blob. Caller must call Close() the
// reader to close on the underlying connection.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179440.aspx
func (b BlobStorageClient) GetBlob(container, name string) (io.ReadCloser, error) {
resp, err := b.getBlobRange(container, name, "")
if err != nil {
return nil, err
}
if err := checkRespCode(resp.statusCode, []int{http.StatusOK}); err != nil {
return nil, err
}
return resp.body, nil
}
// GetBlobRange reads the specified range of a blob to a stream. The bytesRange
// string must be in a format like "0-", "10-100" as defined in HTTP 1.1 spec.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179440.aspx
func (b BlobStorageClient) GetBlobRange(container, name, bytesRange string) (io.ReadCloser, error) {
resp, err := b.getBlobRange(container, name, bytesRange)
if err != nil {
return nil, err
}
if err := checkRespCode(resp.statusCode, []int{http.StatusPartialContent}); err != nil {
return nil, err
}
return resp.body, nil
}
func (b BlobStorageClient) getBlobRange(container, name, bytesRange string) (*storageResponse, error) {
verb := "GET"
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{})
headers := b.client.getStandardHeaders()
if bytesRange != "" {
headers["Range"] = fmt.Sprintf("bytes=%s", bytesRange)
}
resp, err := b.client.exec(verb, uri, headers, nil)
if err != nil {
return nil, err
}
return resp, err
}
// GetBlobProperties provides various information about the specified
// blob. See https://msdn.microsoft.com/en-us/library/azure/dd179394.aspx
func (b BlobStorageClient) GetBlobProperties(container, name string) (*BlobProperties, error) {
verb := "HEAD"
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{})
headers := b.client.getStandardHeaders()
resp, err := b.client.exec(verb, uri, headers, nil)
if err != nil {
return nil, err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusOK}); err != nil {
return nil, err
}
var contentLength int64
contentLengthStr := resp.headers.Get("Content-Length")
if contentLengthStr != "" {
contentLength, err = strconv.ParseInt(contentLengthStr, 0, 64)
if err != nil {
return nil, err
}
}
var sequenceNum int64
sequenceNumStr := resp.headers.Get("x-ms-blob-sequence-number")
if sequenceNumStr != "" {
sequenceNum, err = strconv.ParseInt(sequenceNumStr, 0, 64)
if err != nil {
return nil, err
}
}
return &BlobProperties{
LastModified: resp.headers.Get("Last-Modified"),
Etag: resp.headers.Get("Etag"),
ContentMD5: resp.headers.Get("Content-MD5"),
ContentLength: contentLength,
ContentEncoding: resp.headers.Get("Content-Encoding"),
SequenceNumber: sequenceNum,
CopyCompletionTime: resp.headers.Get("x-ms-copy-completion-time"),
CopyStatusDescription: resp.headers.Get("x-ms-copy-status-description"),
CopyID: resp.headers.Get("x-ms-copy-id"),
CopyProgress: resp.headers.Get("x-ms-copy-progress"),
CopySource: resp.headers.Get("x-ms-copy-source"),
CopyStatus: resp.headers.Get("x-ms-copy-status"),
BlobType: BlobType(resp.headers.Get("x-ms-blob-type")),
}, nil
}
// CreateBlockBlob initializes an empty block blob with no blocks.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179451.aspx
func (b BlobStorageClient) CreateBlockBlob(container, name string) error {
path := fmt.Sprintf("%s/%s", container, name)
uri := b.client.getEndpoint(blobServiceName, path, url.Values{})
headers := b.client.getStandardHeaders()
headers["x-ms-blob-type"] = string(BlobTypeBlock)
headers["Content-Length"] = fmt.Sprintf("%v", 0)
resp, err := b.client.exec("PUT", uri, headers, nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// PutBlock saves the given data chunk to the specified block blob with
// given ID.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd135726.aspx
func (b BlobStorageClient) PutBlock(container, name, blockID string, chunk []byte) error {
return b.PutBlockWithLength(container, name, blockID, uint64(len(chunk)), bytes.NewReader(chunk))
}
// PutBlockWithLength saves the given data stream of exactly specified size to
// the block blob with given ID. It is an alternative to PutBlocks where data
// comes as stream but the length is known in advance.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd135726.aspx
func (b BlobStorageClient) PutBlockWithLength(container, name, blockID string, size uint64, blob io.Reader) error {
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{"comp": {"block"}, "blockid": {blockID}})
headers := b.client.getStandardHeaders()
headers["x-ms-blob-type"] = string(BlobTypeBlock)
headers["Content-Length"] = fmt.Sprintf("%v", size)
resp, err := b.client.exec("PUT", uri, headers, blob)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// PutBlockList saves list of blocks to the specified block blob.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179467.aspx
func (b BlobStorageClient) PutBlockList(container, name string, blocks []Block) error {
blockListXML := prepareBlockListRequest(blocks)
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{"comp": {"blocklist"}})
headers := b.client.getStandardHeaders()
headers["Content-Length"] = fmt.Sprintf("%v", len(blockListXML))
resp, err := b.client.exec("PUT", uri, headers, strings.NewReader(blockListXML))
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// GetBlockList retrieves list of blocks in the specified block blob.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179400.aspx
func (b BlobStorageClient) GetBlockList(container, name string, blockType BlockListType) (BlockListResponse, error) {
params := url.Values{"comp": {"blocklist"}, "blocklisttype": {string(blockType)}}
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), params)
headers := b.client.getStandardHeaders()
var out BlockListResponse
resp, err := b.client.exec("GET", uri, headers, nil)
if err != nil {
return out, err
}
defer resp.body.Close()
err = xmlUnmarshal(resp.body, &out)
return out, err
}
// PutPageBlob initializes an empty page blob with specified name and maximum
// size in bytes (size must be aligned to a 512-byte boundary). A page blob must
// be created using this method before writing pages.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179451.aspx
func (b BlobStorageClient) PutPageBlob(container, name string, size int64) error {
path := fmt.Sprintf("%s/%s", container, name)
uri := b.client.getEndpoint(blobServiceName, path, url.Values{})
headers := b.client.getStandardHeaders()
headers["x-ms-blob-type"] = string(BlobTypePage)
headers["x-ms-blob-content-length"] = fmt.Sprintf("%v", size)
headers["Content-Length"] = fmt.Sprintf("%v", 0)
resp, err := b.client.exec("PUT", uri, headers, nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// PutPage writes a range of pages to a page blob or clears the given range.
// In case of 'clear' writes, given chunk is discarded. Ranges must be aligned
// with 512-byte boundaries and chunk must be of size multiplies by 512.
//
// See https://msdn.microsoft.com/en-us/library/ee691975.aspx
func (b BlobStorageClient) PutPage(container, name string, startByte, endByte int64, writeType PageWriteType, chunk []byte) error {
path := fmt.Sprintf("%s/%s", container, name)
uri := b.client.getEndpoint(blobServiceName, path, url.Values{"comp": {"page"}})
headers := b.client.getStandardHeaders()
headers["x-ms-blob-type"] = string(BlobTypePage)
headers["x-ms-page-write"] = string(writeType)
headers["x-ms-range"] = fmt.Sprintf("bytes=%v-%v", startByte, endByte)
var contentLength int64
var data io.Reader
if writeType == PageWriteTypeClear {
contentLength = 0
data = bytes.NewReader([]byte{})
} else {
contentLength = int64(len(chunk))
data = bytes.NewReader(chunk)
}
headers["Content-Length"] = fmt.Sprintf("%v", contentLength)
resp, err := b.client.exec("PUT", uri, headers, data)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// GetPageRanges returns the list of valid page ranges for a page blob.
//
// See https://msdn.microsoft.com/en-us/library/azure/ee691973.aspx
func (b BlobStorageClient) GetPageRanges(container, name string) (GetPageRangesResponse, error) {
path := fmt.Sprintf("%s/%s", container, name)
uri := b.client.getEndpoint(blobServiceName, path, url.Values{"comp": {"pagelist"}})
headers := b.client.getStandardHeaders()
var out GetPageRangesResponse
resp, err := b.client.exec("GET", uri, headers, nil)
if err != nil {
return out, err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusOK}); err != nil {
return out, err
}
err = xmlUnmarshal(resp.body, &out)
return out, err
}
// CopyBlob starts a blob copy operation and waits for the operation to
// complete. sourceBlob parameter must be a canonical URL to the blob (can be
// obtained using GetBlobURL method.) There is no SLA on blob copy and therefore
// this helper method works faster on smaller files.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd894037.aspx
func (b BlobStorageClient) CopyBlob(container, name, sourceBlob string) error {
copyID, err := b.startBlobCopy(container, name, sourceBlob)
if err != nil {
return err
}
return b.waitForBlobCopy(container, name, copyID)
}
func (b BlobStorageClient) startBlobCopy(container, name, sourceBlob string) (string, error) {
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{})
headers := b.client.getStandardHeaders()
headers["Content-Length"] = "0"
headers["x-ms-copy-source"] = sourceBlob
resp, err := b.client.exec("PUT", uri, headers, nil)
if err != nil {
return "", err
}
defer resp.body.Close()
if err := checkRespCode(resp.statusCode, []int{http.StatusAccepted, http.StatusCreated}); err != nil {
return "", err
}
copyID := resp.headers.Get("x-ms-copy-id")
if copyID == "" {
return "", errors.New("Got empty copy id header")
}
return copyID, nil
}
func (b BlobStorageClient) waitForBlobCopy(container, name, copyID string) error {
for {
props, err := b.GetBlobProperties(container, name)
if err != nil {
return err
}
if props.CopyID != copyID {
return errBlobCopyIDMismatch
}
switch props.CopyStatus {
case blobCopyStatusSuccess:
return nil
case blobCopyStatusPending:
continue
case blobCopyStatusAborted:
return errBlobCopyAborted
case blobCopyStatusFailed:
return fmt.Errorf("storage: blob copy failed. Id=%s Description=%s", props.CopyID, props.CopyStatusDescription)
default:
return fmt.Errorf("storage: unhandled blob copy status: '%s'", props.CopyStatus)
}
}
}
// DeleteBlob deletes the given blob from the specified container.
// If the blob does not exists at the time of the Delete Blob operation, it
// returns error. See https://msdn.microsoft.com/en-us/library/azure/dd179413.aspx
func (b BlobStorageClient) DeleteBlob(container, name string) error {
resp, err := b.deleteBlob(container, name)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusAccepted})
}
// DeleteBlobIfExists deletes the given blob from the specified container If the
// blob is deleted with this call, returns true. Otherwise returns false.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179413.aspx
func (b BlobStorageClient) DeleteBlobIfExists(container, name string) (bool, error) {
resp, err := b.deleteBlob(container, name)
if resp != nil && (resp.statusCode == http.StatusAccepted || resp.statusCode == http.StatusNotFound) {
return resp.statusCode == http.StatusAccepted, nil
}
defer resp.body.Close()
return false, err
}
func (b BlobStorageClient) deleteBlob(container, name string) (*storageResponse, error) {
verb := "DELETE"
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{})
headers := b.client.getStandardHeaders()
return b.client.exec(verb, uri, headers, nil)
}
// helper method to construct the path to a container given its name
func pathForContainer(name string) string {
return fmt.Sprintf("/%s", name)
}
// helper method to construct the path to a blob given its container and blob
// name
func pathForBlob(container, name string) string {
return fmt.Sprintf("/%s/%s", container, name)
}
// GetBlobSASURI creates an URL to the specified blob which contains the Shared
// Access Signature with specified permissions and expiration time.
//
// See https://msdn.microsoft.com/en-us/library/azure/ee395415.aspx
func (b BlobStorageClient) GetBlobSASURI(container, name string, expiry time.Time, permissions string) (string, error) {
var (
signedPermissions = permissions
blobURL = b.GetBlobURL(container, name)
)
canonicalizedResource, err := b.client.buildCanonicalizedResource(blobURL)
if err != nil {
return "", err
}
signedExpiry := expiry.Format(time.RFC3339)
signedResource := "b"
stringToSign, err := blobSASStringToSign(b.client.apiVersion, canonicalizedResource, signedExpiry, signedPermissions)
if err != nil {
return "", err
}
sig := b.client.computeHmac256(stringToSign)
sasParams := url.Values{
"sv": {b.client.apiVersion},
"se": {signedExpiry},
"sr": {signedResource},
"sp": {signedPermissions},
"sig": {sig},
}
sasURL, err := url.Parse(blobURL)
if err != nil {
return "", err
}
sasURL.RawQuery = sasParams.Encode()
return sasURL.String(), nil
}
func blobSASStringToSign(signedVersion, canonicalizedResource, signedExpiry, signedPermissions string) (string, error) {
var signedStart, signedIdentifier, rscc, rscd, rsce, rscl, rsct string
// reference: http://msdn.microsoft.com/en-us/library/azure/dn140255.aspx
if signedVersion >= "2013-08-15" {
return fmt.Sprintf("%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s", signedPermissions, signedStart, signedExpiry, canonicalizedResource, signedIdentifier, signedVersion, rscc, rscd, rsce, rscl, rsct), nil
}
return "", errors.New("storage: not implemented SAS for versions earlier than 2013-08-15")
}

View file

@ -1,625 +0,0 @@
package storage
import (
"bytes"
"crypto/rand"
"encoding/base64"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"sort"
"sync"
"testing"
"time"
chk "gopkg.in/check.v1"
)
type StorageBlobSuite struct{}
var _ = chk.Suite(&StorageBlobSuite{})
const testContainerPrefix = "zzzztest-"
func getBlobClient(c *chk.C) BlobStorageClient {
return getBasicClient(c).GetBlobService()
}
func (s *StorageBlobSuite) Test_pathForContainer(c *chk.C) {
c.Assert(pathForContainer("foo"), chk.Equals, "/foo")
}
func (s *StorageBlobSuite) Test_pathForBlob(c *chk.C) {
c.Assert(pathForBlob("foo", "blob"), chk.Equals, "/foo/blob")
}
func (s *StorageBlobSuite) Test_blobSASStringToSign(c *chk.C) {
_, err := blobSASStringToSign("2012-02-12", "CS", "SE", "SP")
c.Assert(err, chk.NotNil) // not implemented SAS for versions earlier than 2013-08-15
out, err := blobSASStringToSign("2013-08-15", "CS", "SE", "SP")
c.Assert(err, chk.IsNil)
c.Assert(out, chk.Equals, "SP\n\nSE\nCS\n\n2013-08-15\n\n\n\n\n")
}
func (s *StorageBlobSuite) TestGetBlobSASURI(c *chk.C) {
api, err := NewClient("foo", "YmFy", DefaultBaseURL, "2013-08-15", true)
c.Assert(err, chk.IsNil)
cli := api.GetBlobService()
expiry := time.Time{}
expectedParts := url.URL{
Scheme: "https",
Host: "foo.blob.core.windows.net",
Path: "container/name",
RawQuery: url.Values{
"sv": {"2013-08-15"},
"sig": {"/OXG7rWh08jYwtU03GzJM0DHZtidRGpC6g69rSGm3I0="},
"sr": {"b"},
"sp": {"r"},
"se": {"0001-01-01T00:00:00Z"},
}.Encode()}
u, err := cli.GetBlobSASURI("container", "name", expiry, "r")
c.Assert(err, chk.IsNil)
sasParts, err := url.Parse(u)
c.Assert(err, chk.IsNil)
c.Assert(expectedParts.String(), chk.Equals, sasParts.String())
c.Assert(expectedParts.Query(), chk.DeepEquals, sasParts.Query())
}
func (s *StorageBlobSuite) TestBlobSASURICorrectness(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
blob := randString(20)
body := []byte(randString(100))
expiry := time.Now().UTC().Add(time.Hour)
permissions := "r"
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.DeleteContainer(cnt)
c.Assert(cli.putSingleBlockBlob(cnt, blob, body), chk.IsNil)
sasURI, err := cli.GetBlobSASURI(cnt, blob, expiry, permissions)
c.Assert(err, chk.IsNil)
resp, err := http.Get(sasURI)
c.Assert(err, chk.IsNil)
blobResp, err := ioutil.ReadAll(resp.Body)
defer resp.Body.Close()
c.Assert(err, chk.IsNil)
c.Assert(resp.StatusCode, chk.Equals, http.StatusOK)
c.Assert(len(blobResp), chk.Equals, len(body))
}
func (s *StorageBlobSuite) TestListContainersPagination(c *chk.C) {
cli := getBlobClient(c)
c.Assert(deleteTestContainers(cli), chk.IsNil)
const n = 5
const pageSize = 2
// Create test containers
created := []string{}
for i := 0; i < n; i++ {
name := randContainer()
c.Assert(cli.CreateContainer(name, ContainerAccessTypePrivate), chk.IsNil)
created = append(created, name)
}
sort.Strings(created)
// Defer test container deletions
defer func() {
var wg sync.WaitGroup
for _, cnt := range created {
wg.Add(1)
go func(name string) {
c.Assert(cli.DeleteContainer(name), chk.IsNil)
wg.Done()
}(cnt)
}
wg.Wait()
}()
// Paginate results
seen := []string{}
marker := ""
for {
resp, err := cli.ListContainers(ListContainersParameters{
Prefix: testContainerPrefix,
MaxResults: pageSize,
Marker: marker})
c.Assert(err, chk.IsNil)
containers := resp.Containers
if len(containers) > pageSize {
c.Fatalf("Got a bigger page. Expected: %d, got: %d", pageSize, len(containers))
}
for _, c := range containers {
seen = append(seen, c.Name)
}
marker = resp.NextMarker
if marker == "" || len(containers) == 0 {
break
}
}
c.Assert(seen, chk.DeepEquals, created)
}
func (s *StorageBlobSuite) TestContainerExists(c *chk.C) {
cnt := randContainer()
cli := getBlobClient(c)
ok, err := cli.ContainerExists(cnt)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, false)
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypeBlob), chk.IsNil)
defer cli.DeleteContainer(cnt)
ok, err = cli.ContainerExists(cnt)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, true)
}
func (s *StorageBlobSuite) TestCreateDeleteContainer(c *chk.C) {
cnt := randContainer()
cli := getBlobClient(c)
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
c.Assert(cli.DeleteContainer(cnt), chk.IsNil)
}
func (s *StorageBlobSuite) TestCreateContainerIfNotExists(c *chk.C) {
cnt := randContainer()
cli := getBlobClient(c)
// First create
ok, err := cli.CreateContainerIfNotExists(cnt, ContainerAccessTypePrivate)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, true)
// Second create, should not give errors
ok, err = cli.CreateContainerIfNotExists(cnt, ContainerAccessTypePrivate)
c.Assert(err, chk.IsNil)
defer cli.DeleteContainer(cnt)
c.Assert(ok, chk.Equals, false)
}
func (s *StorageBlobSuite) TestDeleteContainerIfExists(c *chk.C) {
cnt := randContainer()
cli := getBlobClient(c)
// Nonexisting container
c.Assert(cli.DeleteContainer(cnt), chk.NotNil)
ok, err := cli.DeleteContainerIfExists(cnt)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, false)
// Existing container
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
ok, err = cli.DeleteContainerIfExists(cnt)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, true)
}
func (s *StorageBlobSuite) TestBlobExists(c *chk.C) {
cnt := randContainer()
blob := randString(20)
cli := getBlobClient(c)
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypeBlob), chk.IsNil)
defer cli.DeleteContainer(cnt)
c.Assert(cli.putSingleBlockBlob(cnt, blob, []byte("Hello!")), chk.IsNil)
defer cli.DeleteBlob(cnt, blob)
ok, err := cli.BlobExists(cnt, blob+".foo")
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, false)
ok, err = cli.BlobExists(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, true)
}
func (s *StorageBlobSuite) TestGetBlobURL(c *chk.C) {
api, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
cli := api.GetBlobService()
c.Assert(cli.GetBlobURL("c", "nested/blob"), chk.Equals, "https://foo.blob.core.windows.net/c/nested/blob")
c.Assert(cli.GetBlobURL("", "blob"), chk.Equals, "https://foo.blob.core.windows.net/$root/blob")
c.Assert(cli.GetBlobURL("", "nested/blob"), chk.Equals, "https://foo.blob.core.windows.net/$root/nested/blob")
}
func (s *StorageBlobSuite) TestBlobCopy(c *chk.C) {
if testing.Short() {
c.Skip("skipping blob copy in short mode, no SLA on async operation")
}
cli := getBlobClient(c)
cnt := randContainer()
src := randString(20)
dst := randString(20)
body := []byte(randString(1024))
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
c.Assert(cli.putSingleBlockBlob(cnt, src, body), chk.IsNil)
defer cli.DeleteBlob(cnt, src)
c.Assert(cli.CopyBlob(cnt, dst, cli.GetBlobURL(cnt, src)), chk.IsNil)
defer cli.DeleteBlob(cnt, dst)
blobBody, err := cli.GetBlob(cnt, dst)
c.Assert(err, chk.IsNil)
b, err := ioutil.ReadAll(blobBody)
defer blobBody.Close()
c.Assert(err, chk.IsNil)
c.Assert(b, chk.DeepEquals, body)
}
func (s *StorageBlobSuite) TestDeleteBlobIfExists(c *chk.C) {
cnt := randContainer()
blob := randString(20)
cli := getBlobClient(c)
c.Assert(cli.DeleteBlob(cnt, blob), chk.NotNil)
ok, err := cli.DeleteBlobIfExists(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, false)
}
func (s *StorageBlobSuite) TestGetBlobProperties(c *chk.C) {
cnt := randContainer()
blob := randString(20)
contents := randString(64)
cli := getBlobClient(c)
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.DeleteContainer(cnt)
// Nonexisting blob
_, err := cli.GetBlobProperties(cnt, blob)
c.Assert(err, chk.NotNil)
// Put the blob
c.Assert(cli.putSingleBlockBlob(cnt, blob, []byte(contents)), chk.IsNil)
// Get blob properties
props, err := cli.GetBlobProperties(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(props.ContentLength, chk.Equals, int64(len(contents)))
c.Assert(props.BlobType, chk.Equals, BlobTypeBlock)
}
func (s *StorageBlobSuite) TestListBlobsPagination(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.DeleteContainer(cnt)
blobs := []string{}
const n = 5
const pageSize = 2
for i := 0; i < n; i++ {
name := randString(20)
c.Assert(cli.putSingleBlockBlob(cnt, name, []byte("Hello, world!")), chk.IsNil)
blobs = append(blobs, name)
}
sort.Strings(blobs)
// Paginate
seen := []string{}
marker := ""
for {
resp, err := cli.ListBlobs(cnt, ListBlobsParameters{
MaxResults: pageSize,
Marker: marker})
c.Assert(err, chk.IsNil)
for _, v := range resp.Blobs {
seen = append(seen, v.Name)
}
marker = resp.NextMarker
if marker == "" || len(resp.Blobs) == 0 {
break
}
}
// Compare
c.Assert(seen, chk.DeepEquals, blobs)
}
func (s *StorageBlobSuite) TestPutEmptyBlockBlob(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
c.Assert(cli.putSingleBlockBlob(cnt, blob, []byte{}), chk.IsNil)
props, err := cli.GetBlobProperties(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(props.ContentLength, chk.Not(chk.Equals), 0)
}
func (s *StorageBlobSuite) TestGetBlobRange(c *chk.C) {
cnt := randContainer()
blob := randString(20)
body := "0123456789"
cli := getBlobClient(c)
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypeBlob), chk.IsNil)
defer cli.DeleteContainer(cnt)
c.Assert(cli.putSingleBlockBlob(cnt, blob, []byte(body)), chk.IsNil)
defer cli.DeleteBlob(cnt, blob)
// Read 1-3
for _, r := range []struct {
rangeStr string
expected string
}{
{"0-", body},
{"1-3", body[1 : 3+1]},
{"3-", body[3:]},
} {
resp, err := cli.GetBlobRange(cnt, blob, r.rangeStr)
c.Assert(err, chk.IsNil)
blobBody, err := ioutil.ReadAll(resp)
c.Assert(err, chk.IsNil)
str := string(blobBody)
c.Assert(str, chk.Equals, r.expected)
}
}
func (s *StorageBlobSuite) TestPutBlock(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
chunk := []byte(randString(1024))
blockID := base64.StdEncoding.EncodeToString([]byte("foo"))
c.Assert(cli.PutBlock(cnt, blob, blockID, chunk), chk.IsNil)
}
func (s *StorageBlobSuite) TestGetBlockList_PutBlockList(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
chunk := []byte(randString(1024))
blockID := base64.StdEncoding.EncodeToString([]byte("foo"))
// Put one block
c.Assert(cli.PutBlock(cnt, blob, blockID, chunk), chk.IsNil)
defer cli.deleteBlob(cnt, blob)
// Get committed blocks
committed, err := cli.GetBlockList(cnt, blob, BlockListTypeCommitted)
c.Assert(err, chk.IsNil)
if len(committed.CommittedBlocks) > 0 {
c.Fatal("There are committed blocks")
}
// Get uncommitted blocks
uncommitted, err := cli.GetBlockList(cnt, blob, BlockListTypeUncommitted)
c.Assert(err, chk.IsNil)
c.Assert(len(uncommitted.UncommittedBlocks), chk.Equals, 1)
// Commit block list
c.Assert(cli.PutBlockList(cnt, blob, []Block{{blockID, BlockStatusUncommitted}}), chk.IsNil)
// Get all blocks
all, err := cli.GetBlockList(cnt, blob, BlockListTypeAll)
c.Assert(err, chk.IsNil)
c.Assert(len(all.CommittedBlocks), chk.Equals, 1)
c.Assert(len(all.UncommittedBlocks), chk.Equals, 0)
// Verify the block
thatBlock := all.CommittedBlocks[0]
c.Assert(thatBlock.Name, chk.Equals, blockID)
c.Assert(thatBlock.Size, chk.Equals, int64(len(chunk)))
}
func (s *StorageBlobSuite) TestCreateBlockBlob(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
c.Assert(cli.CreateBlockBlob(cnt, blob), chk.IsNil)
// Verify
blocks, err := cli.GetBlockList(cnt, blob, BlockListTypeAll)
c.Assert(err, chk.IsNil)
c.Assert(len(blocks.CommittedBlocks), chk.Equals, 0)
c.Assert(len(blocks.UncommittedBlocks), chk.Equals, 0)
}
func (s *StorageBlobSuite) TestPutPageBlob(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
size := int64(10 * 1024 * 1024)
c.Assert(cli.PutPageBlob(cnt, blob, size), chk.IsNil)
// Verify
props, err := cli.GetBlobProperties(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(props.ContentLength, chk.Equals, size)
c.Assert(props.BlobType, chk.Equals, BlobTypePage)
}
func (s *StorageBlobSuite) TestPutPagesUpdate(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
size := int64(10 * 1024 * 1024) // larger than we'll use
c.Assert(cli.PutPageBlob(cnt, blob, size), chk.IsNil)
chunk1 := []byte(randString(1024))
chunk2 := []byte(randString(512))
// Append chunks
c.Assert(cli.PutPage(cnt, blob, 0, int64(len(chunk1)-1), PageWriteTypeUpdate, chunk1), chk.IsNil)
c.Assert(cli.PutPage(cnt, blob, int64(len(chunk1)), int64(len(chunk1)+len(chunk2)-1), PageWriteTypeUpdate, chunk2), chk.IsNil)
// Verify contents
out, err := cli.GetBlobRange(cnt, blob, fmt.Sprintf("%v-%v", 0, len(chunk1)+len(chunk2)-1))
c.Assert(err, chk.IsNil)
defer out.Close()
blobContents, err := ioutil.ReadAll(out)
c.Assert(err, chk.IsNil)
c.Assert(blobContents, chk.DeepEquals, append(chunk1, chunk2...))
out.Close()
// Overwrite first half of chunk1
chunk0 := []byte(randString(512))
c.Assert(cli.PutPage(cnt, blob, 0, int64(len(chunk0)-1), PageWriteTypeUpdate, chunk0), chk.IsNil)
// Verify contents
out, err = cli.GetBlobRange(cnt, blob, fmt.Sprintf("%v-%v", 0, len(chunk1)+len(chunk2)-1))
c.Assert(err, chk.IsNil)
defer out.Close()
blobContents, err = ioutil.ReadAll(out)
c.Assert(err, chk.IsNil)
c.Assert(blobContents, chk.DeepEquals, append(append(chunk0, chunk1[512:]...), chunk2...))
}
func (s *StorageBlobSuite) TestPutPagesClear(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
size := int64(10 * 1024 * 1024) // larger than we'll use
c.Assert(cli.PutPageBlob(cnt, blob, size), chk.IsNil)
// Put 0-2047
chunk := []byte(randString(2048))
c.Assert(cli.PutPage(cnt, blob, 0, 2047, PageWriteTypeUpdate, chunk), chk.IsNil)
// Clear 512-1023
c.Assert(cli.PutPage(cnt, blob, 512, 1023, PageWriteTypeClear, nil), chk.IsNil)
// Verify contents
out, err := cli.GetBlobRange(cnt, blob, "0-2047")
c.Assert(err, chk.IsNil)
contents, err := ioutil.ReadAll(out)
c.Assert(err, chk.IsNil)
defer out.Close()
c.Assert(contents, chk.DeepEquals, append(append(chunk[:512], make([]byte, 512)...), chunk[1024:]...))
}
func (s *StorageBlobSuite) TestGetPageRanges(c *chk.C) {
cli := getBlobClient(c)
cnt := randContainer()
c.Assert(cli.CreateContainer(cnt, ContainerAccessTypePrivate), chk.IsNil)
defer cli.deleteContainer(cnt)
blob := randString(20)
size := int64(10 * 1024 * 1024) // larger than we'll use
c.Assert(cli.PutPageBlob(cnt, blob, size), chk.IsNil)
// Get page ranges on empty blob
out, err := cli.GetPageRanges(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(len(out.PageList), chk.Equals, 0)
// Add 0-512 page
c.Assert(cli.PutPage(cnt, blob, 0, 511, PageWriteTypeUpdate, []byte(randString(512))), chk.IsNil)
out, err = cli.GetPageRanges(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(len(out.PageList), chk.Equals, 1)
// Add 1024-2048
c.Assert(cli.PutPage(cnt, blob, 1024, 2047, PageWriteTypeUpdate, []byte(randString(1024))), chk.IsNil)
out, err = cli.GetPageRanges(cnt, blob)
c.Assert(err, chk.IsNil)
c.Assert(len(out.PageList), chk.Equals, 2)
}
func deleteTestContainers(cli BlobStorageClient) error {
for {
resp, err := cli.ListContainers(ListContainersParameters{Prefix: testContainerPrefix})
if err != nil {
return err
}
if len(resp.Containers) == 0 {
break
}
for _, c := range resp.Containers {
err = cli.DeleteContainer(c.Name)
if err != nil {
return err
}
}
}
return nil
}
func (b BlobStorageClient) putSingleBlockBlob(container, name string, chunk []byte) error {
if len(chunk) > MaxBlobBlockSize {
return fmt.Errorf("storage: provided chunk (%d bytes) cannot fit into single-block blob (max %d bytes)", len(chunk), MaxBlobBlockSize)
}
uri := b.client.getEndpoint(blobServiceName, pathForBlob(container, name), url.Values{})
headers := b.client.getStandardHeaders()
headers["x-ms-blob-type"] = string(BlobTypeBlock)
headers["Content-Length"] = fmt.Sprintf("%v", len(chunk))
resp, err := b.client.exec("PUT", uri, headers, bytes.NewReader(chunk))
if err != nil {
return err
}
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
func randContainer() string {
return testContainerPrefix + randString(32-len(testContainerPrefix))
}
func randString(n int) string {
if n <= 0 {
panic("negative number")
}
const alphanum = "0123456789abcdefghijklmnopqrstuvwxyz"
var bytes = make([]byte, n)
rand.Read(bytes)
for i, b := range bytes {
bytes[i] = alphanum[b%byte(len(alphanum))]
}
return string(bytes)
}

View file

@ -1,359 +0,0 @@
// Package storage provides clients for Microsoft Azure Storage Services.
package storage
import (
"bytes"
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"regexp"
"sort"
"strings"
)
const (
// DefaultBaseURL is the domain name used for storage requests when a
// default client is created.
DefaultBaseURL = "core.windows.net"
// DefaultAPIVersion is the Azure Storage API version string used when a
// basic client is created.
DefaultAPIVersion = "2014-02-14"
defaultUseHTTPS = true
blobServiceName = "blob"
tableServiceName = "table"
queueServiceName = "queue"
)
// Client is the object that needs to be constructed to perform
// operations on the storage account.
type Client struct {
accountName string
accountKey []byte
useHTTPS bool
baseURL string
apiVersion string
}
type storageResponse struct {
statusCode int
headers http.Header
body io.ReadCloser
}
// AzureStorageServiceError contains fields of the error response from
// Azure Storage Service REST API. See https://msdn.microsoft.com/en-us/library/azure/dd179382.aspx
// Some fields might be specific to certain calls.
type AzureStorageServiceError struct {
Code string `xml:"Code"`
Message string `xml:"Message"`
AuthenticationErrorDetail string `xml:"AuthenticationErrorDetail"`
QueryParameterName string `xml:"QueryParameterName"`
QueryParameterValue string `xml:"QueryParameterValue"`
Reason string `xml:"Reason"`
StatusCode int
RequestID string
}
// UnexpectedStatusCodeError is returned when a storage service responds with neither an error
// nor with an HTTP status code indicating success.
type UnexpectedStatusCodeError struct {
allowed []int
got int
}
func (e UnexpectedStatusCodeError) Error() string {
s := func(i int) string { return fmt.Sprintf("%d %s", i, http.StatusText(i)) }
got := s(e.got)
expected := []string{}
for _, v := range e.allowed {
expected = append(expected, s(v))
}
return fmt.Sprintf("storage: status code from service response is %s; was expecting %s", got, strings.Join(expected, " or "))
}
// NewBasicClient constructs a Client with given storage service name and
// key.
func NewBasicClient(accountName, accountKey string) (Client, error) {
return NewClient(accountName, accountKey, DefaultBaseURL, DefaultAPIVersion, defaultUseHTTPS)
}
// NewClient constructs a Client. This should be used if the caller wants
// to specify whether to use HTTPS, a specific REST API version or a custom
// storage endpoint than Azure Public Cloud.
func NewClient(accountName, accountKey, blobServiceBaseURL, apiVersion string, useHTTPS bool) (Client, error) {
var c Client
if accountName == "" {
return c, fmt.Errorf("azure: account name required")
} else if accountKey == "" {
return c, fmt.Errorf("azure: account key required")
} else if blobServiceBaseURL == "" {
return c, fmt.Errorf("azure: base storage service url required")
}
key, err := base64.StdEncoding.DecodeString(accountKey)
if err != nil {
return c, err
}
return Client{
accountName: accountName,
accountKey: key,
useHTTPS: useHTTPS,
baseURL: blobServiceBaseURL,
apiVersion: apiVersion,
}, nil
}
func (c Client) getBaseURL(service string) string {
scheme := "http"
if c.useHTTPS {
scheme = "https"
}
host := fmt.Sprintf("%s.%s.%s", c.accountName, service, c.baseURL)
u := &url.URL{
Scheme: scheme,
Host: host}
return u.String()
}
func (c Client) getEndpoint(service, path string, params url.Values) string {
u, err := url.Parse(c.getBaseURL(service))
if err != nil {
// really should not be happening
panic(err)
}
if path == "" {
path = "/" // API doesn't accept path segments not starting with '/'
}
u.Path = path
u.RawQuery = params.Encode()
return u.String()
}
// GetBlobService returns a BlobStorageClient which can operate on the blob
// service of the storage account.
func (c Client) GetBlobService() BlobStorageClient {
return BlobStorageClient{c}
}
// GetQueueService returns a QueueServiceClient which can operate on the queue
// service of the storage account.
func (c Client) GetQueueService() QueueServiceClient {
return QueueServiceClient{c}
}
func (c Client) createAuthorizationHeader(canonicalizedString string) string {
signature := c.computeHmac256(canonicalizedString)
return fmt.Sprintf("%s %s:%s", "SharedKey", c.accountName, signature)
}
func (c Client) getAuthorizationHeader(verb, url string, headers map[string]string) (string, error) {
canonicalizedResource, err := c.buildCanonicalizedResource(url)
if err != nil {
return "", err
}
canonicalizedString := c.buildCanonicalizedString(verb, headers, canonicalizedResource)
return c.createAuthorizationHeader(canonicalizedString), nil
}
func (c Client) getStandardHeaders() map[string]string {
return map[string]string{
"x-ms-version": c.apiVersion,
"x-ms-date": currentTimeRfc1123Formatted(),
}
}
func (c Client) buildCanonicalizedHeader(headers map[string]string) string {
cm := make(map[string]string)
for k, v := range headers {
headerName := strings.TrimSpace(strings.ToLower(k))
match, _ := regexp.MatchString("x-ms-", headerName)
if match {
cm[headerName] = v
}
}
if len(cm) == 0 {
return ""
}
keys := make([]string, 0, len(cm))
for key := range cm {
keys = append(keys, key)
}
sort.Strings(keys)
ch := ""
for i, key := range keys {
if i == len(keys)-1 {
ch += fmt.Sprintf("%s:%s", key, cm[key])
} else {
ch += fmt.Sprintf("%s:%s\n", key, cm[key])
}
}
return ch
}
func (c Client) buildCanonicalizedResource(uri string) (string, error) {
errMsg := "buildCanonicalizedResource error: %s"
u, err := url.Parse(uri)
if err != nil {
return "", fmt.Errorf(errMsg, err.Error())
}
cr := "/" + c.accountName
if len(u.Path) > 0 {
cr += u.Path
}
params, err := url.ParseQuery(u.RawQuery)
if err != nil {
return "", fmt.Errorf(errMsg, err.Error())
}
if len(params) > 0 {
cr += "\n"
keys := make([]string, 0, len(params))
for key := range params {
keys = append(keys, key)
}
sort.Strings(keys)
for i, key := range keys {
if len(params[key]) > 1 {
sort.Strings(params[key])
}
if i == len(keys)-1 {
cr += fmt.Sprintf("%s:%s", key, strings.Join(params[key], ","))
} else {
cr += fmt.Sprintf("%s:%s\n", key, strings.Join(params[key], ","))
}
}
}
return cr, nil
}
func (c Client) buildCanonicalizedString(verb string, headers map[string]string, canonicalizedResource string) string {
canonicalizedString := fmt.Sprintf("%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s",
verb,
headers["Content-Encoding"],
headers["Content-Language"],
headers["Content-Length"],
headers["Content-MD5"],
headers["Content-Type"],
headers["Date"],
headers["If-Modified-Singe"],
headers["If-Match"],
headers["If-None-Match"],
headers["If-Unmodified-Singe"],
headers["Range"],
c.buildCanonicalizedHeader(headers),
canonicalizedResource)
return canonicalizedString
}
func (c Client) exec(verb, url string, headers map[string]string, body io.Reader) (*storageResponse, error) {
authHeader, err := c.getAuthorizationHeader(verb, url, headers)
if err != nil {
return nil, err
}
headers["Authorization"] = authHeader
if err != nil {
return nil, err
}
req, err := http.NewRequest(verb, url, body)
for k, v := range headers {
req.Header.Add(k, v)
}
httpClient := http.Client{}
resp, err := httpClient.Do(req)
if err != nil {
return nil, err
}
statusCode := resp.StatusCode
if statusCode >= 400 && statusCode <= 505 {
var respBody []byte
respBody, err = readResponseBody(resp)
if err != nil {
return nil, err
}
if len(respBody) == 0 {
// no error in response body
err = fmt.Errorf("storage: service returned without a response body (%s)", resp.Status)
} else {
// response contains storage service error object, unmarshal
storageErr, errIn := serviceErrFromXML(respBody, resp.StatusCode, resp.Header.Get("x-ms-request-id"))
if err != nil { // error unmarshaling the error response
err = errIn
}
err = storageErr
}
return &storageResponse{
statusCode: resp.StatusCode,
headers: resp.Header,
body: ioutil.NopCloser(bytes.NewReader(respBody)), /* restore the body */
}, err
}
return &storageResponse{
statusCode: resp.StatusCode,
headers: resp.Header,
body: resp.Body}, nil
}
func readResponseBody(resp *http.Response) ([]byte, error) {
defer resp.Body.Close()
out, err := ioutil.ReadAll(resp.Body)
if err == io.EOF {
err = nil
}
return out, err
}
func serviceErrFromXML(body []byte, statusCode int, requestID string) (AzureStorageServiceError, error) {
var storageErr AzureStorageServiceError
if err := xml.Unmarshal(body, &storageErr); err != nil {
return storageErr, err
}
storageErr.StatusCode = statusCode
storageErr.RequestID = requestID
return storageErr, nil
}
func (e AzureStorageServiceError) Error() string {
return fmt.Sprintf("storage: service returned error: StatusCode=%d, ErrorCode=%s, ErrorMessage=%s, RequestId=%s", e.StatusCode, e.Code, e.Message, e.RequestID)
}
// checkRespCode returns UnexpectedStatusError if the given response code is not
// one of the allowed status codes; otherwise nil.
func checkRespCode(respCode int, allowed []int) error {
for _, v := range allowed {
if respCode == v {
return nil
}
}
return UnexpectedStatusCodeError{allowed, respCode}
}

View file

@ -1,156 +0,0 @@
package storage
import (
"encoding/base64"
"net/url"
"os"
"testing"
chk "gopkg.in/check.v1"
)
// Hook up gocheck to testing
func Test(t *testing.T) { chk.TestingT(t) }
type StorageClientSuite struct{}
var _ = chk.Suite(&StorageClientSuite{})
// getBasicClient returns a test client from storage credentials in the env
func getBasicClient(c *chk.C) Client {
name := os.Getenv("ACCOUNT_NAME")
if name == "" {
c.Fatal("ACCOUNT_NAME not set, need an empty storage account to test")
}
key := os.Getenv("ACCOUNT_KEY")
if key == "" {
c.Fatal("ACCOUNT_KEY not set")
}
cli, err := NewBasicClient(name, key)
c.Assert(err, chk.IsNil)
return cli
}
func (s *StorageClientSuite) TestGetBaseURL_Basic_Https(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
c.Assert(cli.apiVersion, chk.Equals, DefaultAPIVersion)
c.Assert(err, chk.IsNil)
c.Assert(cli.getBaseURL("table"), chk.Equals, "https://foo.table.core.windows.net")
}
func (s *StorageClientSuite) TestGetBaseURL_Custom_NoHttps(c *chk.C) {
apiVersion := "2015-01-01" // a non existing one
cli, err := NewClient("foo", "YmFy", "core.chinacloudapi.cn", apiVersion, false)
c.Assert(err, chk.IsNil)
c.Assert(cli.apiVersion, chk.Equals, apiVersion)
c.Assert(cli.getBaseURL("table"), chk.Equals, "http://foo.table.core.chinacloudapi.cn")
}
func (s *StorageClientSuite) TestGetEndpoint_None(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
output := cli.getEndpoint(blobServiceName, "", url.Values{})
c.Assert(output, chk.Equals, "https://foo.blob.core.windows.net/")
}
func (s *StorageClientSuite) TestGetEndpoint_PathOnly(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
output := cli.getEndpoint(blobServiceName, "path", url.Values{})
c.Assert(output, chk.Equals, "https://foo.blob.core.windows.net/path")
}
func (s *StorageClientSuite) TestGetEndpoint_ParamsOnly(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
params := url.Values{}
params.Set("a", "b")
params.Set("c", "d")
output := cli.getEndpoint(blobServiceName, "", params)
c.Assert(output, chk.Equals, "https://foo.blob.core.windows.net/?a=b&c=d")
}
func (s *StorageClientSuite) TestGetEndpoint_Mixed(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
params := url.Values{}
params.Set("a", "b")
params.Set("c", "d")
output := cli.getEndpoint(blobServiceName, "path", params)
c.Assert(output, chk.Equals, "https://foo.blob.core.windows.net/path?a=b&c=d")
}
func (s *StorageClientSuite) Test_getStandardHeaders(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
headers := cli.getStandardHeaders()
c.Assert(len(headers), chk.Equals, 2)
c.Assert(headers["x-ms-version"], chk.Equals, cli.apiVersion)
if _, ok := headers["x-ms-date"]; !ok {
c.Fatal("Missing date header")
}
}
func (s *StorageClientSuite) Test_buildCanonicalizedResource(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
type test struct{ url, expected string }
tests := []test{
{"https://foo.blob.core.windows.net/path?a=b&c=d", "/foo/path\na:b\nc:d"},
{"https://foo.blob.core.windows.net/?comp=list", "/foo/\ncomp:list"},
{"https://foo.blob.core.windows.net/cnt/blob", "/foo/cnt/blob"},
}
for _, i := range tests {
out, err := cli.buildCanonicalizedResource(i.url)
c.Assert(err, chk.IsNil)
c.Assert(out, chk.Equals, i.expected)
}
}
func (s *StorageClientSuite) Test_buildCanonicalizedHeader(c *chk.C) {
cli, err := NewBasicClient("foo", "YmFy")
c.Assert(err, chk.IsNil)
type test struct {
headers map[string]string
expected string
}
tests := []test{
{map[string]string{}, ""},
{map[string]string{"x-ms-foo": "bar"}, "x-ms-foo:bar"},
{map[string]string{"foo:": "bar"}, ""},
{map[string]string{"foo:": "bar", "x-ms-foo": "bar"}, "x-ms-foo:bar"},
{map[string]string{
"x-ms-version": "9999-99-99",
"x-ms-blob-type": "BlockBlob"}, "x-ms-blob-type:BlockBlob\nx-ms-version:9999-99-99"}}
for _, i := range tests {
c.Assert(cli.buildCanonicalizedHeader(i.headers), chk.Equals, i.expected)
}
}
func (s *StorageClientSuite) TestReturnsStorageServiceError(c *chk.C) {
// attempt to delete a nonexisting container
_, err := getBlobClient(c).deleteContainer(randContainer())
c.Assert(err, chk.NotNil)
v, ok := err.(AzureStorageServiceError)
c.Check(ok, chk.Equals, true)
c.Assert(v.StatusCode, chk.Equals, 404)
c.Assert(v.Code, chk.Equals, "ContainerNotFound")
c.Assert(v.Code, chk.Not(chk.Equals), "")
}
func (s *StorageClientSuite) Test_createAuthorizationHeader(c *chk.C) {
key := base64.StdEncoding.EncodeToString([]byte("bar"))
cli, err := NewBasicClient("foo", key)
c.Assert(err, chk.IsNil)
canonicalizedString := `foobarzoo`
expected := `SharedKey foo:h5U0ATVX6SpbFX1H6GNuxIMeXXCILLoIvhflPtuQZ30=`
c.Assert(cli.createAuthorizationHeader(canonicalizedString), chk.Equals, expected)
}

View file

@ -1,230 +0,0 @@
package storage
import (
"encoding/xml"
"fmt"
"net/http"
"net/url"
"strconv"
)
// QueueServiceClient contains operations for Microsoft Azure Queue Storage
// Service.
type QueueServiceClient struct {
client Client
}
func pathForQueue(queue string) string { return fmt.Sprintf("/%s", queue) }
func pathForQueueMessages(queue string) string { return fmt.Sprintf("/%s/messages", queue) }
func pathForMessage(queue, name string) string { return fmt.Sprintf("/%s/messages/%s", queue, name) }
type putMessageRequest struct {
XMLName xml.Name `xml:"QueueMessage"`
MessageText string `xml:"MessageText"`
}
// PutMessageParameters is the set of options can be specified for Put Messsage
// operation. A zero struct does not use any preferences for the request.
type PutMessageParameters struct {
VisibilityTimeout int
MessageTTL int
}
func (p PutMessageParameters) getParameters() url.Values {
out := url.Values{}
if p.VisibilityTimeout != 0 {
out.Set("visibilitytimeout", strconv.Itoa(p.VisibilityTimeout))
}
if p.MessageTTL != 0 {
out.Set("messagettl", strconv.Itoa(p.MessageTTL))
}
return out
}
// GetMessagesParameters is the set of options can be specified for Get
// Messsages operation. A zero struct does not use any preferences for the
// request.
type GetMessagesParameters struct {
NumOfMessages int
VisibilityTimeout int
}
func (p GetMessagesParameters) getParameters() url.Values {
out := url.Values{}
if p.NumOfMessages != 0 {
out.Set("numofmessages", strconv.Itoa(p.NumOfMessages))
}
if p.VisibilityTimeout != 0 {
out.Set("visibilitytimeout", strconv.Itoa(p.VisibilityTimeout))
}
return out
}
// PeekMessagesParameters is the set of options can be specified for Peek
// Messsage operation. A zero struct does not use any preferences for the
// request.
type PeekMessagesParameters struct {
NumOfMessages int
}
func (p PeekMessagesParameters) getParameters() url.Values {
out := url.Values{"peekonly": {"true"}} // Required for peek operation
if p.NumOfMessages != 0 {
out.Set("numofmessages", strconv.Itoa(p.NumOfMessages))
}
return out
}
// GetMessagesResponse represents a response returned from Get Messages
// operation.
type GetMessagesResponse struct {
XMLName xml.Name `xml:"QueueMessagesList"`
QueueMessagesList []GetMessageResponse `xml:"QueueMessage"`
}
// GetMessageResponse represents a QueueMessage object returned from Get
// Messages operation response.
type GetMessageResponse struct {
MessageID string `xml:"MessageId"`
InsertionTime string `xml:"InsertionTime"`
ExpirationTime string `xml:"ExpirationTime"`
PopReceipt string `xml:"PopReceipt"`
TimeNextVisible string `xml:"TimeNextVisible"`
DequeueCount int `xml:"DequeueCount"`
MessageText string `xml:"MessageText"`
}
// PeekMessagesResponse represents a response returned from Get Messages
// operation.
type PeekMessagesResponse struct {
XMLName xml.Name `xml:"QueueMessagesList"`
QueueMessagesList []PeekMessageResponse `xml:"QueueMessage"`
}
// PeekMessageResponse represents a QueueMessage object returned from Peek
// Messages operation response.
type PeekMessageResponse struct {
MessageID string `xml:"MessageId"`
InsertionTime string `xml:"InsertionTime"`
ExpirationTime string `xml:"ExpirationTime"`
DequeueCount int `xml:"DequeueCount"`
MessageText string `xml:"MessageText"`
}
// CreateQueue operation creates a queue under the given account.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179342.aspx
func (c QueueServiceClient) CreateQueue(name string) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{})
headers := c.client.getStandardHeaders()
headers["Content-Length"] = "0"
resp, err := c.client.exec("PUT", uri, headers, nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// DeleteQueue operation permanently deletes the specified queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179436.aspx
func (c QueueServiceClient) DeleteQueue(name string) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{})
resp, err := c.client.exec("DELETE", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}
// QueueExists returns true if a queue with given name exists.
func (c QueueServiceClient) QueueExists(name string) (bool, error) {
uri := c.client.getEndpoint(queueServiceName, pathForQueue(name), url.Values{"comp": {"metadata"}})
resp, err := c.client.exec("GET", uri, c.client.getStandardHeaders(), nil)
if resp != nil && (resp.statusCode == http.StatusOK || resp.statusCode == http.StatusNotFound) {
return resp.statusCode == http.StatusOK, nil
}
return false, err
}
// PutMessage operation adds a new message to the back of the message queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179346.aspx
func (c QueueServiceClient) PutMessage(queue string, message string, params PutMessageParameters) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), params.getParameters())
req := putMessageRequest{MessageText: message}
body, nn, err := xmlMarshal(req)
if err != nil {
return err
}
headers := c.client.getStandardHeaders()
headers["Content-Length"] = strconv.Itoa(nn)
resp, err := c.client.exec("POST", uri, headers, body)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusCreated})
}
// ClearMessages operation deletes all messages from the specified queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179454.aspx
func (c QueueServiceClient) ClearMessages(queue string) error {
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), url.Values{})
resp, err := c.client.exec("DELETE", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}
// GetMessages operation retrieves one or more messages from the front of the
// queue.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179474.aspx
func (c QueueServiceClient) GetMessages(queue string, params GetMessagesParameters) (GetMessagesResponse, error) {
var r GetMessagesResponse
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), params.getParameters())
resp, err := c.client.exec("GET", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return r, err
}
defer resp.body.Close()
err = xmlUnmarshal(resp.body, &r)
return r, err
}
// PeekMessages retrieves one or more messages from the front of the queue, but
// does not alter the visibility of the message.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179472.aspx
func (c QueueServiceClient) PeekMessages(queue string, params PeekMessagesParameters) (PeekMessagesResponse, error) {
var r PeekMessagesResponse
uri := c.client.getEndpoint(queueServiceName, pathForQueueMessages(queue), params.getParameters())
resp, err := c.client.exec("GET", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return r, err
}
defer resp.body.Close()
err = xmlUnmarshal(resp.body, &r)
return r, err
}
// DeleteMessage operation deletes the specified message.
//
// See https://msdn.microsoft.com/en-us/library/azure/dd179347.aspx
func (c QueueServiceClient) DeleteMessage(queue, messageID, popReceipt string) error {
uri := c.client.getEndpoint(queueServiceName, pathForMessage(queue, messageID), url.Values{
"popreceipt": {popReceipt}})
resp, err := c.client.exec("DELETE", uri, c.client.getStandardHeaders(), nil)
if err != nil {
return err
}
defer resp.body.Close()
return checkRespCode(resp.statusCode, []int{http.StatusNoContent})
}

View file

@ -1,91 +0,0 @@
package storage
import (
chk "gopkg.in/check.v1"
)
type StorageQueueSuite struct{}
var _ = chk.Suite(&StorageQueueSuite{})
func getQueueClient(c *chk.C) QueueServiceClient {
return getBasicClient(c).GetQueueService()
}
func (s *StorageQueueSuite) Test_pathForQueue(c *chk.C) {
c.Assert(pathForQueue("q"), chk.Equals, "/q")
}
func (s *StorageQueueSuite) Test_pathForQueueMessages(c *chk.C) {
c.Assert(pathForQueueMessages("q"), chk.Equals, "/q/messages")
}
func (s *StorageQueueSuite) Test_pathForMessage(c *chk.C) {
c.Assert(pathForMessage("q", "m"), chk.Equals, "/q/messages/m")
}
func (s *StorageQueueSuite) TestCreateQueue_DeleteQueue(c *chk.C) {
cli := getQueueClient(c)
name := randString(20)
c.Assert(cli.CreateQueue(name), chk.IsNil)
c.Assert(cli.DeleteQueue(name), chk.IsNil)
}
func (s *StorageQueueSuite) TestQueueExists(c *chk.C) {
cli := getQueueClient(c)
ok, err := cli.QueueExists("nonexistent-queue")
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, false)
name := randString(20)
c.Assert(cli.CreateQueue(name), chk.IsNil)
defer cli.DeleteQueue(name)
ok, err = cli.QueueExists(name)
c.Assert(err, chk.IsNil)
c.Assert(ok, chk.Equals, true)
}
func (s *StorageQueueSuite) TestPostMessage_PeekMessage_DeleteMessage(c *chk.C) {
q := randString(20)
cli := getQueueClient(c)
c.Assert(cli.CreateQueue(q), chk.IsNil)
defer cli.DeleteQueue(q)
msg := randString(64 * 1024) // exercise max length
c.Assert(cli.PutMessage(q, msg, PutMessageParameters{}), chk.IsNil)
r, err := cli.PeekMessages(q, PeekMessagesParameters{})
c.Assert(err, chk.IsNil)
c.Assert(len(r.QueueMessagesList), chk.Equals, 1)
c.Assert(r.QueueMessagesList[0].MessageText, chk.Equals, msg)
}
func (s *StorageQueueSuite) TestGetMessages(c *chk.C) {
q := randString(20)
cli := getQueueClient(c)
c.Assert(cli.CreateQueue(q), chk.IsNil)
defer cli.DeleteQueue(q)
n := 4
for i := 0; i < n; i++ {
c.Assert(cli.PutMessage(q, randString(10), PutMessageParameters{}), chk.IsNil)
}
r, err := cli.GetMessages(q, GetMessagesParameters{NumOfMessages: n})
c.Assert(err, chk.IsNil)
c.Assert(len(r.QueueMessagesList), chk.Equals, n)
}
func (s *StorageQueueSuite) TestDeleteMessages(c *chk.C) {
q := randString(20)
cli := getQueueClient(c)
c.Assert(cli.CreateQueue(q), chk.IsNil)
defer cli.DeleteQueue(q)
c.Assert(cli.PutMessage(q, "message", PutMessageParameters{}), chk.IsNil)
r, err := cli.GetMessages(q, GetMessagesParameters{VisibilityTimeout: 1})
c.Assert(err, chk.IsNil)
c.Assert(len(r.QueueMessagesList), chk.Equals, 1)
m := r.QueueMessagesList[0]
c.Assert(cli.DeleteMessage(q, m.MessageID, m.PopReceipt), chk.IsNil)
}

View file

@ -1,71 +0,0 @@
package storage
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"encoding/xml"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"time"
)
func (c Client) computeHmac256(message string) string {
h := hmac.New(sha256.New, c.accountKey)
h.Write([]byte(message))
return base64.StdEncoding.EncodeToString(h.Sum(nil))
}
func currentTimeRfc1123Formatted() string {
return timeRfc1123Formatted(time.Now().UTC())
}
func timeRfc1123Formatted(t time.Time) string {
return t.Format(http.TimeFormat)
}
func mergeParams(v1, v2 url.Values) url.Values {
out := url.Values{}
for k, v := range v1 {
out[k] = v
}
for k, v := range v2 {
vals, ok := out[k]
if ok {
vals = append(vals, v...)
out[k] = vals
} else {
out[k] = v
}
}
return out
}
func prepareBlockListRequest(blocks []Block) string {
s := `<?xml version="1.0" encoding="utf-8"?><BlockList>`
for _, v := range blocks {
s += fmt.Sprintf("<%s>%s</%s>", v.Status, v.ID, v.Status)
}
s += `</BlockList>`
return s
}
func xmlUnmarshal(body io.Reader, v interface{}) error {
data, err := ioutil.ReadAll(body)
if err != nil {
return err
}
return xml.Unmarshal(data, v)
}
func xmlMarshal(v interface{}) (io.Reader, int, error) {
b, err := xml.Marshal(v)
if err != nil {
return nil, 0, err
}
return bytes.NewReader(b), len(b), nil
}

View file

@ -1,69 +0,0 @@
package storage
import (
"encoding/xml"
"io/ioutil"
"net/url"
"strings"
"time"
chk "gopkg.in/check.v1"
)
func (s *StorageClientSuite) Test_timeRfc1123Formatted(c *chk.C) {
now := time.Now().UTC()
expectedLayout := "Mon, 02 Jan 2006 15:04:05 GMT"
c.Assert(timeRfc1123Formatted(now), chk.Equals, now.Format(expectedLayout))
}
func (s *StorageClientSuite) Test_mergeParams(c *chk.C) {
v1 := url.Values{
"k1": {"v1"},
"k2": {"v2"}}
v2 := url.Values{
"k1": {"v11"},
"k3": {"v3"}}
out := mergeParams(v1, v2)
c.Assert(out.Get("k1"), chk.Equals, "v1")
c.Assert(out.Get("k2"), chk.Equals, "v2")
c.Assert(out.Get("k3"), chk.Equals, "v3")
c.Assert(out["k1"], chk.DeepEquals, []string{"v1", "v11"})
}
func (s *StorageClientSuite) Test_prepareBlockListRequest(c *chk.C) {
empty := []Block{}
expected := `<?xml version="1.0" encoding="utf-8"?><BlockList></BlockList>`
c.Assert(prepareBlockListRequest(empty), chk.DeepEquals, expected)
blocks := []Block{{"foo", BlockStatusLatest}, {"bar", BlockStatusUncommitted}}
expected = `<?xml version="1.0" encoding="utf-8"?><BlockList><Latest>foo</Latest><Uncommitted>bar</Uncommitted></BlockList>`
c.Assert(prepareBlockListRequest(blocks), chk.DeepEquals, expected)
}
func (s *StorageClientSuite) Test_xmlUnmarshal(c *chk.C) {
xml := `<?xml version="1.0" encoding="utf-8"?>
<Blob>
<Name>myblob</Name>
</Blob>`
var blob Blob
body := ioutil.NopCloser(strings.NewReader(xml))
c.Assert(xmlUnmarshal(body, &blob), chk.IsNil)
c.Assert(blob.Name, chk.Equals, "myblob")
}
func (s *StorageClientSuite) Test_xmlMarshal(c *chk.C) {
type t struct {
XMLName xml.Name `xml:"S"`
Name string `xml:"Name"`
}
b := t{Name: "myblob"}
expected := `<S><Name>myblob</Name></S>`
r, i, err := xmlMarshal(b)
c.Assert(err, chk.IsNil)
o, err := ioutil.ReadAll(r)
c.Assert(err, chk.IsNil)
out := string(o)
c.Assert(out, chk.Equals, expected)
c.Assert(i, chk.Equals, len(expected))
}

View file

@ -1,8 +0,0 @@
language: go
go:
- 1.2
- 1.3
- 1.4
- tip
install:
- go get -t ./...

View file

@ -1,7 +0,0 @@
# 0.7.3
formatter/\*: allow configuration of timestamp layout
# 0.7.2
formatter/text: Add configuration option for time format (#158)

View file

@ -1,349 +0,0 @@
# Logrus <img src="http://i.imgur.com/hTeVwmJ.png" width="40" height="40" alt=":walrus:" class="emoji" title=":walrus:"/>&nbsp;[![Build Status](https://travis-ci.org/Sirupsen/logrus.svg?branch=master)](https://travis-ci.org/Sirupsen/logrus)&nbsp;[![godoc reference](https://godoc.org/github.com/Sirupsen/logrus?status.png)][godoc]
Logrus is a structured logger for Go (golang), completely API compatible with
the standard library logger. [Godoc][godoc]. **Please note the Logrus API is not
yet stable (pre 1.0). Logrus itself is completely stable and has been used in
many large deployments. The core API is unlikely to change much but please
version control your Logrus to make sure you aren't fetching latest `master` on
every build.**
Nicely color-coded in development (when a TTY is attached, otherwise just
plain text):
![Colored](http://i.imgur.com/PY7qMwd.png)
With `log.Formatter = new(logrus.JSONFormatter)`, for easy parsing by logstash
or Splunk:
```json
{"animal":"walrus","level":"info","msg":"A group of walrus emerges from the
ocean","size":10,"time":"2014-03-10 19:57:38.562264131 -0400 EDT"}
{"level":"warning","msg":"The group's number increased tremendously!",
"number":122,"omg":true,"time":"2014-03-10 19:57:38.562471297 -0400 EDT"}
{"animal":"walrus","level":"info","msg":"A giant walrus appears!",
"size":10,"time":"2014-03-10 19:57:38.562500591 -0400 EDT"}
{"animal":"walrus","level":"info","msg":"Tremendously sized cow enters the ocean.",
"size":9,"time":"2014-03-10 19:57:38.562527896 -0400 EDT"}
{"level":"fatal","msg":"The ice breaks!","number":100,"omg":true,
"time":"2014-03-10 19:57:38.562543128 -0400 EDT"}
```
With the default `log.Formatter = new(logrus.TextFormatter)` when a TTY is not
attached, the output is compatible with the
[logfmt](http://godoc.org/github.com/kr/logfmt) format:
```text
time="2015-03-26T01:27:38-04:00" level=debug msg="Started observing beach" animal=walrus number=8
time="2015-03-26T01:27:38-04:00" level=info msg="A group of walrus emerges from the ocean" animal=walrus size=10
time="2015-03-26T01:27:38-04:00" level=warning msg="The group's number increased tremendously!" number=122 omg=true
time="2015-03-26T01:27:38-04:00" level=debug msg="Temperature changes" temperature=-4
time="2015-03-26T01:27:38-04:00" level=panic msg="It's over 9000!" animal=orca size=9009
time="2015-03-26T01:27:38-04:00" level=fatal msg="The ice breaks!" err=&{0x2082280c0 map[animal:orca size:9009] 2015-03-26 01:27:38.441574009 -0400 EDT panic It's over 9000!} number=100 omg=true
exit status 1
```
#### Example
The simplest way to use Logrus is simply the package-level exported logger:
```go
package main
import (
log "github.com/Sirupsen/logrus"
)
func main() {
log.WithFields(log.Fields{
"animal": "walrus",
}).Info("A walrus appears")
}
```
Note that it's completely api-compatible with the stdlib logger, so you can
replace your `log` imports everywhere with `log "github.com/Sirupsen/logrus"`
and you'll now have the flexibility of Logrus. You can customize it all you
want:
```go
package main
import (
"os"
log "github.com/Sirupsen/logrus"
"github.com/Sirupsen/logrus/hooks/airbrake"
)
func init() {
// Log as JSON instead of the default ASCII formatter.
log.SetFormatter(&log.JSONFormatter{})
// Use the Airbrake hook to report errors that have Error severity or above to
// an exception tracker. You can create custom hooks, see the Hooks section.
log.AddHook(airbrake.NewHook("https://example.com", "xyz", "development"))
// Output to stderr instead of stdout, could also be a file.
log.SetOutput(os.Stderr)
// Only log the warning severity or above.
log.SetLevel(log.WarnLevel)
}
func main() {
log.WithFields(log.Fields{
"animal": "walrus",
"size": 10,
}).Info("A group of walrus emerges from the ocean")
log.WithFields(log.Fields{
"omg": true,
"number": 122,
}).Warn("The group's number increased tremendously!")
log.WithFields(log.Fields{
"omg": true,
"number": 100,
}).Fatal("The ice breaks!")
// A common pattern is to re-use fields between logging statements by re-using
// the logrus.Entry returned from WithFields()
contextLogger := log.WithFields(log.Fields{
"common": "this is a common field",
"other": "I also should be logged always",
})
contextLogger.Info("I'll be logged with common and other field")
contextLogger.Info("Me too")
}
```
For more advanced usage such as logging to multiple locations from the same
application, you can also create an instance of the `logrus` Logger:
```go
package main
import (
"github.com/Sirupsen/logrus"
)
// Create a new instance of the logger. You can have any number of instances.
var log = logrus.New()
func main() {
// The API for setting attributes is a little different than the package level
// exported logger. See Godoc.
log.Out = os.Stderr
log.WithFields(logrus.Fields{
"animal": "walrus",
"size": 10,
}).Info("A group of walrus emerges from the ocean")
}
```
#### Fields
Logrus encourages careful, structured logging though logging fields instead of
long, unparseable error messages. For example, instead of: `log.Fatalf("Failed
to send event %s to topic %s with key %d")`, you should log the much more
discoverable:
```go
log.WithFields(log.Fields{
"event": event,
"topic": topic,
"key": key,
}).Fatal("Failed to send event")
```
We've found this API forces you to think about logging in a way that produces
much more useful logging messages. We've been in countless situations where just
a single added field to a log statement that was already there would've saved us
hours. The `WithFields` call is optional.
In general, with Logrus using any of the `printf`-family functions should be
seen as a hint you should add a field, however, you can still use the
`printf`-family functions with Logrus.
#### Hooks
You can add hooks for logging levels. For example to send errors to an exception
tracking service on `Error`, `Fatal` and `Panic`, info to StatsD or log to
multiple places simultaneously, e.g. syslog.
Logrus comes with [built-in hooks](hooks/). Add those, or your custom hook, in
`init`:
```go
import (
log "github.com/Sirupsen/logrus"
"github.com/Sirupsen/logrus/hooks/airbrake"
"github.com/Sirupsen/logrus/hooks/syslog"
"log/syslog"
)
func init() {
log.AddHook(airbrake.NewHook("https://example.com", "xyz", "development"))
hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
if err != nil {
log.Error("Unable to connect to local syslog daemon")
} else {
log.AddHook(hook)
}
}
```
| Hook | Description |
| ----- | ----------- |
| [Airbrake](https://github.com/Sirupsen/logrus/blob/master/hooks/airbrake/airbrake.go) | Send errors to an exception tracking service compatible with the Airbrake API. Uses [`airbrake-go`](https://github.com/tobi/airbrake-go) behind the scenes. |
| [Papertrail](https://github.com/Sirupsen/logrus/blob/master/hooks/papertrail/papertrail.go) | Send errors to the Papertrail hosted logging service via UDP. |
| [Syslog](https://github.com/Sirupsen/logrus/blob/master/hooks/syslog/syslog.go) | Send errors to remote syslog server. Uses standard library `log/syslog` behind the scenes. |
| [BugSnag](https://github.com/Sirupsen/logrus/blob/master/hooks/bugsnag/bugsnag.go) | Send errors to the Bugsnag exception tracking service. |
| [Hiprus](https://github.com/nubo/hiprus) | Send errors to a channel in hipchat. |
| [Logrusly](https://github.com/sebest/logrusly) | Send logs to [Loggly](https://www.loggly.com/) |
| [Slackrus](https://github.com/johntdyer/slackrus) | Hook for Slack chat. |
| [Journalhook](https://github.com/wercker/journalhook) | Hook for logging to `systemd-journald` |
| [Graylog](https://github.com/gemnasium/logrus-hooks/tree/master/graylog) | Hook for logging to [Graylog](http://graylog2.org/) |
#### Level logging
Logrus has six logging levels: Debug, Info, Warning, Error, Fatal and Panic.
```go
log.Debug("Useful debugging information.")
log.Info("Something noteworthy happened!")
log.Warn("You should probably take a look at this.")
log.Error("Something failed but I'm not quitting.")
// Calls os.Exit(1) after logging
log.Fatal("Bye.")
// Calls panic() after logging
log.Panic("I'm bailing.")
```
You can set the logging level on a `Logger`, then it will only log entries with
that severity or anything above it:
```go
// Will log anything that is info or above (warn, error, fatal, panic). Default.
log.SetLevel(log.InfoLevel)
```
It may be useful to set `log.Level = logrus.DebugLevel` in a debug or verbose
environment if your application has that.
#### Entries
Besides the fields added with `WithField` or `WithFields` some fields are
automatically added to all logging events:
1. `time`. The timestamp when the entry was created.
2. `msg`. The logging message passed to `{Info,Warn,Error,Fatal,Panic}` after
the `AddFields` call. E.g. `Failed to send event.`
3. `level`. The logging level. E.g. `info`.
#### Environments
Logrus has no notion of environment.
If you wish for hooks and formatters to only be used in specific environments,
you should handle that yourself. For example, if your application has a global
variable `Environment`, which is a string representation of the environment you
could do:
```go
import (
log "github.com/Sirupsen/logrus"
)
init() {
// do something here to set environment depending on an environment variable
// or command-line flag
if Environment == "production" {
log.SetFormatter(logrus.JSONFormatter)
} else {
// The TextFormatter is default, you don't actually have to do this.
log.SetFormatter(logrus.TextFormatter)
}
}
```
This configuration is how `logrus` was intended to be used, but JSON in
production is mostly only useful if you do log aggregation with tools like
Splunk or Logstash.
#### Formatters
The built-in logging formatters are:
* `logrus.TextFormatter`. Logs the event in colors if stdout is a tty, otherwise
without colors.
* *Note:* to force colored output when there is no TTY, set the `ForceColors`
field to `true`. To force no colored output even if there is a TTY set the
`DisableColors` field to `true`
* `logrus.JSONFormatter`. Logs fields as JSON.
* `logrus_logstash.LogstashFormatter`. Logs fields as Logstash Events (http://logstash.net).
```go
logrus.SetFormatter(&logrus_logstash.LogstashFormatter{Type: “application_name"})
```
Third party logging formatters:
* [`zalgo`](https://github.com/aybabtme/logzalgo): invoking the P͉̫o̳̼̊w̖͈̰͎e̬͔̭͂r͚̼̹̲ ̫͓͉̳͈ō̠͕͖̚f̝͍̠ ͕̲̞͖͑Z̖̫̤̫ͪa͉̬͈̗l͖͎g̳̥o̰̥̅!̣͔̲̻͊̄ ̙̘̦̹̦.
You can define your formatter by implementing the `Formatter` interface,
requiring a `Format` method. `Format` takes an `*Entry`. `entry.Data` is a
`Fields` type (`map[string]interface{}`) with all your fields as well as the
default ones (see Entries section above):
```go
type MyJSONFormatter struct {
}
log.SetFormatter(new(MyJSONFormatter))
func (f *JSONFormatter) Format(entry *Entry) ([]byte, error) {
// Note this doesn't include Time, Level and Message which are available on
// the Entry. Consult `godoc` on information about those fields or read the
// source of the official loggers.
serialized, err := json.Marshal(entry.Data)
if err != nil {
return nil, fmt.Errorf("Failed to marshal fields to JSON, %v", err)
}
return append(serialized, '\n'), nil
}
```
#### Logger as an `io.Writer`
Logrus can be transormed into an `io.Writer`. That writer is the end of an `io.Pipe` and it is your responsibility to close it.
```go
w := logger.Writer()
defer w.Close()
srv := http.Server{
// create a stdlib log.Logger that writes to
// logrus.Logger.
ErrorLog: log.New(w, "", 0),
}
```
Each line written to that writer will be printed the usual way, using formatters
and hooks. The level for those entries is `info`.
#### Rotation
Log rotation is not provided with Logrus. Log rotation should be done by an
external program (like `logrotate(8)`) that can compress and delete old log
entries. It should not be a feature of the application-level logger.
[godoc]: https://godoc.org/github.com/Sirupsen/logrus

View file

@ -1,252 +0,0 @@
package logrus
import (
"bytes"
"fmt"
"io"
"os"
"time"
)
// An entry is the final or intermediate Logrus logging entry. It contains all
// the fields passed with WithField{,s}. It's finally logged when Debug, Info,
// Warn, Error, Fatal or Panic is called on it. These objects can be reused and
// passed around as much as you wish to avoid field duplication.
type Entry struct {
Logger *Logger
// Contains all the fields set by the user.
Data Fields
// Time at which the log entry was created
Time time.Time
// Level the log entry was logged at: Debug, Info, Warn, Error, Fatal or Panic
Level Level
// Message passed to Debug, Info, Warn, Error, Fatal or Panic
Message string
}
func NewEntry(logger *Logger) *Entry {
return &Entry{
Logger: logger,
// Default is three fields, give a little extra room
Data: make(Fields, 5),
}
}
// Returns a reader for the entry, which is a proxy to the formatter.
func (entry *Entry) Reader() (*bytes.Buffer, error) {
serialized, err := entry.Logger.Formatter.Format(entry)
return bytes.NewBuffer(serialized), err
}
// Returns the string representation from the reader and ultimately the
// formatter.
func (entry *Entry) String() (string, error) {
reader, err := entry.Reader()
if err != nil {
return "", err
}
return reader.String(), err
}
// Add a single field to the Entry.
func (entry *Entry) WithField(key string, value interface{}) *Entry {
return entry.WithFields(Fields{key: value})
}
// Add a map of fields to the Entry.
func (entry *Entry) WithFields(fields Fields) *Entry {
data := Fields{}
for k, v := range entry.Data {
data[k] = v
}
for k, v := range fields {
data[k] = v
}
return &Entry{Logger: entry.Logger, Data: data}
}
func (entry *Entry) log(level Level, msg string) {
entry.Time = time.Now()
entry.Level = level
entry.Message = msg
if err := entry.Logger.Hooks.Fire(level, entry); err != nil {
entry.Logger.mu.Lock()
fmt.Fprintf(os.Stderr, "Failed to fire hook: %v\n", err)
entry.Logger.mu.Unlock()
}
reader, err := entry.Reader()
if err != nil {
entry.Logger.mu.Lock()
fmt.Fprintf(os.Stderr, "Failed to obtain reader, %v\n", err)
entry.Logger.mu.Unlock()
}
entry.Logger.mu.Lock()
defer entry.Logger.mu.Unlock()
_, err = io.Copy(entry.Logger.Out, reader)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to write to log, %v\n", err)
}
// To avoid Entry#log() returning a value that only would make sense for
// panic() to use in Entry#Panic(), we avoid the allocation by checking
// directly here.
if level <= PanicLevel {
panic(entry)
}
}
func (entry *Entry) Debug(args ...interface{}) {
if entry.Logger.Level >= DebugLevel {
entry.log(DebugLevel, fmt.Sprint(args...))
}
}
func (entry *Entry) Print(args ...interface{}) {
entry.Info(args...)
}
func (entry *Entry) Info(args ...interface{}) {
if entry.Logger.Level >= InfoLevel {
entry.log(InfoLevel, fmt.Sprint(args...))
}
}
func (entry *Entry) Warn(args ...interface{}) {
if entry.Logger.Level >= WarnLevel {
entry.log(WarnLevel, fmt.Sprint(args...))
}
}
func (entry *Entry) Warning(args ...interface{}) {
entry.Warn(args...)
}
func (entry *Entry) Error(args ...interface{}) {
if entry.Logger.Level >= ErrorLevel {
entry.log(ErrorLevel, fmt.Sprint(args...))
}
}
func (entry *Entry) Fatal(args ...interface{}) {
if entry.Logger.Level >= FatalLevel {
entry.log(FatalLevel, fmt.Sprint(args...))
}
os.Exit(1)
}
func (entry *Entry) Panic(args ...interface{}) {
if entry.Logger.Level >= PanicLevel {
entry.log(PanicLevel, fmt.Sprint(args...))
}
panic(fmt.Sprint(args...))
}
// Entry Printf family functions
func (entry *Entry) Debugf(format string, args ...interface{}) {
if entry.Logger.Level >= DebugLevel {
entry.Debug(fmt.Sprintf(format, args...))
}
}
func (entry *Entry) Infof(format string, args ...interface{}) {
if entry.Logger.Level >= InfoLevel {
entry.Info(fmt.Sprintf(format, args...))
}
}
func (entry *Entry) Printf(format string, args ...interface{}) {
entry.Infof(format, args...)
}
func (entry *Entry) Warnf(format string, args ...interface{}) {
if entry.Logger.Level >= WarnLevel {
entry.Warn(fmt.Sprintf(format, args...))
}
}
func (entry *Entry) Warningf(format string, args ...interface{}) {
entry.Warnf(format, args...)
}
func (entry *Entry) Errorf(format string, args ...interface{}) {
if entry.Logger.Level >= ErrorLevel {
entry.Error(fmt.Sprintf(format, args...))
}
}
func (entry *Entry) Fatalf(format string, args ...interface{}) {
if entry.Logger.Level >= FatalLevel {
entry.Fatal(fmt.Sprintf(format, args...))
}
}
func (entry *Entry) Panicf(format string, args ...interface{}) {
if entry.Logger.Level >= PanicLevel {
entry.Panic(fmt.Sprintf(format, args...))
}
}
// Entry Println family functions
func (entry *Entry) Debugln(args ...interface{}) {
if entry.Logger.Level >= DebugLevel {
entry.Debug(entry.sprintlnn(args...))
}
}
func (entry *Entry) Infoln(args ...interface{}) {
if entry.Logger.Level >= InfoLevel {
entry.Info(entry.sprintlnn(args...))
}
}
func (entry *Entry) Println(args ...interface{}) {
entry.Infoln(args...)
}
func (entry *Entry) Warnln(args ...interface{}) {
if entry.Logger.Level >= WarnLevel {
entry.Warn(entry.sprintlnn(args...))
}
}
func (entry *Entry) Warningln(args ...interface{}) {
entry.Warnln(args...)
}
func (entry *Entry) Errorln(args ...interface{}) {
if entry.Logger.Level >= ErrorLevel {
entry.Error(entry.sprintlnn(args...))
}
}
func (entry *Entry) Fatalln(args ...interface{}) {
if entry.Logger.Level >= FatalLevel {
entry.Fatal(entry.sprintlnn(args...))
}
}
func (entry *Entry) Panicln(args ...interface{}) {
if entry.Logger.Level >= PanicLevel {
entry.Panic(entry.sprintlnn(args...))
}
}
// Sprintlnn => Sprint no newline. This is to get the behavior of how
// fmt.Sprintln where spaces are always added between operands, regardless of
// their type. Instead of vendoring the Sprintln implementation to spare a
// string allocation, we do the simplest thing.
func (entry *Entry) sprintlnn(args ...interface{}) string {
msg := fmt.Sprintln(args...)
return msg[:len(msg)-1]
}

View file

@ -1,53 +0,0 @@
package logrus
import (
"bytes"
"fmt"
"testing"
"github.com/stretchr/testify/assert"
)
func TestEntryPanicln(t *testing.T) {
errBoom := fmt.Errorf("boom time")
defer func() {
p := recover()
assert.NotNil(t, p)
switch pVal := p.(type) {
case *Entry:
assert.Equal(t, "kaboom", pVal.Message)
assert.Equal(t, errBoom, pVal.Data["err"])
default:
t.Fatalf("want type *Entry, got %T: %#v", pVal, pVal)
}
}()
logger := New()
logger.Out = &bytes.Buffer{}
entry := NewEntry(logger)
entry.WithField("err", errBoom).Panicln("kaboom")
}
func TestEntryPanicf(t *testing.T) {
errBoom := fmt.Errorf("boom again")
defer func() {
p := recover()
assert.NotNil(t, p)
switch pVal := p.(type) {
case *Entry:
assert.Equal(t, "kaboom true", pVal.Message)
assert.Equal(t, errBoom, pVal.Data["err"])
default:
t.Fatalf("want type *Entry, got %T: %#v", pVal, pVal)
}
}()
logger := New()
logger.Out = &bytes.Buffer{}
entry := NewEntry(logger)
entry.WithField("err", errBoom).Panicf("kaboom %v", true)
}

View file

@ -1,50 +0,0 @@
package main
import (
"github.com/Sirupsen/logrus"
)
var log = logrus.New()
func init() {
log.Formatter = new(logrus.JSONFormatter)
log.Formatter = new(logrus.TextFormatter) // default
log.Level = logrus.DebugLevel
}
func main() {
defer func() {
err := recover()
if err != nil {
log.WithFields(logrus.Fields{
"omg": true,
"err": err,
"number": 100,
}).Fatal("The ice breaks!")
}
}()
log.WithFields(logrus.Fields{
"animal": "walrus",
"number": 8,
}).Debug("Started observing beach")
log.WithFields(logrus.Fields{
"animal": "walrus",
"size": 10,
}).Info("A group of walrus emerges from the ocean")
log.WithFields(logrus.Fields{
"omg": true,
"number": 122,
}).Warn("The group's number increased tremendously!")
log.WithFields(logrus.Fields{
"temperature": -4,
}).Debug("Temperature changes")
log.WithFields(logrus.Fields{
"animal": "orca",
"size": 9009,
}).Panic("It's over 9000!")
}

View file

@ -1,30 +0,0 @@
package main
import (
"github.com/Sirupsen/logrus"
"github.com/Sirupsen/logrus/hooks/airbrake"
)
var log = logrus.New()
func init() {
log.Formatter = new(logrus.TextFormatter) // default
log.Hooks.Add(airbrake.NewHook("https://example.com", "xyz", "development"))
}
func main() {
log.WithFields(logrus.Fields{
"animal": "walrus",
"size": 10,
}).Info("A group of walrus emerges from the ocean")
log.WithFields(logrus.Fields{
"omg": true,
"number": 122,
}).Warn("The group's number increased tremendously!")
log.WithFields(logrus.Fields{
"omg": true,
"number": 100,
}).Fatal("The ice breaks!")
}

View file

@ -1,48 +0,0 @@
package logrus
import "time"
const DefaultTimestampFormat = time.RFC3339
// The Formatter interface is used to implement a custom Formatter. It takes an
// `Entry`. It exposes all the fields, including the default ones:
//
// * `entry.Data["msg"]`. The message passed from Info, Warn, Error ..
// * `entry.Data["time"]`. The timestamp.
// * `entry.Data["level"]. The level the entry was logged at.
//
// Any additional fields added with `WithField` or `WithFields` are also in
// `entry.Data`. Format is expected to return an array of bytes which are then
// logged to `logger.Out`.
type Formatter interface {
Format(*Entry) ([]byte, error)
}
// This is to not silently overwrite `time`, `msg` and `level` fields when
// dumping it. If this code wasn't there doing:
//
// logrus.WithField("level", 1).Info("hello")
//
// Would just silently drop the user provided level. Instead with this code
// it'll logged as:
//
// {"level": "info", "fields.level": 1, "msg": "hello", "time": "..."}
//
// It's not exported because it's still using Data in an opinionated way. It's to
// avoid code duplication between the two default formatters.
func prefixFieldClashes(data Fields) {
_, ok := data["time"]
if ok {
data["fields.time"] = data["time"]
}
_, ok = data["msg"]
if ok {
data["fields.msg"] = data["msg"]
}
_, ok = data["level"]
if ok {
data["fields.level"] = data["level"]
}
}

View file

@ -1,88 +0,0 @@
package logrus
import (
"testing"
"time"
)
// smallFields is a small size data set for benchmarking
var smallFields = Fields{
"foo": "bar",
"baz": "qux",
"one": "two",
"three": "four",
}
// largeFields is a large size data set for benchmarking
var largeFields = Fields{
"foo": "bar",
"baz": "qux",
"one": "two",
"three": "four",
"five": "six",
"seven": "eight",
"nine": "ten",
"eleven": "twelve",
"thirteen": "fourteen",
"fifteen": "sixteen",
"seventeen": "eighteen",
"nineteen": "twenty",
"a": "b",
"c": "d",
"e": "f",
"g": "h",
"i": "j",
"k": "l",
"m": "n",
"o": "p",
"q": "r",
"s": "t",
"u": "v",
"w": "x",
"y": "z",
"this": "will",
"make": "thirty",
"entries": "yeah",
}
func BenchmarkSmallTextFormatter(b *testing.B) {
doBenchmark(b, &TextFormatter{DisableColors: true}, smallFields)
}
func BenchmarkLargeTextFormatter(b *testing.B) {
doBenchmark(b, &TextFormatter{DisableColors: true}, largeFields)
}
func BenchmarkSmallColoredTextFormatter(b *testing.B) {
doBenchmark(b, &TextFormatter{ForceColors: true}, smallFields)
}
func BenchmarkLargeColoredTextFormatter(b *testing.B) {
doBenchmark(b, &TextFormatter{ForceColors: true}, largeFields)
}
func BenchmarkSmallJSONFormatter(b *testing.B) {
doBenchmark(b, &JSONFormatter{}, smallFields)
}
func BenchmarkLargeJSONFormatter(b *testing.B) {
doBenchmark(b, &JSONFormatter{}, largeFields)
}
func doBenchmark(b *testing.B, formatter Formatter, fields Fields) {
entry := &Entry{
Time: time.Time{},
Level: InfoLevel,
Message: "message",
Data: fields,
}
var d []byte
var err error
for i := 0; i < b.N; i++ {
d, err = formatter.Format(entry)
if err != nil {
b.Fatal(err)
}
b.SetBytes(int64(len(d)))
}
}

View file

@ -1,56 +0,0 @@
package logstash
import (
"encoding/json"
"fmt"
"github.com/Sirupsen/logrus"
)
// Formatter generates json in logstash format.
// Logstash site: http://logstash.net/
type LogstashFormatter struct {
Type string // if not empty use for logstash type field.
// TimestampFormat sets the format used for timestamps.
TimestampFormat string
}
func (f *LogstashFormatter) Format(entry *logrus.Entry) ([]byte, error) {
entry.Data["@version"] = 1
if f.TimestampFormat == "" {
f.TimestampFormat = logrus.DefaultTimestampFormat
}
entry.Data["@timestamp"] = entry.Time.Format(f.TimestampFormat)
// set message field
v, ok := entry.Data["message"]
if ok {
entry.Data["fields.message"] = v
}
entry.Data["message"] = entry.Message
// set level field
v, ok = entry.Data["level"]
if ok {
entry.Data["fields.level"] = v
}
entry.Data["level"] = entry.Level.String()
// set type field
if f.Type != "" {
v, ok = entry.Data["type"]
if ok {
entry.Data["fields.type"] = v
}
entry.Data["type"] = f.Type
}
serialized, err := json.Marshal(entry.Data)
if err != nil {
return nil, fmt.Errorf("Failed to marshal fields to JSON, %v", err)
}
return append(serialized, '\n'), nil
}

View file

@ -1,52 +0,0 @@
package logstash
import (
"bytes"
"encoding/json"
"github.com/Sirupsen/logrus"
"github.com/stretchr/testify/assert"
"testing"
)
func TestLogstashFormatter(t *testing.T) {
assert := assert.New(t)
lf := LogstashFormatter{Type: "abc"}
fields := logrus.Fields{
"message": "def",
"level": "ijk",
"type": "lmn",
"one": 1,
"pi": 3.14,
"bool": true,
}
entry := logrus.WithFields(fields)
entry.Message = "msg"
entry.Level = logrus.InfoLevel
b, _ := lf.Format(entry)
var data map[string]interface{}
dec := json.NewDecoder(bytes.NewReader(b))
dec.UseNumber()
dec.Decode(&data)
// base fields
assert.Equal(json.Number("1"), data["@version"])
assert.NotEmpty(data["@timestamp"])
assert.Equal("abc", data["type"])
assert.Equal("msg", data["message"])
assert.Equal("info", data["level"])
// substituted fields
assert.Equal("def", data["fields.message"])
assert.Equal("ijk", data["fields.level"])
assert.Equal("lmn", data["fields.type"])
// formats
assert.Equal(json.Number("1"), data["one"])
assert.Equal(json.Number("3.14"), data["pi"])
assert.Equal(true, data["bool"])
}

View file

@ -1,122 +0,0 @@
package logrus
import (
"testing"
"github.com/stretchr/testify/assert"
)
type TestHook struct {
Fired bool
}
func (hook *TestHook) Fire(entry *Entry) error {
hook.Fired = true
return nil
}
func (hook *TestHook) Levels() []Level {
return []Level{
DebugLevel,
InfoLevel,
WarnLevel,
ErrorLevel,
FatalLevel,
PanicLevel,
}
}
func TestHookFires(t *testing.T) {
hook := new(TestHook)
LogAndAssertJSON(t, func(log *Logger) {
log.Hooks.Add(hook)
assert.Equal(t, hook.Fired, false)
log.Print("test")
}, func(fields Fields) {
assert.Equal(t, hook.Fired, true)
})
}
type ModifyHook struct {
}
func (hook *ModifyHook) Fire(entry *Entry) error {
entry.Data["wow"] = "whale"
return nil
}
func (hook *ModifyHook) Levels() []Level {
return []Level{
DebugLevel,
InfoLevel,
WarnLevel,
ErrorLevel,
FatalLevel,
PanicLevel,
}
}
func TestHookCanModifyEntry(t *testing.T) {
hook := new(ModifyHook)
LogAndAssertJSON(t, func(log *Logger) {
log.Hooks.Add(hook)
log.WithField("wow", "elephant").Print("test")
}, func(fields Fields) {
assert.Equal(t, fields["wow"], "whale")
})
}
func TestCanFireMultipleHooks(t *testing.T) {
hook1 := new(ModifyHook)
hook2 := new(TestHook)
LogAndAssertJSON(t, func(log *Logger) {
log.Hooks.Add(hook1)
log.Hooks.Add(hook2)
log.WithField("wow", "elephant").Print("test")
}, func(fields Fields) {
assert.Equal(t, fields["wow"], "whale")
assert.Equal(t, hook2.Fired, true)
})
}
type ErrorHook struct {
Fired bool
}
func (hook *ErrorHook) Fire(entry *Entry) error {
hook.Fired = true
return nil
}
func (hook *ErrorHook) Levels() []Level {
return []Level{
ErrorLevel,
}
}
func TestErrorHookShouldntFireOnInfo(t *testing.T) {
hook := new(ErrorHook)
LogAndAssertJSON(t, func(log *Logger) {
log.Hooks.Add(hook)
log.Info("test")
}, func(fields Fields) {
assert.Equal(t, hook.Fired, false)
})
}
func TestErrorHookShouldFireOnError(t *testing.T) {
hook := new(ErrorHook)
LogAndAssertJSON(t, func(log *Logger) {
log.Hooks.Add(hook)
log.Error("test")
}, func(fields Fields) {
assert.Equal(t, hook.Fired, true)
})
}

View file

@ -1,54 +0,0 @@
package airbrake
import (
"errors"
"fmt"
"github.com/Sirupsen/logrus"
"github.com/tobi/airbrake-go"
)
// AirbrakeHook to send exceptions to an exception-tracking service compatible
// with the Airbrake API.
type airbrakeHook struct {
APIKey string
Endpoint string
Environment string
}
func NewHook(endpoint, apiKey, env string) *airbrakeHook {
return &airbrakeHook{
APIKey: apiKey,
Endpoint: endpoint,
Environment: env,
}
}
func (hook *airbrakeHook) Fire(entry *logrus.Entry) error {
airbrake.ApiKey = hook.APIKey
airbrake.Endpoint = hook.Endpoint
airbrake.Environment = hook.Environment
var notifyErr error
err, ok := entry.Data["error"].(error)
if ok {
notifyErr = err
} else {
notifyErr = errors.New(entry.Message)
}
airErr := airbrake.Notify(notifyErr)
if airErr != nil {
return fmt.Errorf("Failed to send error to Airbrake: %s", airErr)
}
return nil
}
func (hook *airbrakeHook) Levels() []logrus.Level {
return []logrus.Level{
logrus.ErrorLevel,
logrus.FatalLevel,
logrus.PanicLevel,
}
}

View file

@ -1,133 +0,0 @@
package airbrake
import (
"encoding/xml"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/Sirupsen/logrus"
)
type notice struct {
Error NoticeError `xml:"error"`
}
type NoticeError struct {
Class string `xml:"class"`
Message string `xml:"message"`
}
type customErr struct {
msg string
}
func (e *customErr) Error() string {
return e.msg
}
const (
testAPIKey = "abcxyz"
testEnv = "development"
expectedClass = "*airbrake.customErr"
expectedMsg = "foo"
unintendedMsg = "Airbrake will not see this string"
)
var (
noticeError = make(chan NoticeError, 1)
)
// TestLogEntryMessageReceived checks if invoking Logrus' log.Error
// method causes an XML payload containing the log entry message is received
// by a HTTP server emulating an Airbrake-compatible endpoint.
func TestLogEntryMessageReceived(t *testing.T) {
log := logrus.New()
ts := startAirbrakeServer(t)
defer ts.Close()
hook := NewHook(ts.URL, testAPIKey, "production")
log.Hooks.Add(hook)
log.Error(expectedMsg)
select {
case received := <-noticeError:
if received.Message != expectedMsg {
t.Errorf("Unexpected message received: %s", received.Message)
}
case <-time.After(time.Second):
t.Error("Timed out; no notice received by Airbrake API")
}
}
// TestLogEntryMessageReceived confirms that, when passing an error type using
// logrus.Fields, a HTTP server emulating an Airbrake endpoint receives the
// error message returned by the Error() method on the error interface
// rather than the logrus.Entry.Message string.
func TestLogEntryWithErrorReceived(t *testing.T) {
log := logrus.New()
ts := startAirbrakeServer(t)
defer ts.Close()
hook := NewHook(ts.URL, testAPIKey, "production")
log.Hooks.Add(hook)
log.WithFields(logrus.Fields{
"error": &customErr{expectedMsg},
}).Error(unintendedMsg)
select {
case received := <-noticeError:
if received.Message != expectedMsg {
t.Errorf("Unexpected message received: %s", received.Message)
}
if received.Class != expectedClass {
t.Errorf("Unexpected error class: %s", received.Class)
}
case <-time.After(time.Second):
t.Error("Timed out; no notice received by Airbrake API")
}
}
// TestLogEntryWithNonErrorTypeNotReceived confirms that, when passing a
// non-error type using logrus.Fields, a HTTP server emulating an Airbrake
// endpoint receives the logrus.Entry.Message string.
//
// Only error types are supported when setting the 'error' field using
// logrus.WithFields().
func TestLogEntryWithNonErrorTypeNotReceived(t *testing.T) {
log := logrus.New()
ts := startAirbrakeServer(t)
defer ts.Close()
hook := NewHook(ts.URL, testAPIKey, "production")
log.Hooks.Add(hook)
log.WithFields(logrus.Fields{
"error": expectedMsg,
}).Error(unintendedMsg)
select {
case received := <-noticeError:
if received.Message != unintendedMsg {
t.Errorf("Unexpected message received: %s", received.Message)
}
case <-time.After(time.Second):
t.Error("Timed out; no notice received by Airbrake API")
}
}
func startAirbrakeServer(t *testing.T) *httptest.Server {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var notice notice
if err := xml.NewDecoder(r.Body).Decode(&notice); err != nil {
t.Error(err)
}
r.Body.Close()
noticeError <- notice.Error
}))
return ts
}

View file

@ -1,64 +0,0 @@
package logrus_bugsnag
import (
"encoding/json"
"errors"
"io/ioutil"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/Sirupsen/logrus"
"github.com/bugsnag/bugsnag-go"
)
type notice struct {
Events []struct {
Exceptions []struct {
Message string `json:"message"`
} `json:"exceptions"`
} `json:"events"`
}
func TestNoticeReceived(t *testing.T) {
msg := make(chan string, 1)
expectedMsg := "foo"
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var notice notice
data, _ := ioutil.ReadAll(r.Body)
if err := json.Unmarshal(data, &notice); err != nil {
t.Error(err)
}
_ = r.Body.Close()
msg <- notice.Events[0].Exceptions[0].Message
}))
defer ts.Close()
hook := &bugsnagHook{}
bugsnag.Configure(bugsnag.Configuration{
Endpoint: ts.URL,
ReleaseStage: "production",
APIKey: "12345678901234567890123456789012",
Synchronous: true,
})
log := logrus.New()
log.Hooks.Add(hook)
log.WithFields(logrus.Fields{
"error": errors.New(expectedMsg),
}).Error("Bugsnag will not see this string")
select {
case received := <-msg:
if received != expectedMsg {
t.Errorf("Unexpected message received: %s", received)
}
case <-time.After(time.Second):
t.Error("Timed out; no notice received by Bugsnag API")
}
}

View file

@ -1,28 +0,0 @@
# Papertrail Hook for Logrus <img src="http://i.imgur.com/hTeVwmJ.png" width="40" height="40" alt=":walrus:" class="emoji" title=":walrus:" />
[Papertrail](https://papertrailapp.com) provides hosted log management. Once stored in Papertrail, you can [group](http://help.papertrailapp.com/kb/how-it-works/groups/) your logs on various dimensions, [search](http://help.papertrailapp.com/kb/how-it-works/search-syntax) them, and trigger [alerts](http://help.papertrailapp.com/kb/how-it-works/alerts).
In most deployments, you'll want to send logs to Papertrail via their [remote_syslog](http://help.papertrailapp.com/kb/configuration/configuring-centralized-logging-from-text-log-files-in-unix/) daemon, which requires no application-specific configuration. This hook is intended for relatively low-volume logging, likely in managed cloud hosting deployments where installing `remote_syslog` is not possible.
## Usage
You can find your Papertrail UDP port on your [Papertrail account page](https://papertrailapp.com/account/destinations). Substitute it below for `YOUR_PAPERTRAIL_UDP_PORT`.
For `YOUR_APP_NAME`, substitute a short string that will readily identify your application or service in the logs.
```go
import (
"log/syslog"
"github.com/Sirupsen/logrus"
"github.com/Sirupsen/logrus/hooks/papertrail"
)
func main() {
log := logrus.New()
hook, err := logrus_papertrail.NewPapertrailHook("logs.papertrailapp.com", YOUR_PAPERTRAIL_UDP_PORT, YOUR_APP_NAME)
if err == nil {
log.Hooks.Add(hook)
}
}
```

View file

@ -1,55 +0,0 @@
package logrus_papertrail
import (
"fmt"
"net"
"os"
"time"
"github.com/Sirupsen/logrus"
)
const (
format = "Jan 2 15:04:05"
)
// PapertrailHook to send logs to a logging service compatible with the Papertrail API.
type PapertrailHook struct {
Host string
Port int
AppName string
UDPConn net.Conn
}
// NewPapertrailHook creates a hook to be added to an instance of logger.
func NewPapertrailHook(host string, port int, appName string) (*PapertrailHook, error) {
conn, err := net.Dial("udp", fmt.Sprintf("%s:%d", host, port))
return &PapertrailHook{host, port, appName, conn}, err
}
// Fire is called when a log event is fired.
func (hook *PapertrailHook) Fire(entry *logrus.Entry) error {
date := time.Now().Format(format)
msg, _ := entry.String()
payload := fmt.Sprintf("<22> %s %s: %s", date, hook.AppName, msg)
bytesWritten, err := hook.UDPConn.Write([]byte(payload))
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to send log line to Papertrail via UDP. Wrote %d bytes before error: %v", bytesWritten, err)
return err
}
return nil
}
// Levels returns the available logging levels.
func (hook *PapertrailHook) Levels() []logrus.Level {
return []logrus.Level{
logrus.PanicLevel,
logrus.FatalLevel,
logrus.ErrorLevel,
logrus.WarnLevel,
logrus.InfoLevel,
logrus.DebugLevel,
}
}

View file

@ -1,26 +0,0 @@
package logrus_papertrail
import (
"fmt"
"testing"
"github.com/Sirupsen/logrus"
"github.com/stvp/go-udp-testing"
)
func TestWritingToUDP(t *testing.T) {
port := 16661
udp.SetAddr(fmt.Sprintf(":%d", port))
hook, err := NewPapertrailHook("localhost", port, "test")
if err != nil {
t.Errorf("Unable to connect to local UDP server.")
}
log := logrus.New()
log.Hooks.Add(hook)
udp.ShouldReceive(t, "foo", func() {
log.Info("foo")
})
}

View file

@ -1,61 +0,0 @@
# Sentry Hook for Logrus <img src="http://i.imgur.com/hTeVwmJ.png" width="40" height="40" alt=":walrus:" class="emoji" title=":walrus:" />
[Sentry](https://getsentry.com) provides both self-hosted and hosted
solutions for exception tracking.
Both client and server are
[open source](https://github.com/getsentry/sentry).
## Usage
Every sentry application defined on the server gets a different
[DSN](https://www.getsentry.com/docs/). In the example below replace
`YOUR_DSN` with the one created for your application.
```go
import (
"github.com/Sirupsen/logrus"
"github.com/Sirupsen/logrus/hooks/sentry"
)
func main() {
log := logrus.New()
hook, err := logrus_sentry.NewSentryHook(YOUR_DSN, []logrus.Level{
logrus.PanicLevel,
logrus.FatalLevel,
logrus.ErrorLevel,
})
if err == nil {
log.Hooks.Add(hook)
}
}
```
## Special fields
Some logrus fields have a special meaning in this hook,
these are server_name and logger.
When logs are sent to sentry these fields are treated differently.
- server_name (also known as hostname) is the name of the server which
is logging the event (hostname.example.com)
- logger is the part of the application which is logging the event.
In go this usually means setting it to the name of the package.
## Timeout
`Timeout` is the time the sentry hook will wait for a response
from the sentry server.
If this time elapses with no response from
the server an error will be returned.
If `Timeout` is set to 0 the SentryHook will not wait for a reply
and will assume a correct delivery.
The SentryHook has a default timeout of `100 milliseconds` when created
with a call to `NewSentryHook`. This can be changed by assigning a value to the `Timeout` field:
```go
hook, _ := logrus_sentry.NewSentryHook(...)
hook.Timeout = 20*time.Second
```

View file

@ -1,100 +0,0 @@
package logrus_sentry
import (
"fmt"
"time"
"github.com/Sirupsen/logrus"
"github.com/getsentry/raven-go"
)
var (
severityMap = map[logrus.Level]raven.Severity{
logrus.DebugLevel: raven.DEBUG,
logrus.InfoLevel: raven.INFO,
logrus.WarnLevel: raven.WARNING,
logrus.ErrorLevel: raven.ERROR,
logrus.FatalLevel: raven.FATAL,
logrus.PanicLevel: raven.FATAL,
}
)
func getAndDel(d logrus.Fields, key string) (string, bool) {
var (
ok bool
v interface{}
val string
)
if v, ok = d[key]; !ok {
return "", false
}
if val, ok = v.(string); !ok {
return "", false
}
delete(d, key)
return val, true
}
// SentryHook delivers logs to a sentry server.
type SentryHook struct {
// Timeout sets the time to wait for a delivery error from the sentry server.
// If this is set to zero the server will not wait for any response and will
// consider the message correctly sent
Timeout time.Duration
client *raven.Client
levels []logrus.Level
}
// NewSentryHook creates a hook to be added to an instance of logger
// and initializes the raven client.
// This method sets the timeout to 100 milliseconds.
func NewSentryHook(DSN string, levels []logrus.Level) (*SentryHook, error) {
client, err := raven.NewClient(DSN, nil)
if err != nil {
return nil, err
}
return &SentryHook{100 * time.Millisecond, client, levels}, nil
}
// Called when an event should be sent to sentry
// Special fields that sentry uses to give more information to the server
// are extracted from entry.Data (if they are found)
// These fields are: logger and server_name
func (hook *SentryHook) Fire(entry *logrus.Entry) error {
packet := &raven.Packet{
Message: entry.Message,
Timestamp: raven.Timestamp(entry.Time),
Level: severityMap[entry.Level],
Platform: "go",
}
d := entry.Data
if logger, ok := getAndDel(d, "logger"); ok {
packet.Logger = logger
}
if serverName, ok := getAndDel(d, "server_name"); ok {
packet.ServerName = serverName
}
packet.Extra = map[string]interface{}(d)
_, errCh := hook.client.Capture(packet, nil)
timeout := hook.Timeout
if timeout != 0 {
timeoutCh := time.After(timeout)
select {
case err := <-errCh:
return err
case <-timeoutCh:
return fmt.Errorf("no response from sentry server in %s", timeout)
}
}
return nil
}
// Levels returns the available logging levels.
func (hook *SentryHook) Levels() []logrus.Level {
return hook.levels
}

View file

@ -1,97 +0,0 @@
package logrus_sentry
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/Sirupsen/logrus"
"github.com/getsentry/raven-go"
)
const (
message = "error message"
server_name = "testserver.internal"
logger_name = "test.logger"
)
func getTestLogger() *logrus.Logger {
l := logrus.New()
l.Out = ioutil.Discard
return l
}
func WithTestDSN(t *testing.T, tf func(string, <-chan *raven.Packet)) {
pch := make(chan *raven.Packet, 1)
s := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
defer req.Body.Close()
d := json.NewDecoder(req.Body)
p := &raven.Packet{}
err := d.Decode(p)
if err != nil {
t.Fatal(err.Error())
}
pch <- p
}))
defer s.Close()
fragments := strings.SplitN(s.URL, "://", 2)
dsn := fmt.Sprintf(
"%s://public:secret@%s/sentry/project-id",
fragments[0],
fragments[1],
)
tf(dsn, pch)
}
func TestSpecialFields(t *testing.T) {
WithTestDSN(t, func(dsn string, pch <-chan *raven.Packet) {
logger := getTestLogger()
hook, err := NewSentryHook(dsn, []logrus.Level{
logrus.ErrorLevel,
})
if err != nil {
t.Fatal(err.Error())
}
logger.Hooks.Add(hook)
logger.WithFields(logrus.Fields{
"server_name": server_name,
"logger": logger_name,
}).Error(message)
packet := <-pch
if packet.Logger != logger_name {
t.Errorf("logger should have been %s, was %s", logger_name, packet.Logger)
}
if packet.ServerName != server_name {
t.Errorf("server_name should have been %s, was %s", server_name, packet.ServerName)
}
})
}
func TestSentryHandler(t *testing.T) {
WithTestDSN(t, func(dsn string, pch <-chan *raven.Packet) {
logger := getTestLogger()
hook, err := NewSentryHook(dsn, []logrus.Level{
logrus.ErrorLevel,
})
if err != nil {
t.Fatal(err.Error())
}
logger.Hooks.Add(hook)
logger.Error(message)
packet := <-pch
if packet.Message != message {
t.Errorf("message should have been %s, was %s", message, packet.Message)
}
})
}

View file

@ -1,20 +0,0 @@
# Syslog Hooks for Logrus <img src="http://i.imgur.com/hTeVwmJ.png" width="40" height="40" alt=":walrus:" class="emoji" title=":walrus:"/>
## Usage
```go
import (
"log/syslog"
"github.com/Sirupsen/logrus"
logrus_syslog "github.com/Sirupsen/logrus/hooks/syslog"
)
func main() {
log := logrus.New()
hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
if err == nil {
log.Hooks.Add(hook)
}
}
```

View file

@ -1,59 +0,0 @@
package logrus_syslog
import (
"fmt"
"github.com/Sirupsen/logrus"
"log/syslog"
"os"
)
// SyslogHook to send logs via syslog.
type SyslogHook struct {
Writer *syslog.Writer
SyslogNetwork string
SyslogRaddr string
}
// Creates a hook to be added to an instance of logger. This is called with
// `hook, err := NewSyslogHook("udp", "localhost:514", syslog.LOG_DEBUG, "")`
// `if err == nil { log.Hooks.Add(hook) }`
func NewSyslogHook(network, raddr string, priority syslog.Priority, tag string) (*SyslogHook, error) {
w, err := syslog.Dial(network, raddr, priority, tag)
return &SyslogHook{w, network, raddr}, err
}
func (hook *SyslogHook) Fire(entry *logrus.Entry) error {
line, err := entry.String()
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to read entry, %v", err)
return err
}
switch entry.Level {
case logrus.PanicLevel:
return hook.Writer.Crit(line)
case logrus.FatalLevel:
return hook.Writer.Crit(line)
case logrus.ErrorLevel:
return hook.Writer.Err(line)
case logrus.WarnLevel:
return hook.Writer.Warning(line)
case logrus.InfoLevel:
return hook.Writer.Info(line)
case logrus.DebugLevel:
return hook.Writer.Debug(line)
default:
return nil
}
}
func (hook *SyslogHook) Levels() []logrus.Level {
return []logrus.Level{
logrus.PanicLevel,
logrus.FatalLevel,
logrus.ErrorLevel,
logrus.WarnLevel,
logrus.InfoLevel,
logrus.DebugLevel,
}
}

View file

@ -1,26 +0,0 @@
package logrus_syslog
import (
"github.com/Sirupsen/logrus"
"log/syslog"
"testing"
)
func TestLocalhostAddAndPrint(t *testing.T) {
log := logrus.New()
hook, err := NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
if err != nil {
t.Errorf("Unable to connect to local syslog.")
}
log.Hooks.Add(hook)
for _, level := range hook.Levels() {
if len(log.Hooks[level]) != 1 {
t.Errorf("SyslogHook was not added. The length of log.Hooks[%v]: %v", level, len(log.Hooks[level]))
}
}
log.Info("Congratulations!")
}

View file

@ -1,40 +0,0 @@
package logrus
import (
"encoding/json"
"fmt"
)
type JSONFormatter struct {
// TimestampFormat sets the format used for marshaling timestamps.
TimestampFormat string
}
func (f *JSONFormatter) Format(entry *Entry) ([]byte, error) {
data := make(Fields, len(entry.Data)+3)
for k, v := range entry.Data {
switch v := v.(type) {
case error:
// Otherwise errors are ignored by `encoding/json`
// https://github.com/Sirupsen/logrus/issues/137
data[k] = v.Error()
default:
data[k] = v
}
}
prefixFieldClashes(data)
if f.TimestampFormat == "" {
f.TimestampFormat = DefaultTimestampFormat
}
data["time"] = entry.Time.Format(f.TimestampFormat)
data["msg"] = entry.Message
data["level"] = entry.Level.String()
serialized, err := json.Marshal(data)
if err != nil {
return nil, fmt.Errorf("Failed to marshal fields to JSON, %v", err)
}
return append(serialized, '\n'), nil
}

View file

@ -1,120 +0,0 @@
package logrus
import (
"encoding/json"
"errors"
"testing"
)
func TestErrorNotLost(t *testing.T) {
formatter := &JSONFormatter{}
b, err := formatter.Format(WithField("error", errors.New("wild walrus")))
if err != nil {
t.Fatal("Unable to format entry: ", err)
}
entry := make(map[string]interface{})
err = json.Unmarshal(b, &entry)
if err != nil {
t.Fatal("Unable to unmarshal formatted entry: ", err)
}
if entry["error"] != "wild walrus" {
t.Fatal("Error field not set")
}
}
func TestErrorNotLostOnFieldNotNamedError(t *testing.T) {
formatter := &JSONFormatter{}
b, err := formatter.Format(WithField("omg", errors.New("wild walrus")))
if err != nil {
t.Fatal("Unable to format entry: ", err)
}
entry := make(map[string]interface{})
err = json.Unmarshal(b, &entry)
if err != nil {
t.Fatal("Unable to unmarshal formatted entry: ", err)
}
if entry["omg"] != "wild walrus" {
t.Fatal("Error field not set")
}
}
func TestFieldClashWithTime(t *testing.T) {
formatter := &JSONFormatter{}
b, err := formatter.Format(WithField("time", "right now!"))
if err != nil {
t.Fatal("Unable to format entry: ", err)
}
entry := make(map[string]interface{})
err = json.Unmarshal(b, &entry)
if err != nil {
t.Fatal("Unable to unmarshal formatted entry: ", err)
}
if entry["fields.time"] != "right now!" {
t.Fatal("fields.time not set to original time field")
}
if entry["time"] != "0001-01-01T00:00:00Z" {
t.Fatal("time field not set to current time, was: ", entry["time"])
}
}
func TestFieldClashWithMsg(t *testing.T) {
formatter := &JSONFormatter{}
b, err := formatter.Format(WithField("msg", "something"))
if err != nil {
t.Fatal("Unable to format entry: ", err)
}
entry := make(map[string]interface{})
err = json.Unmarshal(b, &entry)
if err != nil {
t.Fatal("Unable to unmarshal formatted entry: ", err)
}
if entry["fields.msg"] != "something" {
t.Fatal("fields.msg not set to original msg field")
}
}
func TestFieldClashWithLevel(t *testing.T) {
formatter := &JSONFormatter{}
b, err := formatter.Format(WithField("level", "something"))
if err != nil {
t.Fatal("Unable to format entry: ", err)
}
entry := make(map[string]interface{})
err = json.Unmarshal(b, &entry)
if err != nil {
t.Fatal("Unable to unmarshal formatted entry: ", err)
}
if entry["fields.level"] != "something" {
t.Fatal("fields.level not set to original level field")
}
}
func TestJSONEntryEndsWithNewline(t *testing.T) {
formatter := &JSONFormatter{}
b, err := formatter.Format(WithField("level", "something"))
if err != nil {
t.Fatal("Unable to format entry: ", err)
}
if b[len(b)-1] != '\n' {
t.Fatal("Expected JSON log entry to end with a newline")
}
}

View file

@ -1,203 +0,0 @@
package logrus
import (
"io"
"os"
"sync"
)
type Logger struct {
// The logs are `io.Copy`'d to this in a mutex. It's common to set this to a
// file, or leave it default which is `os.Stdout`. You can also set this to
// something more adventorous, such as logging to Kafka.
Out io.Writer
// Hooks for the logger instance. These allow firing events based on logging
// levels and log entries. For example, to send errors to an error tracking
// service, log to StatsD or dump the core on fatal errors.
Hooks levelHooks
// All log entries pass through the formatter before logged to Out. The
// included formatters are `TextFormatter` and `JSONFormatter` for which
// TextFormatter is the default. In development (when a TTY is attached) it
// logs with colors, but to a file it wouldn't. You can easily implement your
// own that implements the `Formatter` interface, see the `README` or included
// formatters for examples.
Formatter Formatter
// The logging level the logger should log at. This is typically (and defaults
// to) `logrus.Info`, which allows Info(), Warn(), Error() and Fatal() to be
// logged. `logrus.Debug` is useful in
Level Level
// Used to sync writing to the log.
mu sync.Mutex
}
// Creates a new logger. Configuration should be set by changing `Formatter`,
// `Out` and `Hooks` directly on the default logger instance. You can also just
// instantiate your own:
//
// var log = &Logger{
// Out: os.Stderr,
// Formatter: new(JSONFormatter),
// Hooks: make(levelHooks),
// Level: logrus.DebugLevel,
// }
//
// It's recommended to make this a global instance called `log`.
func New() *Logger {
return &Logger{
Out: os.Stdout,
Formatter: new(TextFormatter),
Hooks: make(levelHooks),
Level: InfoLevel,
}
}
// Adds a field to the log entry, note that you it doesn't log until you call
// Debug, Print, Info, Warn, Fatal or Panic. It only creates a log entry.
// Ff you want multiple fields, use `WithFields`.
func (logger *Logger) WithField(key string, value interface{}) *Entry {
return NewEntry(logger).WithField(key, value)
}
// Adds a struct of fields to the log entry. All it does is call `WithField` for
// each `Field`.
func (logger *Logger) WithFields(fields Fields) *Entry {
return NewEntry(logger).WithFields(fields)
}
func (logger *Logger) Debugf(format string, args ...interface{}) {
if logger.Level >= DebugLevel {
NewEntry(logger).Debugf(format, args...)
}
}
func (logger *Logger) Infof(format string, args ...interface{}) {
if logger.Level >= InfoLevel {
NewEntry(logger).Infof(format, args...)
}
}
func (logger *Logger) Printf(format string, args ...interface{}) {
NewEntry(logger).Printf(format, args...)
}
func (logger *Logger) Warnf(format string, args ...interface{}) {
if logger.Level >= WarnLevel {
NewEntry(logger).Warnf(format, args...)
}
}
func (logger *Logger) Warningf(format string, args ...interface{}) {
if logger.Level >= WarnLevel {
NewEntry(logger).Warnf(format, args...)
}
}
func (logger *Logger) Errorf(format string, args ...interface{}) {
if logger.Level >= ErrorLevel {
NewEntry(logger).Errorf(format, args...)
}
}
func (logger *Logger) Fatalf(format string, args ...interface{}) {
if logger.Level >= FatalLevel {
NewEntry(logger).Fatalf(format, args...)
}
}
func (logger *Logger) Panicf(format string, args ...interface{}) {
if logger.Level >= PanicLevel {
NewEntry(logger).Panicf(format, args...)
}
}
func (logger *Logger) Debug(args ...interface{}) {
if logger.Level >= DebugLevel {
NewEntry(logger).Debug(args...)
}
}
func (logger *Logger) Info(args ...interface{}) {
if logger.Level >= InfoLevel {
NewEntry(logger).Info(args...)
}
}
func (logger *Logger) Print(args ...interface{}) {
NewEntry(logger).Info(args...)
}
func (logger *Logger) Warn(args ...interface{}) {
if logger.Level >= WarnLevel {
NewEntry(logger).Warn(args...)
}
}
func (logger *Logger) Warning(args ...interface{}) {
if logger.Level >= WarnLevel {
NewEntry(logger).Warn(args...)
}
}
func (logger *Logger) Error(args ...interface{}) {
if logger.Level >= ErrorLevel {
NewEntry(logger).Error(args...)
}
}
func (logger *Logger) Fatal(args ...interface{}) {
if logger.Level >= FatalLevel {
NewEntry(logger).Fatal(args...)
}
}
func (logger *Logger) Panic(args ...interface{}) {
if logger.Level >= PanicLevel {
NewEntry(logger).Panic(args...)
}
}
func (logger *Logger) Debugln(args ...interface{}) {
if logger.Level >= DebugLevel {
NewEntry(logger).Debugln(args...)
}
}
func (logger *Logger) Infoln(args ...interface{}) {
if logger.Level >= InfoLevel {
NewEntry(logger).Infoln(args...)
}
}
func (logger *Logger) Println(args ...interface{}) {
NewEntry(logger).Println(args...)
}
func (logger *Logger) Warnln(args ...interface{}) {
if logger.Level >= WarnLevel {
NewEntry(logger).Warnln(args...)
}
}
func (logger *Logger) Warningln(args ...interface{}) {
if logger.Level >= WarnLevel {
NewEntry(logger).Warnln(args...)
}
}
func (logger *Logger) Errorln(args ...interface{}) {
if logger.Level >= ErrorLevel {
NewEntry(logger).Errorln(args...)
}
}
func (logger *Logger) Fatalln(args ...interface{}) {
if logger.Level >= FatalLevel {
NewEntry(logger).Fatalln(args...)
}
}
func (logger *Logger) Panicln(args ...interface{}) {
if logger.Level >= PanicLevel {
NewEntry(logger).Panicln(args...)
}
}

View file

@ -1,94 +0,0 @@
package logrus
import (
"fmt"
"log"
)
// Fields type, used to pass to `WithFields`.
type Fields map[string]interface{}
// Level type
type Level uint8
// Convert the Level to a string. E.g. PanicLevel becomes "panic".
func (level Level) String() string {
switch level {
case DebugLevel:
return "debug"
case InfoLevel:
return "info"
case WarnLevel:
return "warning"
case ErrorLevel:
return "error"
case FatalLevel:
return "fatal"
case PanicLevel:
return "panic"
}
return "unknown"
}
// ParseLevel takes a string level and returns the Logrus log level constant.
func ParseLevel(lvl string) (Level, error) {
switch lvl {
case "panic":
return PanicLevel, nil
case "fatal":
return FatalLevel, nil
case "error":
return ErrorLevel, nil
case "warn", "warning":
return WarnLevel, nil
case "info":
return InfoLevel, nil
case "debug":
return DebugLevel, nil
}
var l Level
return l, fmt.Errorf("not a valid logrus Level: %q", lvl)
}
// These are the different logging levels. You can set the logging level to log
// on your instance of logger, obtained with `logrus.New()`.
const (
// PanicLevel level, highest level of severity. Logs and then calls panic with the
// message passed to Debug, Info, ...
PanicLevel Level = iota
// FatalLevel level. Logs and then calls `os.Exit(1)`. It will exit even if the
// logging level is set to Panic.
FatalLevel
// ErrorLevel level. Logs. Used for errors that should definitely be noted.
// Commonly used for hooks to send errors to an error tracking service.
ErrorLevel
// WarnLevel level. Non-critical entries that deserve eyes.
WarnLevel
// InfoLevel level. General operational entries about what's going on inside the
// application.
InfoLevel
// DebugLevel level. Usually only enabled when debugging. Very verbose logging.
DebugLevel
)
// Won't compile if StdLogger can't be realized by a log.Logger
var _ StdLogger = &log.Logger{}
// StdLogger is what your logrus-enabled library should take, that way
// it'll accept a stdlib logger and a logrus logger. There's no standard
// interface, this is the closest we get, unfortunately.
type StdLogger interface {
Print(...interface{})
Printf(string, ...interface{})
Println(...interface{})
Fatal(...interface{})
Fatalf(string, ...interface{})
Fatalln(...interface{})
Panic(...interface{})
Panicf(string, ...interface{})
Panicln(...interface{})
}

View file

@ -1,301 +0,0 @@
package logrus
import (
"bytes"
"encoding/json"
"strconv"
"strings"
"sync"
"testing"
"github.com/stretchr/testify/assert"
)
func LogAndAssertJSON(t *testing.T, log func(*Logger), assertions func(fields Fields)) {
var buffer bytes.Buffer
var fields Fields
logger := New()
logger.Out = &buffer
logger.Formatter = new(JSONFormatter)
log(logger)
err := json.Unmarshal(buffer.Bytes(), &fields)
assert.Nil(t, err)
assertions(fields)
}
func LogAndAssertText(t *testing.T, log func(*Logger), assertions func(fields map[string]string)) {
var buffer bytes.Buffer
logger := New()
logger.Out = &buffer
logger.Formatter = &TextFormatter{
DisableColors: true,
}
log(logger)
fields := make(map[string]string)
for _, kv := range strings.Split(buffer.String(), " ") {
if !strings.Contains(kv, "=") {
continue
}
kvArr := strings.Split(kv, "=")
key := strings.TrimSpace(kvArr[0])
val := kvArr[1]
if kvArr[1][0] == '"' {
var err error
val, err = strconv.Unquote(val)
assert.NoError(t, err)
}
fields[key] = val
}
assertions(fields)
}
func TestPrint(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Print("test")
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test")
assert.Equal(t, fields["level"], "info")
})
}
func TestInfo(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Info("test")
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test")
assert.Equal(t, fields["level"], "info")
})
}
func TestWarn(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Warn("test")
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test")
assert.Equal(t, fields["level"], "warning")
})
}
func TestInfolnShouldAddSpacesBetweenStrings(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Infoln("test", "test")
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test test")
})
}
func TestInfolnShouldAddSpacesBetweenStringAndNonstring(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Infoln("test", 10)
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test 10")
})
}
func TestInfolnShouldAddSpacesBetweenTwoNonStrings(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Infoln(10, 10)
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "10 10")
})
}
func TestInfoShouldAddSpacesBetweenTwoNonStrings(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Infoln(10, 10)
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "10 10")
})
}
func TestInfoShouldNotAddSpacesBetweenStringAndNonstring(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Info("test", 10)
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test10")
})
}
func TestInfoShouldNotAddSpacesBetweenStrings(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.Info("test", "test")
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "testtest")
})
}
func TestWithFieldsShouldAllowAssignments(t *testing.T) {
var buffer bytes.Buffer
var fields Fields
logger := New()
logger.Out = &buffer
logger.Formatter = new(JSONFormatter)
localLog := logger.WithFields(Fields{
"key1": "value1",
})
localLog.WithField("key2", "value2").Info("test")
err := json.Unmarshal(buffer.Bytes(), &fields)
assert.Nil(t, err)
assert.Equal(t, "value2", fields["key2"])
assert.Equal(t, "value1", fields["key1"])
buffer = bytes.Buffer{}
fields = Fields{}
localLog.Info("test")
err = json.Unmarshal(buffer.Bytes(), &fields)
assert.Nil(t, err)
_, ok := fields["key2"]
assert.Equal(t, false, ok)
assert.Equal(t, "value1", fields["key1"])
}
func TestUserSuppliedFieldDoesNotOverwriteDefaults(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.WithField("msg", "hello").Info("test")
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test")
})
}
func TestUserSuppliedMsgFieldHasPrefix(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.WithField("msg", "hello").Info("test")
}, func(fields Fields) {
assert.Equal(t, fields["msg"], "test")
assert.Equal(t, fields["fields.msg"], "hello")
})
}
func TestUserSuppliedTimeFieldHasPrefix(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.WithField("time", "hello").Info("test")
}, func(fields Fields) {
assert.Equal(t, fields["fields.time"], "hello")
})
}
func TestUserSuppliedLevelFieldHasPrefix(t *testing.T) {
LogAndAssertJSON(t, func(log *Logger) {
log.WithField("level", 1).Info("test")
}, func(fields Fields) {
assert.Equal(t, fields["level"], "info")
assert.Equal(t, fields["fields.level"], 1)
})
}
func TestDefaultFieldsAreNotPrefixed(t *testing.T) {
LogAndAssertText(t, func(log *Logger) {
ll := log.WithField("herp", "derp")
ll.Info("hello")
ll.Info("bye")
}, func(fields map[string]string) {
for _, fieldName := range []string{"fields.level", "fields.time", "fields.msg"} {
if _, ok := fields[fieldName]; ok {
t.Fatalf("should not have prefixed %q: %v", fieldName, fields)
}
}
})
}
func TestDoubleLoggingDoesntPrefixPreviousFields(t *testing.T) {
var buffer bytes.Buffer
var fields Fields
logger := New()
logger.Out = &buffer
logger.Formatter = new(JSONFormatter)
llog := logger.WithField("context", "eating raw fish")
llog.Info("looks delicious")
err := json.Unmarshal(buffer.Bytes(), &fields)
assert.NoError(t, err, "should have decoded first message")
assert.Equal(t, len(fields), 4, "should only have msg/time/level/context fields")
assert.Equal(t, fields["msg"], "looks delicious")
assert.Equal(t, fields["context"], "eating raw fish")
buffer.Reset()
llog.Warn("omg it is!")
err = json.Unmarshal(buffer.Bytes(), &fields)
assert.NoError(t, err, "should have decoded second message")
assert.Equal(t, len(fields), 4, "should only have msg/time/level/context fields")
assert.Equal(t, fields["msg"], "omg it is!")
assert.Equal(t, fields["context"], "eating raw fish")
assert.Nil(t, fields["fields.msg"], "should not have prefixed previous `msg` entry")
}
func TestConvertLevelToString(t *testing.T) {
assert.Equal(t, "debug", DebugLevel.String())
assert.Equal(t, "info", InfoLevel.String())
assert.Equal(t, "warning", WarnLevel.String())
assert.Equal(t, "error", ErrorLevel.String())
assert.Equal(t, "fatal", FatalLevel.String())
assert.Equal(t, "panic", PanicLevel.String())
}
func TestParseLevel(t *testing.T) {
l, err := ParseLevel("panic")
assert.Nil(t, err)
assert.Equal(t, PanicLevel, l)
l, err = ParseLevel("fatal")
assert.Nil(t, err)
assert.Equal(t, FatalLevel, l)
l, err = ParseLevel("error")
assert.Nil(t, err)
assert.Equal(t, ErrorLevel, l)
l, err = ParseLevel("warn")
assert.Nil(t, err)
assert.Equal(t, WarnLevel, l)
l, err = ParseLevel("warning")
assert.Nil(t, err)
assert.Equal(t, WarnLevel, l)
l, err = ParseLevel("info")
assert.Nil(t, err)
assert.Equal(t, InfoLevel, l)
l, err = ParseLevel("debug")
assert.Nil(t, err)
assert.Equal(t, DebugLevel, l)
l, err = ParseLevel("invalid")
assert.Equal(t, "not a valid logrus Level: \"invalid\"", err.Error())
}
func TestGetSetLevelRace(t *testing.T) {
wg := sync.WaitGroup{}
for i := 0; i < 100; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
if i%2 == 0 {
SetLevel(InfoLevel)
} else {
GetLevel()
}
}(i)
}
wg.Wait()
}

View file

@ -1,12 +0,0 @@
// Based on ssh/terminal:
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package logrus
import "syscall"
const ioctlReadTermios = syscall.TIOCGETA
type Termios syscall.Termios

View file

@ -1,20 +0,0 @@
/*
Go 1.2 doesn't include Termios for FreeBSD. This should be added in 1.3 and this could be merged with terminal_darwin.
*/
package logrus
import (
"syscall"
)
const ioctlReadTermios = syscall.TIOCGETA
type Termios struct {
Iflag uint32
Oflag uint32
Cflag uint32
Lflag uint32
Cc [20]uint8
Ispeed uint32
Ospeed uint32
}

View file

@ -1,12 +0,0 @@
// Based on ssh/terminal:
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package logrus
import "syscall"
const ioctlReadTermios = syscall.TCGETS
type Termios syscall.Termios

View file

@ -1,21 +0,0 @@
// Based on ssh/terminal:
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build linux darwin freebsd openbsd
package logrus
import (
"syscall"
"unsafe"
)
// IsTerminal returns true if the given file descriptor is a terminal.
func IsTerminal() bool {
fd := syscall.Stdout
var termios Termios
_, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlReadTermios, uintptr(unsafe.Pointer(&termios)), 0, 0, 0)
return err == 0
}

View file

@ -1,7 +0,0 @@
package logrus
import "syscall"
const ioctlReadTermios = syscall.TIOCGETA
type Termios syscall.Termios

View file

@ -1,27 +0,0 @@
// Based on ssh/terminal:
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build windows
package logrus
import (
"syscall"
"unsafe"
)
var kernel32 = syscall.NewLazyDLL("kernel32.dll")
var (
procGetConsoleMode = kernel32.NewProc("GetConsoleMode")
)
// IsTerminal returns true if the given file descriptor is a terminal.
func IsTerminal() bool {
fd := syscall.Stdout
var st uint32
r, _, e := syscall.Syscall(procGetConsoleMode.Addr(), 2, uintptr(fd), uintptr(unsafe.Pointer(&st)), 0)
return r != 0 && e == 0
}

View file

@ -1,149 +0,0 @@
package logrus
import (
"bytes"
"fmt"
"sort"
"strings"
"time"
)
const (
nocolor = 0
red = 31
green = 32
yellow = 33
blue = 34
gray = 37
)
var (
baseTimestamp time.Time
isTerminal bool
)
func init() {
baseTimestamp = time.Now()
isTerminal = IsTerminal()
}
func miniTS() int {
return int(time.Since(baseTimestamp) / time.Second)
}
type TextFormatter struct {
// Set to true to bypass checking for a TTY before outputting colors.
ForceColors bool
// Force disabling colors.
DisableColors bool
// Disable timestamp logging. useful when output is redirected to logging
// system that already adds timestamps.
DisableTimestamp bool
// Enable logging the full timestamp when a TTY is attached instead of just
// the time passed since beginning of execution.
FullTimestamp bool
// TimestampFormat to use for display when a full timestamp is printed
TimestampFormat string
// The fields are sorted by default for a consistent output. For applications
// that log extremely frequently and don't use the JSON formatter this may not
// be desired.
DisableSorting bool
}
func (f *TextFormatter) Format(entry *Entry) ([]byte, error) {
var keys []string = make([]string, 0, len(entry.Data))
for k := range entry.Data {
keys = append(keys, k)
}
if !f.DisableSorting {
sort.Strings(keys)
}
b := &bytes.Buffer{}
prefixFieldClashes(entry.Data)
isColored := (f.ForceColors || isTerminal) && !f.DisableColors
if f.TimestampFormat == "" {
f.TimestampFormat = DefaultTimestampFormat
}
if isColored {
f.printColored(b, entry, keys)
} else {
if !f.DisableTimestamp {
f.appendKeyValue(b, "time", entry.Time.Format(f.TimestampFormat))
}
f.appendKeyValue(b, "level", entry.Level.String())
f.appendKeyValue(b, "msg", entry.Message)
for _, key := range keys {
f.appendKeyValue(b, key, entry.Data[key])
}
}
b.WriteByte('\n')
return b.Bytes(), nil
}
func (f *TextFormatter) printColored(b *bytes.Buffer, entry *Entry, keys []string) {
var levelColor int
switch entry.Level {
case DebugLevel:
levelColor = gray
case WarnLevel:
levelColor = yellow
case ErrorLevel, FatalLevel, PanicLevel:
levelColor = red
default:
levelColor = blue
}
levelText := strings.ToUpper(entry.Level.String())[0:4]
if !f.FullTimestamp {
fmt.Fprintf(b, "\x1b[%dm%s\x1b[0m[%04d] %-44s ", levelColor, levelText, miniTS(), entry.Message)
} else {
fmt.Fprintf(b, "\x1b[%dm%s\x1b[0m[%s] %-44s ", levelColor, levelText, entry.Time.Format(f.TimestampFormat), entry.Message)
}
for _, k := range keys {
v := entry.Data[k]
fmt.Fprintf(b, " \x1b[%dm%s\x1b[0m=%v", levelColor, k, v)
}
}
func needsQuoting(text string) bool {
for _, ch := range text {
if !((ch >= 'a' && ch <= 'z') ||
(ch >= 'A' && ch <= 'Z') ||
(ch >= '0' && ch <= '9') ||
ch == '-' || ch == '.') {
return false
}
}
return true
}
func (f *TextFormatter) appendKeyValue(b *bytes.Buffer, key, value interface{}) {
switch value.(type) {
case string:
if needsQuoting(value.(string)) {
fmt.Fprintf(b, "%v=%s ", key, value)
} else {
fmt.Fprintf(b, "%v=%q ", key, value)
}
case error:
if needsQuoting(value.(error).Error()) {
fmt.Fprintf(b, "%v=%s ", key, value)
} else {
fmt.Fprintf(b, "%v=%q ", key, value)
}
default:
fmt.Fprintf(b, "%v=%v ", key, value)
}
}

View file

@ -1,61 +0,0 @@
package logrus
import (
"bytes"
"errors"
"testing"
"time"
)
func TestQuoting(t *testing.T) {
tf := &TextFormatter{DisableColors: true}
checkQuoting := func(q bool, value interface{}) {
b, _ := tf.Format(WithField("test", value))
idx := bytes.Index(b, ([]byte)("test="))
cont := bytes.Contains(b[idx+5:], []byte{'"'})
if cont != q {
if q {
t.Errorf("quoting expected for: %#v", value)
} else {
t.Errorf("quoting not expected for: %#v", value)
}
}
}
checkQuoting(false, "abcd")
checkQuoting(false, "v1.0")
checkQuoting(false, "1234567890")
checkQuoting(true, "/foobar")
checkQuoting(true, "x y")
checkQuoting(true, "x,y")
checkQuoting(false, errors.New("invalid"))
checkQuoting(true, errors.New("invalid argument"))
}
func TestTimestampFormat(t *testing.T) {
checkTimeStr := func(format string) {
customFormatter := &TextFormatter{DisableColors: true, TimestampFormat: format}
customStr, _ := customFormatter.Format(WithField("test", "test"))
timeStart := bytes.Index(customStr, ([]byte)("time="))
timeEnd := bytes.Index(customStr, ([]byte)("level="))
timeStr := customStr[timeStart+5 : timeEnd-1]
if timeStr[0] == '"' && timeStr[len(timeStr)-1] == '"' {
timeStr = timeStr[1 : len(timeStr)-1]
}
if format == "" {
format = time.RFC3339
}
_, e := time.Parse(format, (string)(timeStr))
if e != nil {
t.Errorf("time string \"%s\" did not match provided time format \"%s\": %s", timeStr, format, e)
}
}
checkTimeStr("2006-01-02T15:04:05.000000000Z07:00")
checkTimeStr("Mon Jan _2 15:04:05 2006")
checkTimeStr("")
}
// TODO add tests for sorting etc., this requires a parser for the text
// formatter output.

View file

@ -1,31 +0,0 @@
package logrus
import (
"bufio"
"io"
"runtime"
)
func (logger *Logger) Writer() *io.PipeWriter {
reader, writer := io.Pipe()
go logger.writerScanner(reader)
runtime.SetFinalizer(writer, writerFinalizer)
return writer
}
func (logger *Logger) writerScanner(reader *io.PipeReader) {
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
logger.Print(scanner.Text())
}
if err := scanner.Err(); err != nil {
logger.Errorf("Error while reading from Writer: %s", err)
}
reader.Close()
}
func writerFinalizer(writer *io.PipeWriter) {
writer.Close()
}

View file

@ -1,461 +0,0 @@
package bugsnag
import (
"fmt"
"io/ioutil"
"log"
"net"
"net/http"
"strings"
"sync"
"testing"
"time"
"github.com/bitly/go-simplejson"
)
func TestConfigure(t *testing.T) {
Configure(Configuration{
APIKey: testAPIKey,
})
if Config.APIKey != testAPIKey {
t.Errorf("Setting APIKey didn't work")
}
if New().Config.APIKey != testAPIKey {
t.Errorf("Setting APIKey didn't work for new notifiers")
}
}
var postedJSON = make(chan []byte, 10)
var testOnce sync.Once
var testEndpoint string
var testAPIKey = "166f5ad3590596f9aa8d601ea89af845"
func startTestServer() {
testOnce.Do(func() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
postedJSON <- body
})
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
panic(err)
}
testEndpoint = "http://" + l.Addr().String() + "/"
go http.Serve(l, mux)
})
}
type _recurse struct {
*_recurse
}
func TestNotify(t *testing.T) {
startTestServer()
recurse := _recurse{}
recurse._recurse = &recurse
OnBeforeNotify(func(event *Event, config *Configuration) error {
if event.Context == "testing" {
event.GroupingHash = "lol"
}
return nil
})
Notify(fmt.Errorf("hello world"),
Configuration{
APIKey: testAPIKey,
Endpoint: testEndpoint,
ReleaseStage: "test",
AppVersion: "1.2.3",
Hostname: "web1",
ProjectPackages: []string{"github.com/bugsnag/bugsnag-go"},
},
User{Id: "123", Name: "Conrad", Email: "me@cirw.in"},
Context{"testing"},
MetaData{"test": {
"password": "sneaky",
"value": "able",
"broken": complex(1, 2),
"recurse": recurse,
}},
)
json, err := simplejson.NewJson(<-postedJSON)
if err != nil {
t.Fatal(err)
}
if json.Get("apiKey").MustString() != testAPIKey {
t.Errorf("Wrong api key in payload")
}
if json.GetPath("notifier", "name").MustString() != "Bugsnag Go" {
t.Errorf("Wrong notifier name in payload")
}
event := json.Get("events").GetIndex(0)
for k, value := range map[string]string{
"payloadVersion": "2",
"severity": "warning",
"context": "testing",
"groupingHash": "lol",
"app.releaseStage": "test",
"app.version": "1.2.3",
"device.hostname": "web1",
"user.id": "123",
"user.name": "Conrad",
"user.email": "me@cirw.in",
"metaData.test.password": "[REDACTED]",
"metaData.test.value": "able",
"metaData.test.broken": "[complex128]",
"metaData.test.recurse._recurse": "[RECURSION]",
} {
key := strings.Split(k, ".")
if event.GetPath(key...).MustString() != value {
t.Errorf("Wrong %v: %v != %v", key, event.GetPath(key...).MustString(), value)
}
}
exception := event.Get("exceptions").GetIndex(0)
if exception.Get("message").MustString() != "hello world" {
t.Errorf("Wrong message in payload")
}
if exception.Get("errorClass").MustString() != "*errors.errorString" {
t.Errorf("Wrong errorClass in payload: %v", exception.Get("errorClass").MustString())
}
frame0 := exception.Get("stacktrace").GetIndex(0)
if frame0.Get("file").MustString() != "bugsnag_test.go" ||
frame0.Get("method").MustString() != "TestNotify" ||
frame0.Get("inProject").MustBool() != true ||
frame0.Get("lineNumber").MustInt() == 0 {
t.Errorf("Wrong frame0")
}
frame1 := exception.Get("stacktrace").GetIndex(1)
if frame1.Get("file").MustString() != "testing/testing.go" ||
frame1.Get("method").MustString() != "tRunner" ||
frame1.Get("inProject").MustBool() != false ||
frame1.Get("lineNumber").MustInt() == 0 {
t.Errorf("Wrong frame1")
}
}
func crashyHandler(w http.ResponseWriter, r *http.Request) {
c := make(chan int)
close(c)
c <- 1
}
func runCrashyServer(rawData ...interface{}) (net.Listener, error) {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
return nil, err
}
mux := http.NewServeMux()
mux.HandleFunc("/", crashyHandler)
srv := http.Server{
Addr: l.Addr().String(),
Handler: Handler(mux, rawData...),
ErrorLog: log.New(ioutil.Discard, log.Prefix(), 0),
}
go srv.Serve(l)
return l, err
}
func TestHandler(t *testing.T) {
startTestServer()
l, err := runCrashyServer(Configuration{
APIKey: testAPIKey,
Endpoint: testEndpoint,
ProjectPackages: []string{"github.com/bugsnag/bugsnag-go"},
Logger: log.New(ioutil.Discard, log.Prefix(), log.Flags()),
}, SeverityInfo)
if err != nil {
t.Fatal(err)
}
http.Get("http://" + l.Addr().String() + "/ok?foo=bar")
l.Close()
json, err := simplejson.NewJson(<-postedJSON)
if err != nil {
t.Fatal(err)
}
if json.Get("apiKey").MustString() != testAPIKey {
t.Errorf("Wrong api key in payload")
}
if json.GetPath("notifier", "name").MustString() != "Bugsnag Go" {
t.Errorf("Wrong notifier name in payload")
}
event := json.Get("events").GetIndex(0)
for k, value := range map[string]string{
"payloadVersion": "2",
"severity": "info",
"user.id": "127.0.0.1",
"metaData.Request.Url": "http://" + l.Addr().String() + "/ok?foo=bar",
"metaData.Request.Method": "GET",
} {
key := strings.Split(k, ".")
if event.GetPath(key...).MustString() != value {
t.Errorf("Wrong %v: %v != %v", key, event.GetPath(key...).MustString(), value)
}
}
if event.GetPath("metaData", "Request", "Params", "foo").GetIndex(0).MustString() != "bar" {
t.Errorf("missing GET params in request metadata")
}
if event.GetPath("metaData", "Headers", "Accept-Encoding").GetIndex(0).MustString() != "gzip" {
t.Errorf("missing GET params in request metadata: %v", event.GetPath("metaData", "Headers"))
}
exception := event.Get("exceptions").GetIndex(0)
if exception.Get("message").MustString() != "runtime error: send on closed channel" {
t.Errorf("Wrong message in payload: %v", exception.Get("message").MustString())
}
if exception.Get("errorClass").MustString() != "runtime.errorCString" {
t.Errorf("Wrong errorClass in payload: %v", exception.Get("errorClass").MustString())
}
// TODO:CI these are probably dependent on go version.
frame0 := exception.Get("stacktrace").GetIndex(0)
if frame0.Get("file").MustString() != "runtime/panic.c" ||
frame0.Get("method").MustString() != "panicstring" ||
frame0.Get("inProject").MustBool() != false ||
frame0.Get("lineNumber").MustInt() == 0 {
t.Errorf("Wrong frame0: %v", frame0)
}
frame3 := exception.Get("stacktrace").GetIndex(3)
if frame3.Get("file").MustString() != "bugsnag_test.go" ||
frame3.Get("method").MustString() != "crashyHandler" ||
frame3.Get("inProject").MustBool() != true ||
frame3.Get("lineNumber").MustInt() == 0 {
t.Errorf("Wrong frame3: %v", frame3)
}
}
func TestAutoNotify(t *testing.T) {
var panicked interface{}
func() {
defer func() {
panicked = recover()
}()
defer AutoNotify(Configuration{Endpoint: testEndpoint, APIKey: testAPIKey})
panic("eggs")
}()
if panicked.(string) != "eggs" {
t.Errorf("didn't re-panic")
}
json, err := simplejson.NewJson(<-postedJSON)
if err != nil {
t.Fatal(err)
}
event := json.Get("events").GetIndex(0)
if event.Get("severity").MustString() != "error" {
t.Errorf("severity should be error")
}
exception := event.Get("exceptions").GetIndex(0)
if exception.Get("message").MustString() != "eggs" {
t.Errorf("caught wrong panic")
}
}
func TestRecover(t *testing.T) {
var panicked interface{}
func() {
defer func() {
panicked = recover()
}()
defer Recover(Configuration{Endpoint: testEndpoint, APIKey: testAPIKey})
panic("ham")
}()
if panicked != nil {
t.Errorf("re-panick'd")
}
json, err := simplejson.NewJson(<-postedJSON)
if err != nil {
t.Fatal(err)
}
event := json.Get("events").GetIndex(0)
if event.Get("severity").MustString() != "warning" {
t.Errorf("severity should be warning")
}
exception := event.Get("exceptions").GetIndex(0)
if exception.Get("message").MustString() != "ham" {
t.Errorf("caught wrong panic")
}
}
func handleGet(w http.ResponseWriter, r *http.Request) {
}
var createAccount = handleGet
type _job struct {
Name string
Process func()
}
func ExampleAutoNotify() interface{} {
return func(w http.ResponseWriter, request *http.Request) {
defer AutoNotify(request, Context{"createAccount"})
createAccount(w, request)
}
}
func ExampleRecover(job _job) {
go func() {
defer Recover(Context{job.Name}, SeverityWarning)
job.Process()
}()
}
func ExampleConfigure() {
Configure(Configuration{
APIKey: "YOUR_API_KEY_HERE",
ReleaseStage: "production",
// See Configuration{} for other fields
})
}
func ExampleHandler() {
// Set up your http handlers as usual
http.HandleFunc("/", handleGet)
// use bugsnag.Handler(nil) to wrap the default http handlers
// so that Bugsnag is automatically notified about panics.
http.ListenAndServe(":1234", Handler(nil))
}
func ExampleHandler_customServer() {
// If you're using a custom server, set the handlers explicitly.
http.HandleFunc("/", handleGet)
srv := http.Server{
Addr: ":1234",
ReadTimeout: 10 * time.Second,
// use bugsnag.Handler(nil) to wrap the default http handlers
// so that Bugsnag is automatically notified about panics.
Handler: Handler(nil),
}
srv.ListenAndServe()
}
func ExampleHandler_customHandlers() {
// If you're using custom handlers, wrap the handlers explicitly.
handler := http.NewServeMux()
http.HandleFunc("/", handleGet)
// use bugsnag.Handler(handler) to wrap the handlers so that Bugsnag is
// automatically notified about panics
http.ListenAndServe(":1234", Handler(handler))
}
func ExampleNotify() {
_, err := net.Listen("tcp", ":80")
if err != nil {
Notify(err)
}
}
func ExampleNotify_details(userID string) {
_, err := net.Listen("tcp", ":80")
if err != nil {
Notify(err,
// show as low-severity
SeverityInfo,
// set the context
Context{"createlistener"},
// pass the user id in to count users affected.
User{Id: userID},
// custom meta-data tab
MetaData{
"Listen": {
"Protocol": "tcp",
"Port": "80",
},
},
)
}
}
type Job struct {
Retry bool
UserId string
UserEmail string
Name string
Params map[string]string
}
func ExampleOnBeforeNotify() {
OnBeforeNotify(func(event *Event, config *Configuration) error {
// Search all the RawData for any *Job pointers that we're passed in
// to bugsnag.Notify() and friends.
for _, datum := range event.RawData {
if job, ok := datum.(*Job); ok {
// don't notify bugsnag about errors in retries
if job.Retry {
return fmt.Errorf("bugsnag middleware: not notifying about job retry")
}
// add the job as a tab on Bugsnag.com
event.MetaData.AddStruct("Job", job)
// set the user correctly
event.User = &User{Id: job.UserId, Email: job.UserEmail}
}
}
// continue notifying as normal
return nil
})
}

View file

@ -1,58 +0,0 @@
package bugsnag
import (
"testing"
)
func TestNotifyReleaseStages(t *testing.T) {
var testCases = []struct {
stage string
configured []string
notify bool
msg string
}{
{
stage: "production",
notify: true,
msg: "Should notify in all release stages by default",
},
{
stage: "production",
configured: []string{"development", "production"},
notify: true,
msg: "Failed to notify in configured release stage",
},
{
stage: "staging",
configured: []string{"development", "production"},
notify: false,
msg: "Failed to prevent notification in excluded release stage",
},
}
for _, testCase := range testCases {
Configure(Configuration{ReleaseStage: testCase.stage, NotifyReleaseStages: testCase.configured})
if Config.notifyInReleaseStage() != testCase.notify {
t.Error(testCase.msg)
}
}
}
func TestProjectPackages(t *testing.T) {
Configure(Configuration{ProjectPackages: []string{"main", "github.com/ConradIrwin/*"}})
if !Config.isProjectPackage("main") {
t.Error("literal project package doesn't work")
}
if !Config.isProjectPackage("github.com/ConradIrwin/foo") {
t.Error("wildcard project package doesn't work")
}
if Config.isProjectPackage("runtime") {
t.Error("wrong packges being marked in project")
}
if Config.isProjectPackage("github.com/ConradIrwin/foo/bar") {
t.Error("wrong packges being marked in project")
}
}

View file

@ -1,117 +0,0 @@
package errors
import (
"bytes"
"fmt"
"io"
"runtime/debug"
"testing"
)
func TestStackFormatMatches(t *testing.T) {
defer func() {
err := recover()
if err != 'a' {
t.Fatal(err)
}
bs := [][]byte{Errorf("hi").Stack(), debug.Stack()}
// Ignore the first line (as it contains the PC of the .Stack() call)
bs[0] = bytes.SplitN(bs[0], []byte("\n"), 2)[1]
bs[1] = bytes.SplitN(bs[1], []byte("\n"), 2)[1]
if bytes.Compare(bs[0], bs[1]) != 0 {
t.Errorf("Stack didn't match")
t.Errorf("%s", bs[0])
t.Errorf("%s", bs[1])
}
}()
a()
}
func TestSkipWorks(t *testing.T) {
defer func() {
err := recover()
if err != 'a' {
t.Fatal(err)
}
bs := [][]byte{New("hi", 2).Stack(), debug.Stack()}
// should skip four lines of debug.Stack()
bs[1] = bytes.SplitN(bs[1], []byte("\n"), 5)[4]
if bytes.Compare(bs[0], bs[1]) != 0 {
t.Errorf("Stack didn't match")
t.Errorf("%s", bs[0])
t.Errorf("%s", bs[1])
}
}()
a()
}
func TestNewError(t *testing.T) {
e := func() error {
return New("hi", 1)
}()
if e.Error() != "hi" {
t.Errorf("Constructor with a string failed")
}
if New(fmt.Errorf("yo"), 0).Error() != "yo" {
t.Errorf("Constructor with an error failed")
}
if New(e, 0) != e {
t.Errorf("Constructor with an Error failed")
}
if New(nil, 0).Error() != "<nil>" {
t.Errorf("Constructor with nil failed")
}
}
func ExampleErrorf(x int) (int, error) {
if x%2 == 1 {
return 0, Errorf("can only halve even numbers, got %d", x)
}
return x / 2, nil
}
func ExampleNewError() (error, error) {
// Wrap io.EOF with the current stack-trace and return it
return nil, New(io.EOF, 0)
}
func ExampleNewError_skip() {
defer func() {
if err := recover(); err != nil {
// skip 1 frame (the deferred function) and then return the wrapped err
err = New(err, 1)
}
}()
}
func ExampleError_Stack(err Error) {
fmt.Printf("Error: %s\n%s", err.Error(), err.Stack())
}
func a() error {
b(5)
return nil
}
func b(i int) {
c()
}
func c() {
panic('a')
}

View file

@ -1,142 +0,0 @@
package errors
import (
"reflect"
"testing"
)
var createdBy = `panic: hello!
goroutine 54 [running]:
runtime.panic(0x35ce40, 0xc208039db0)
/0/c/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers.func·001()
/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers/app.go:13 +0x74
net/http.(*Server).Serve(0xc20806c780, 0x910c88, 0xc20803e168, 0x0, 0x0)
/0/c/go/src/pkg/net/http/server.go:1698 +0x91
created by github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers.App.Index
/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers/app.go:14 +0x3e
goroutine 16 [IO wait]:
net.runtime_pollWait(0x911c30, 0x72, 0x0)
/0/c/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080ba990, 0x72, 0x0, 0x0)
/0/c/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080ba990, 0x0, 0x0)
/0/c/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2080ba930, 0x58be30, 0x0, 0x9103f0, 0x23)
/0/c/go/src/pkg/net/fd_unix.go:409 +0x343
net.(*TCPListener).AcceptTCP(0xc20803e168, 0x8, 0x0, 0x0)
/0/c/go/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20803e168, 0x0, 0x0, 0x0, 0x0)
/0/c/go/src/pkg/net/tcpsock_posix.go:244 +0x4b
github.com/revel/revel.Run(0xe6d9)
/0/go/src/github.com/revel/revel/server.go:113 +0x926
main.main()
/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/tmp/main.go:109 +0xe1a
`
var normalSplit = `panic: hello!
goroutine 54 [running]:
runtime.panic(0x35ce40, 0xc208039db0)
/0/c/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers.func·001()
/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers/app.go:13 +0x74
net/http.(*Server).Serve(0xc20806c780, 0x910c88, 0xc20803e168, 0x0, 0x0)
/0/c/go/src/pkg/net/http/server.go:1698 +0x91
goroutine 16 [IO wait]:
net.runtime_pollWait(0x911c30, 0x72, 0x0)
/0/c/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080ba990, 0x72, 0x0, 0x0)
/0/c/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080ba990, 0x0, 0x0)
/0/c/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2080ba930, 0x58be30, 0x0, 0x9103f0, 0x23)
/0/c/go/src/pkg/net/fd_unix.go:409 +0x343
net.(*TCPListener).AcceptTCP(0xc20803e168, 0x8, 0x0, 0x0)
/0/c/go/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20803e168, 0x0, 0x0, 0x0, 0x0)
/0/c/go/src/pkg/net/tcpsock_posix.go:244 +0x4b
github.com/revel/revel.Run(0xe6d9)
/0/go/src/github.com/revel/revel/server.go:113 +0x926
main.main()
/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/tmp/main.go:109 +0xe1a
`
var lastGoroutine = `panic: hello!
goroutine 16 [IO wait]:
net.runtime_pollWait(0x911c30, 0x72, 0x0)
/0/c/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080ba990, 0x72, 0x0, 0x0)
/0/c/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080ba990, 0x0, 0x0)
/0/c/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2080ba930, 0x58be30, 0x0, 0x9103f0, 0x23)
/0/c/go/src/pkg/net/fd_unix.go:409 +0x343
net.(*TCPListener).AcceptTCP(0xc20803e168, 0x8, 0x0, 0x0)
/0/c/go/src/pkg/net/tcpsock_posix.go:234 +0x5d
net.(*TCPListener).Accept(0xc20803e168, 0x0, 0x0, 0x0, 0x0)
/0/c/go/src/pkg/net/tcpsock_posix.go:244 +0x4b
github.com/revel/revel.Run(0xe6d9)
/0/go/src/github.com/revel/revel/server.go:113 +0x926
main.main()
/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/tmp/main.go:109 +0xe1a
goroutine 54 [running]:
runtime.panic(0x35ce40, 0xc208039db0)
/0/c/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers.func·001()
/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers/app.go:13 +0x74
net/http.(*Server).Serve(0xc20806c780, 0x910c88, 0xc20803e168, 0x0, 0x0)
/0/c/go/src/pkg/net/http/server.go:1698 +0x91
`
var result = []StackFrame{
StackFrame{File: "/0/c/go/src/pkg/runtime/panic.c", LineNumber: 279, Name: "panic", Package: "runtime"},
StackFrame{File: "/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers/app.go", LineNumber: 13, Name: "func.001", Package: "github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers"},
StackFrame{File: "/0/c/go/src/pkg/net/http/server.go", LineNumber: 1698, Name: "(*Server).Serve", Package: "net/http"},
}
var resultCreatedBy = append(result,
StackFrame{File: "/0/go/src/github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers/app.go", LineNumber: 14, Name: "App.Index", Package: "github.com/loopj/bugsnag-example-apps/go/revelapp/app/controllers", ProgramCounter: 0x0})
func TestParsePanic(t *testing.T) {
todo := map[string]string{
"createdBy": createdBy,
"normalSplit": normalSplit,
"lastGoroutine": lastGoroutine,
}
for key, val := range todo {
Err, err := ParsePanic(val)
if err != nil {
t.Fatal(err)
}
if Err.TypeName() != "panic" {
t.Errorf("Wrong type: %s", Err.TypeName())
}
if Err.Error() != "hello!" {
t.Errorf("Wrong message: %s", Err.TypeName())
}
if Err.StackFrames()[0].Func() != nil {
t.Errorf("Somehow managed to find a func...")
}
result := result
if key == "createdBy" {
result = resultCreatedBy
}
if !reflect.DeepEqual(Err.StackFrames(), result) {
t.Errorf("Wrong stack for %s: %#v", key, Err.StackFrames())
}
}
}

View file

@ -1,182 +0,0 @@
package bugsnag
import (
"reflect"
"testing"
"unsafe"
"github.com/bugsnag/bugsnag-go/errors"
)
type _account struct {
ID string
Name string
Plan struct {
Premium bool
}
Password string
secret string
Email string `json:"email"`
EmptyEmail string `json:"emptyemail,omitempty"`
NotEmptyEmail string `json:"not_empty_email,omitempty"`
}
type _broken struct {
Me *_broken
Data string
}
var account = _account{}
var notifier = New(Configuration{})
func TestMetaDataAdd(t *testing.T) {
m := MetaData{
"one": {
"key": "value",
"override": false,
}}
m.Add("one", "override", true)
m.Add("one", "new", "key")
m.Add("new", "tab", account)
m.AddStruct("lol", "not really a struct")
m.AddStruct("account", account)
if !reflect.DeepEqual(m, MetaData{
"one": {
"key": "value",
"override": true,
"new": "key",
},
"new": {
"tab": account,
},
"Extra data": {
"lol": "not really a struct",
},
"account": {
"ID": "",
"Name": "",
"Plan": map[string]interface{}{
"Premium": false,
},
"Password": "",
"email": "",
},
}) {
t.Errorf("metadata.Add didn't work: %#v", m)
}
}
func TestMetaDataUpdate(t *testing.T) {
m := MetaData{
"one": {
"key": "value",
"override": false,
}}
m.Update(MetaData{
"one": {
"override": true,
"new": "key",
},
"new": {
"tab": account,
},
})
if !reflect.DeepEqual(m, MetaData{
"one": {
"key": "value",
"override": true,
"new": "key",
},
"new": {
"tab": account,
},
}) {
t.Errorf("metadata.Update didn't work: %#v", m)
}
}
func TestMetaDataSanitize(t *testing.T) {
var broken = _broken{}
broken.Me = &broken
broken.Data = "ohai"
account.Name = "test"
account.ID = "test"
account.secret = "hush"
account.Email = "example@example.com"
account.EmptyEmail = ""
account.NotEmptyEmail = "not_empty_email@example.com"
m := MetaData{
"one": {
"bool": true,
"int": 7,
"float": 7.1,
"complex": complex(1, 1),
"func": func() {},
"unsafe": unsafe.Pointer(broken.Me),
"string": "string",
"password": "secret",
"array": []hash{{
"creditcard": "1234567812345678",
"broken": broken,
}},
"broken": broken,
"account": account,
},
}
n := m.sanitize([]string{"password", "creditcard"})
if !reflect.DeepEqual(n, map[string]interface{}{
"one": map[string]interface{}{
"bool": true,
"int": 7,
"float": 7.1,
"complex": "[complex128]",
"string": "string",
"unsafe": "[unsafe.Pointer]",
"func": "[func()]",
"password": "[REDACTED]",
"array": []interface{}{map[string]interface{}{
"creditcard": "[REDACTED]",
"broken": map[string]interface{}{
"Me": "[RECURSION]",
"Data": "ohai",
},
}},
"broken": map[string]interface{}{
"Me": "[RECURSION]",
"Data": "ohai",
},
"account": map[string]interface{}{
"ID": "test",
"Name": "test",
"Plan": map[string]interface{}{
"Premium": false,
},
"Password": "[REDACTED]",
"email": "example@example.com",
"not_empty_email": "not_empty_email@example.com",
},
},
}) {
t.Errorf("metadata.Sanitize didn't work: %#v", n)
}
}
func ExampleMetaData() {
notifier.Notify(errors.Errorf("hi world"),
MetaData{"Account": {
"id": account.ID,
"name": account.Name,
"paying?": account.Plan.Premium,
}})
}

View file

@ -1,88 +0,0 @@
package bugsnag
import (
"bytes"
"fmt"
"log"
"reflect"
"testing"
)
func TestMiddlewareOrder(t *testing.T) {
result := make([]int, 0, 7)
stack := middlewareStack{}
stack.OnBeforeNotify(func(e *Event, c *Configuration) error {
result = append(result, 2)
return nil
})
stack.OnBeforeNotify(func(e *Event, c *Configuration) error {
result = append(result, 1)
return nil
})
stack.OnBeforeNotify(func(e *Event, c *Configuration) error {
result = append(result, 0)
return nil
})
stack.Run(nil, nil, func() error {
result = append(result, 3)
return nil
})
if !reflect.DeepEqual(result, []int{0, 1, 2, 3}) {
t.Errorf("unexpected middleware order %v", result)
}
}
func TestBeforeNotifyReturnErr(t *testing.T) {
stack := middlewareStack{}
err := fmt.Errorf("test")
stack.OnBeforeNotify(func(e *Event, c *Configuration) error {
return err
})
called := false
e := stack.Run(nil, nil, func() error {
called = true
return nil
})
if e != err {
t.Errorf("Middleware didn't return the error")
}
if called == true {
t.Errorf("Notify was called when BeforeNotify returned False")
}
}
func TestBeforeNotifyPanic(t *testing.T) {
stack := middlewareStack{}
stack.OnBeforeNotify(func(e *Event, c *Configuration) error {
panic("oops")
})
called := false
b := &bytes.Buffer{}
stack.Run(nil, &Configuration{Logger: log.New(b, log.Prefix(), 0)}, func() error {
called = true
return nil
})
logged := b.String()
if logged != "bugsnag/middleware: unexpected panic: oops\n" {
t.Errorf("Logged: %s", logged)
}
if called == false {
t.Errorf("Notify was not called when BeforeNotify panicked")
}
}

View file

@ -1,79 +0,0 @@
// +build !appengine
package bugsnag
import (
"github.com/bitly/go-simplejson"
"github.com/mitchellh/osext"
"os"
"os/exec"
"testing"
"time"
)
func TestPanicHandler(t *testing.T) {
startTestServer()
exePath, err := osext.Executable()
if err != nil {
t.Fatal(err)
}
// Use the same trick as panicwrap() to re-run ourselves.
// In the init() block below, we will then panic.
cmd := exec.Command(exePath, os.Args[1:]...)
cmd.Env = append(os.Environ(), "BUGSNAG_API_KEY="+testAPIKey, "BUGSNAG_ENDPOINT="+testEndpoint, "please_panic=please_panic")
if err = cmd.Start(); err != nil {
t.Fatal(err)
}
if err = cmd.Wait(); err.Error() != "exit status 2" {
t.Fatal(err)
}
json, err := simplejson.NewJson(<-postedJSON)
if err != nil {
t.Fatal(err)
}
event := json.Get("events").GetIndex(0)
if event.Get("severity").MustString() != "error" {
t.Errorf("severity should be error")
}
exception := event.Get("exceptions").GetIndex(0)
if exception.Get("message").MustString() != "ruh roh" {
t.Errorf("caught wrong panic")
}
if exception.Get("errorClass").MustString() != "panic" {
t.Errorf("caught wrong panic")
}
frame := exception.Get("stacktrace").GetIndex(1)
// Yeah, we just caught a panic from the init() function below and sent it to the server running above (mindblown)
if frame.Get("inProject").MustBool() != true ||
frame.Get("file").MustString() != "panicwrap_test.go" ||
frame.Get("method").MustString() != "panick" ||
frame.Get("lineNumber").MustInt() == 0 {
t.Errorf("stack trace seemed wrong")
}
}
func init() {
if os.Getenv("please_panic") != "" {
Configure(Configuration{APIKey: os.Getenv("BUGSNAG_API_KEY"), Endpoint: os.Getenv("BUGSNAG_ENDPOINT"), ProjectPackages: []string{"github.com/bugsnag/bugsnag-go"}})
go func() {
panick()
}()
// Plenty of time to crash, it shouldn't need any of it.
time.Sleep(1 * time.Second)
}
}
func panick() {
panic("ruh roh")
}

View file

@ -1,60 +0,0 @@
// Package bugsnagrevel adds Bugsnag to revel.
// It lets you pass *revel.Controller into bugsnag.Notify(),
// and provides a Filter to catch errors.
package bugsnagrevel
import (
"strings"
"sync"
"github.com/bugsnag/bugsnag-go"
"github.com/revel/revel"
)
var once sync.Once
// Filter should be added to the filter chain just after the PanicFilter.
// It sends errors to Bugsnag automatically. Configuration is read out of
// conf/app.conf, you should set bugsnag.apikey, and can also set
// bugsnag.endpoint, bugsnag.releasestage, bugsnag.appversion,
// bugsnag.projectroot, bugsnag.projectpackages if needed.
func Filter(c *revel.Controller, fc []revel.Filter) {
defer bugsnag.AutoNotify(c)
fc[0](c, fc[1:])
}
// Add support to bugsnag for reading data out of *revel.Controllers
func middleware(event *bugsnag.Event, config *bugsnag.Configuration) error {
for _, datum := range event.RawData {
if controller, ok := datum.(*revel.Controller); ok {
// make the request visible to the builtin HttpMIddleware
event.RawData = append(event.RawData, controller.Request.Request)
event.Context = controller.Action
event.MetaData.AddStruct("Session", controller.Session)
}
}
return nil
}
func init() {
revel.OnAppStart(func() {
bugsnag.OnBeforeNotify(middleware)
var projectPackages []string
if packages, ok := revel.Config.String("bugsnag.projectpackages"); ok {
projectPackages = strings.Split(packages, ",")
} else {
projectPackages = []string{revel.ImportPath + "/app/*", revel.ImportPath + "/app"}
}
bugsnag.Configure(bugsnag.Configuration{
APIKey: revel.Config.StringDefault("bugsnag.apikey", ""),
Endpoint: revel.Config.StringDefault("bugsnag.endpoint", ""),
AppVersion: revel.Config.StringDefault("bugsnag.appversion", ""),
ReleaseStage: revel.Config.StringDefault("bugsnag.releasestage", revel.RunMode),
ProjectPackages: projectPackages,
Logger: revel.ERROR,
})
})
}

View file

@ -1,79 +0,0 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build darwin linux freebsd netbsd windows
package osext
import (
"fmt"
"os"
oexec "os/exec"
"path/filepath"
"runtime"
"testing"
)
const execPath_EnvVar = "OSTEST_OUTPUT_EXECPATH"
func TestExecPath(t *testing.T) {
ep, err := Executable()
if err != nil {
t.Fatalf("ExecPath failed: %v", err)
}
// we want fn to be of the form "dir/prog"
dir := filepath.Dir(filepath.Dir(ep))
fn, err := filepath.Rel(dir, ep)
if err != nil {
t.Fatalf("filepath.Rel: %v", err)
}
cmd := &oexec.Cmd{}
// make child start with a relative program path
cmd.Dir = dir
cmd.Path = fn
// forge argv[0] for child, so that we can verify we could correctly
// get real path of the executable without influenced by argv[0].
cmd.Args = []string{"-", "-test.run=XXXX"}
cmd.Env = []string{fmt.Sprintf("%s=1", execPath_EnvVar)}
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("exec(self) failed: %v", err)
}
outs := string(out)
if !filepath.IsAbs(outs) {
t.Fatalf("Child returned %q, want an absolute path", out)
}
if !sameFile(outs, ep) {
t.Fatalf("Child returned %q, not the same file as %q", out, ep)
}
}
func sameFile(fn1, fn2 string) bool {
fi1, err := os.Stat(fn1)
if err != nil {
return false
}
fi2, err := os.Stat(fn2)
if err != nil {
return false
}
return os.SameFile(fi1, fi2)
}
func init() {
if e := os.Getenv(execPath_EnvVar); e != "" {
// first chdir to another path
dir := "/"
if runtime.GOOS == "windows" {
dir = filepath.VolumeName(".")
}
os.Chdir(dir)
if ep, err := Executable(); err != nil {
fmt.Fprint(os.Stderr, "ERROR: ", err)
} else {
fmt.Fprint(os.Stderr, ep)
}
os.Exit(0)
}
}

View file

@ -1,360 +0,0 @@
package panicwrap
import (
"bytes"
"fmt"
"os"
"os/exec"
"strings"
"testing"
"time"
)
func helperProcess(s ...string) *exec.Cmd {
cs := []string{"-test.run=TestHelperProcess", "--"}
cs = append(cs, s...)
env := []string{
"GO_WANT_HELPER_PROCESS=1",
}
cmd := exec.Command(os.Args[0], cs...)
cmd.Env = append(env, os.Environ()...)
cmd.Stdin = os.Stdin
cmd.Stderr = os.Stderr
cmd.Stdout = os.Stdout
return cmd
}
// This is executed by `helperProcess` in a separate process in order to
// provider a proper sub-process environment to test some of our functionality.
func TestHelperProcess(*testing.T) {
if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
return
}
// Find the arguments to our helper, which are the arguments past
// the "--" in the command line.
args := os.Args
for len(args) > 0 {
if args[0] == "--" {
args = args[1:]
break
}
args = args[1:]
}
if len(args) == 0 {
fmt.Fprintf(os.Stderr, "No command\n")
os.Exit(2)
}
panicHandler := func(s string) {
fmt.Fprintf(os.Stdout, "wrapped: %d", len(s))
os.Exit(0)
}
cmd, args := args[0], args[1:]
switch cmd {
case "no-panic-ordered-output":
exitStatus, err := BasicWrap(panicHandler)
if err != nil {
fmt.Fprintf(os.Stderr, "wrap error: %s", err)
os.Exit(1)
}
if exitStatus < 0 {
for i := 0; i < 1000; i++ {
os.Stdout.Write([]byte("a"))
os.Stderr.Write([]byte("b"))
}
os.Exit(0)
}
os.Exit(exitStatus)
case "no-panic-output":
fmt.Fprint(os.Stdout, "i am output")
fmt.Fprint(os.Stderr, "stderr out")
os.Exit(0)
case "panic-boundary":
exitStatus, err := BasicWrap(panicHandler)
if err != nil {
fmt.Fprintf(os.Stderr, "wrap error: %s", err)
os.Exit(1)
}
if exitStatus < 0 {
// Simulate a panic but on two boundaries...
fmt.Fprint(os.Stderr, "pan")
os.Stderr.Sync()
time.Sleep(100 * time.Millisecond)
fmt.Fprint(os.Stderr, "ic: oh crap")
os.Exit(2)
}
os.Exit(exitStatus)
case "panic-long":
exitStatus, err := BasicWrap(panicHandler)
if err != nil {
fmt.Fprintf(os.Stderr, "wrap error: %s", err)
os.Exit(1)
}
if exitStatus < 0 {
// Make a fake panic by faking the header and adding a
// bunch of garbage.
fmt.Fprint(os.Stderr, "panic: foo\n\n")
for i := 0; i < 1024; i++ {
fmt.Fprint(os.Stderr, "foobarbaz")
}
// Sleep so that it dumps the previous data
//time.Sleep(1 * time.Millisecond)
time.Sleep(500 * time.Millisecond)
// Make a real panic
panic("I AM REAL!")
}
os.Exit(exitStatus)
case "panic":
hidePanic := false
if args[0] == "hide" {
hidePanic = true
}
config := &WrapConfig{
Handler: panicHandler,
HidePanic: hidePanic,
}
exitStatus, err := Wrap(config)
if err != nil {
fmt.Fprintf(os.Stderr, "wrap error: %s", err)
os.Exit(1)
}
if exitStatus < 0 {
panic("uh oh")
}
os.Exit(exitStatus)
case "wrapped":
child := false
if len(args) > 0 && args[0] == "child" {
child = true
}
config := &WrapConfig{
Handler: panicHandler,
}
exitStatus, err := Wrap(config)
if err != nil {
fmt.Fprintf(os.Stderr, "wrap error: %s", err)
os.Exit(1)
}
if exitStatus < 0 {
if child {
fmt.Printf("%v", Wrapped(config))
}
os.Exit(0)
}
if !child {
fmt.Printf("%v", Wrapped(config))
}
os.Exit(exitStatus)
case "panic-monitor":
config := &WrapConfig{
Handler: panicHandler,
HidePanic: true,
Monitor: true,
}
exitStatus, err := Wrap(config)
if err != nil {
fmt.Fprintf(os.Stderr, "wrap error: %s", err)
os.Exit(1)
}
if exitStatus != -1 {
fmt.Fprintf(os.Stderr, "wrap error: %s", err)
os.Exit(1)
}
panic("uh oh")
default:
fmt.Fprintf(os.Stderr, "Unknown command: %q\n", cmd)
os.Exit(2)
}
}
func TestPanicWrap_Output(t *testing.T) {
stderr := new(bytes.Buffer)
stdout := new(bytes.Buffer)
p := helperProcess("no-panic-output")
p.Stdout = stdout
p.Stderr = stderr
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "i am output") {
t.Fatalf("didn't forward: %#v", stdout.String())
}
if !strings.Contains(stderr.String(), "stderr out") {
t.Fatalf("didn't forward: %#v", stderr.String())
}
}
/*
TODO(mitchellh): This property would be nice to gain.
func TestPanicWrap_Output_Order(t *testing.T) {
output := new(bytes.Buffer)
p := helperProcess("no-panic-ordered-output")
p.Stdout = output
p.Stderr = output
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
expectedBuf := new(bytes.Buffer)
for i := 0; i < 1000; i++ {
expectedBuf.WriteString("ab")
}
actual := strings.TrimSpace(output.String())
expected := strings.TrimSpace(expectedBuf.String())
if actual != expected {
t.Fatalf("bad: %#v", actual)
}
}
*/
func TestPanicWrap_panicHide(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
p := helperProcess("panic", "hide")
p.Stdout = stdout
p.Stderr = stderr
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "wrapped:") {
t.Fatalf("didn't wrap: %#v", stdout.String())
}
if strings.Contains(stderr.String(), "panic:") {
t.Fatalf("shouldn't have panic: %#v", stderr.String())
}
}
func TestPanicWrap_panicShow(t *testing.T) {
stdout := new(bytes.Buffer)
stderr := new(bytes.Buffer)
p := helperProcess("panic", "show")
p.Stdout = stdout
p.Stderr = stderr
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "wrapped:") {
t.Fatalf("didn't wrap: %#v", stdout.String())
}
if !strings.Contains(stderr.String(), "panic:") {
t.Fatalf("should have panic: %#v", stderr.String())
}
}
func TestPanicWrap_panicLong(t *testing.T) {
stdout := new(bytes.Buffer)
p := helperProcess("panic-long")
p.Stdout = stdout
p.Stderr = new(bytes.Buffer)
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "wrapped:") {
t.Fatalf("didn't wrap: %#v", stdout.String())
}
}
func TestPanicWrap_panicBoundary(t *testing.T) {
// TODO(mitchellh): panics are currently lost on boundaries
t.SkipNow()
stdout := new(bytes.Buffer)
p := helperProcess("panic-boundary")
p.Stdout = stdout
//p.Stderr = new(bytes.Buffer)
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "wrapped: 1015") {
t.Fatalf("didn't wrap: %#v", stdout.String())
}
}
func TestPanicWrap_monitor(t *testing.T) {
stdout := new(bytes.Buffer)
p := helperProcess("panic-monitor")
p.Stdout = stdout
//p.Stderr = new(bytes.Buffer)
if err := p.Run(); err == nil || err.Error() != "exit status 2" {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "wrapped:") {
t.Fatalf("didn't wrap: %#v", stdout.String())
}
}
func TestWrapped(t *testing.T) {
stdout := new(bytes.Buffer)
p := helperProcess("wrapped", "child")
p.Stdout = stdout
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "true") {
t.Fatalf("bad: %#v", stdout.String())
}
}
func TestWrapped_parent(t *testing.T) {
stdout := new(bytes.Buffer)
p := helperProcess("wrapped")
p.Stdout = stdout
if err := p.Run(); err != nil {
t.Fatalf("err: %s", err)
}
if !strings.Contains(stdout.String(), "false") {
t.Fatalf("bad: %#v", stdout.String())
}
}

View file

@ -1,6 +0,0 @@
language: go
go: 1.1
script:
- go vet ./...
- go test -v ./...

View file

@ -1,298 +0,0 @@
[![Build Status](https://travis-ci.org/codegangsta/cli.png?branch=master)](https://travis-ci.org/codegangsta/cli)
# cli.go
cli.go is simple, fast, and fun package for building command line apps in Go. The goal is to enable developers to write fast and distributable command line applications in an expressive way.
You can view the API docs here:
http://godoc.org/github.com/codegangsta/cli
## Overview
Command line apps are usually so tiny that there is absolutely no reason why your code should *not* be self-documenting. Things like generating help text and parsing command flags/options should not hinder productivity when writing a command line app.
**This is where cli.go comes into play.** cli.go makes command line programming fun, organized, and expressive!
## Installation
Make sure you have a working Go environment (go 1.1 is *required*). [See the install instructions](http://golang.org/doc/install.html).
To install `cli.go`, simply run:
```
$ go get github.com/codegangsta/cli
```
Make sure your `PATH` includes to the `$GOPATH/bin` directory so your commands can be easily used:
```
export PATH=$PATH:$GOPATH/bin
```
## Getting Started
One of the philosophies behind cli.go is that an API should be playful and full of discovery. So a cli.go app can be as little as one line of code in `main()`.
``` go
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
cli.NewApp().Run(os.Args)
}
```
This app will run and show help text, but is not very useful. Let's give an action to execute and some help documentation:
``` go
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
app := cli.NewApp()
app.Name = "boom"
app.Usage = "make an explosive entrance"
app.Action = func(c *cli.Context) {
println("boom! I say!")
}
app.Run(os.Args)
}
```
Running this already gives you a ton of functionality, plus support for things like subcommands and flags, which are covered below.
## Example
Being a programmer can be a lonely job. Thankfully by the power of automation that is not the case! Let's create a greeter app to fend off our demons of loneliness!
Start by creating a directory named `greet`, and within it, add a file, `greet.go` with the following code in it:
``` go
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
app := cli.NewApp()
app.Name = "greet"
app.Usage = "fight the loneliness!"
app.Action = func(c *cli.Context) {
println("Hello friend!")
}
app.Run(os.Args)
}
```
Install our command to the `$GOPATH/bin` directory:
```
$ go install
```
Finally run our new command:
```
$ greet
Hello friend!
```
cli.go also generates some bitchass help text:
```
$ greet help
NAME:
greet - fight the loneliness!
USAGE:
greet [global options] command [command options] [arguments...]
VERSION:
0.0.0
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS
--version Shows version information
```
### Arguments
You can lookup arguments by calling the `Args` function on `cli.Context`.
``` go
...
app.Action = func(c *cli.Context) {
println("Hello", c.Args()[0])
}
...
```
### Flags
Setting and querying flags is simple.
``` go
...
app.Flags = []cli.Flag {
cli.StringFlag{
Name: "lang",
Value: "english",
Usage: "language for the greeting",
},
}
app.Action = func(c *cli.Context) {
name := "someone"
if len(c.Args()) > 0 {
name = c.Args()[0]
}
if c.String("lang") == "spanish" {
println("Hola", name)
} else {
println("Hello", name)
}
}
...
```
#### Alternate Names
You can set alternate (or short) names for flags by providing a comma-delimited list for the `Name`. e.g.
``` go
app.Flags = []cli.Flag {
cli.StringFlag{
Name: "lang, l",
Value: "english",
Usage: "language for the greeting",
},
}
```
That flag can then be set with `--lang spanish` or `-l spanish`. Note that giving two different forms of the same flag in the same command invocation is an error.
#### Values from the Environment
You can also have the default value set from the environment via `EnvVar`. e.g.
``` go
app.Flags = []cli.Flag {
cli.StringFlag{
Name: "lang, l",
Value: "english",
Usage: "language for the greeting",
EnvVar: "APP_LANG",
},
}
```
The `EnvVar` may also be given as a comma-delimited "cascade", where the first environment variable that resolves is used as the default.
``` go
app.Flags = []cli.Flag {
cli.StringFlag{
Name: "lang, l",
Value: "english",
Usage: "language for the greeting",
EnvVar: "LEGACY_COMPAT_LANG,APP_LANG,LANG",
},
}
```
### Subcommands
Subcommands can be defined for a more git-like command line app.
```go
...
app.Commands = []cli.Command{
{
Name: "add",
ShortName: "a",
Usage: "add a task to the list",
Action: func(c *cli.Context) {
println("added task: ", c.Args().First())
},
},
{
Name: "complete",
ShortName: "c",
Usage: "complete a task on the list",
Action: func(c *cli.Context) {
println("completed task: ", c.Args().First())
},
},
{
Name: "template",
ShortName: "r",
Usage: "options for task templates",
Subcommands: []cli.Command{
{
Name: "add",
Usage: "add a new template",
Action: func(c *cli.Context) {
println("new task template: ", c.Args().First())
},
},
{
Name: "remove",
Usage: "remove an existing template",
Action: func(c *cli.Context) {
println("removed task template: ", c.Args().First())
},
},
},
},
}
...
```
### Bash Completion
You can enable completion commands by setting the `EnableBashCompletion`
flag on the `App` object. By default, this setting will only auto-complete to
show an app's subcommands, but you can write your own completion methods for
the App or its subcommands.
```go
...
var tasks = []string{"cook", "clean", "laundry", "eat", "sleep", "code"}
app := cli.NewApp()
app.EnableBashCompletion = true
app.Commands = []cli.Command{
{
Name: "complete",
ShortName: "c",
Usage: "complete a task on the list",
Action: func(c *cli.Context) {
println("completed task: ", c.Args().First())
},
BashComplete: func(c *cli.Context) {
// This will complete if no args are passed
if len(c.Args()) > 0 {
return
}
for _, t := range tasks {
fmt.Println(t)
}
},
}
}
...
```
#### To Enable
Source the `autocomplete/bash_autocomplete` file in your `.bashrc` file while
setting the `PROG` variable to the name of your program:
`PROG=myprogram source /.../cli/autocomplete/bash_autocomplete`
## Contribution Guidelines
Feel free to put up a pull request to fix a bug or maybe add a feature. I will give it a code review and make sure that it does not break backwards compatibility. If I or any other collaborators agree that it is in line with the vision of the project, we will work with you to get the code into a mergeable state and merge it into the master branch.
If you have contributed something significant to the project, I will most likely add you as a collaborator. As a collaborator you are given the ability to merge others pull requests. It is very important that new code does not break existing code, so be careful about what code you do choose to merge. If you have any questions feel free to link @codegangsta to the issue in question and we can review it together.
If you feel like you have contributed to the project but have not yet been added as a collaborator, I probably forgot to add you. Hit @codegangsta up over email and we will get it figured out.

View file

@ -1,275 +0,0 @@
package cli
import (
"fmt"
"io"
"io/ioutil"
"os"
"text/tabwriter"
"text/template"
"time"
)
// App is the main structure of a cli application. It is recomended that
// and app be created with the cli.NewApp() function
type App struct {
// The name of the program. Defaults to os.Args[0]
Name string
// Description of the program.
Usage string
// Version of the program
Version string
// List of commands to execute
Commands []Command
// List of flags to parse
Flags []Flag
// Boolean to enable bash completion commands
EnableBashCompletion bool
// Boolean to hide built-in help command
HideHelp bool
// Boolean to hide built-in version flag
HideVersion bool
// An action to execute when the bash-completion flag is set
BashComplete func(context *Context)
// An action to execute before any subcommands are run, but after the context is ready
// If a non-nil error is returned, no subcommands are run
Before func(context *Context) error
// The action to execute when no subcommands are specified
Action func(context *Context)
// Execute this function if the proper command cannot be found
CommandNotFound func(context *Context, command string)
// Compilation date
Compiled time.Time
// Author
Author string
// Author e-mail
Email string
// Writer writer to write output to
Writer io.Writer
}
// Tries to find out when this binary was compiled.
// Returns the current time if it fails to find it.
func compileTime() time.Time {
info, err := os.Stat(os.Args[0])
if err != nil {
return time.Now()
}
return info.ModTime()
}
// Creates a new cli Application with some reasonable defaults for Name, Usage, Version and Action.
func NewApp() *App {
return &App{
Name: os.Args[0],
Usage: "A new cli application",
Version: "0.0.0",
BashComplete: DefaultAppComplete,
Action: helpCommand.Action,
Compiled: compileTime(),
Author: "Author",
Email: "unknown@email",
Writer: os.Stdout,
}
}
// Entry point to the cli app. Parses the arguments slice and routes to the proper flag/args combination
func (a *App) Run(arguments []string) error {
if HelpPrinter == nil {
defer func() {
HelpPrinter = nil
}()
HelpPrinter = func(templ string, data interface{}) {
w := tabwriter.NewWriter(a.Writer, 0, 8, 1, '\t', 0)
t := template.Must(template.New("help").Parse(templ))
err := t.Execute(w, data)
if err != nil {
panic(err)
}
w.Flush()
}
}
// append help to commands
if a.Command(helpCommand.Name) == nil && !a.HideHelp {
a.Commands = append(a.Commands, helpCommand)
if (HelpFlag != BoolFlag{}) {
a.appendFlag(HelpFlag)
}
}
//append version/help flags
if a.EnableBashCompletion {
a.appendFlag(BashCompletionFlag)
}
if !a.HideVersion {
a.appendFlag(VersionFlag)
}
// parse flags
set := flagSet(a.Name, a.Flags)
set.SetOutput(ioutil.Discard)
err := set.Parse(arguments[1:])
nerr := normalizeFlags(a.Flags, set)
if nerr != nil {
fmt.Fprintln(a.Writer, nerr)
context := NewContext(a, set, set)
ShowAppHelp(context)
fmt.Fprintln(a.Writer)
return nerr
}
context := NewContext(a, set, set)
if err != nil {
fmt.Fprintf(a.Writer, "Incorrect Usage.\n\n")
ShowAppHelp(context)
fmt.Fprintln(a.Writer)
return err
}
if checkCompletions(context) {
return nil
}
if checkHelp(context) {
return nil
}
if checkVersion(context) {
return nil
}
if a.Before != nil {
err := a.Before(context)
if err != nil {
return err
}
}
args := context.Args()
if args.Present() {
name := args.First()
c := a.Command(name)
if c != nil {
return c.Run(context)
}
}
// Run default Action
a.Action(context)
return nil
}
// Another entry point to the cli app, takes care of passing arguments and error handling
func (a *App) RunAndExitOnError() {
if err := a.Run(os.Args); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
// Invokes the subcommand given the context, parses ctx.Args() to generate command-specific flags
func (a *App) RunAsSubcommand(ctx *Context) error {
// append help to commands
if len(a.Commands) > 0 {
if a.Command(helpCommand.Name) == nil && !a.HideHelp {
a.Commands = append(a.Commands, helpCommand)
if (HelpFlag != BoolFlag{}) {
a.appendFlag(HelpFlag)
}
}
}
// append flags
if a.EnableBashCompletion {
a.appendFlag(BashCompletionFlag)
}
// parse flags
set := flagSet(a.Name, a.Flags)
set.SetOutput(ioutil.Discard)
err := set.Parse(ctx.Args().Tail())
nerr := normalizeFlags(a.Flags, set)
context := NewContext(a, set, ctx.globalSet)
if nerr != nil {
fmt.Fprintln(a.Writer, nerr)
if len(a.Commands) > 0 {
ShowSubcommandHelp(context)
} else {
ShowCommandHelp(ctx, context.Args().First())
}
fmt.Fprintln(a.Writer)
return nerr
}
if err != nil {
fmt.Fprintf(a.Writer, "Incorrect Usage.\n\n")
ShowSubcommandHelp(context)
return err
}
if checkCompletions(context) {
return nil
}
if len(a.Commands) > 0 {
if checkSubcommandHelp(context) {
return nil
}
} else {
if checkCommandHelp(ctx, context.Args().First()) {
return nil
}
}
if a.Before != nil {
err := a.Before(context)
if err != nil {
return err
}
}
args := context.Args()
if args.Present() {
name := args.First()
c := a.Command(name)
if c != nil {
return c.Run(context)
}
}
// Run default Action
a.Action(context)
return nil
}
// Returns the named command on App. Returns nil if the command does not exist
func (a *App) Command(name string) *Command {
for _, c := range a.Commands {
if c.HasName(name) {
return &c
}
}
return nil
}
func (a *App) hasFlag(flag Flag) bool {
for _, f := range a.Flags {
if flag == f {
return true
}
}
return false
}
func (a *App) appendFlag(flag Flag) {
if !a.hasFlag(flag) {
a.Flags = append(a.Flags, flag)
}
}

View file

@ -1,554 +0,0 @@
package cli_test
import (
"flag"
"fmt"
"os"
"testing"
"github.com/codegangsta/cli"
)
func ExampleApp() {
// set args for examples sake
os.Args = []string{"greet", "--name", "Jeremy"}
app := cli.NewApp()
app.Name = "greet"
app.Flags = []cli.Flag{
cli.StringFlag{Name: "name", Value: "bob", Usage: "a name to say"},
}
app.Action = func(c *cli.Context) {
fmt.Printf("Hello %v\n", c.String("name"))
}
app.Run(os.Args)
// Output:
// Hello Jeremy
}
func ExampleAppSubcommand() {
// set args for examples sake
os.Args = []string{"say", "hi", "english", "--name", "Jeremy"}
app := cli.NewApp()
app.Name = "say"
app.Commands = []cli.Command{
{
Name: "hello",
ShortName: "hi",
Usage: "use it to see a description",
Description: "This is how we describe hello the function",
Subcommands: []cli.Command{
{
Name: "english",
ShortName: "en",
Usage: "sends a greeting in english",
Description: "greets someone in english",
Flags: []cli.Flag{
cli.StringFlag{
Name: "name",
Value: "Bob",
Usage: "Name of the person to greet",
},
},
Action: func(c *cli.Context) {
fmt.Println("Hello,", c.String("name"))
},
},
},
},
}
app.Run(os.Args)
// Output:
// Hello, Jeremy
}
func ExampleAppHelp() {
// set args for examples sake
os.Args = []string{"greet", "h", "describeit"}
app := cli.NewApp()
app.Name = "greet"
app.Flags = []cli.Flag{
cli.StringFlag{Name: "name", Value: "bob", Usage: "a name to say"},
}
app.Commands = []cli.Command{
{
Name: "describeit",
ShortName: "d",
Usage: "use it to see a description",
Description: "This is how we describe describeit the function",
Action: func(c *cli.Context) {
fmt.Printf("i like to describe things")
},
},
}
app.Run(os.Args)
// Output:
// NAME:
// describeit - use it to see a description
//
// USAGE:
// command describeit [arguments...]
//
// DESCRIPTION:
// This is how we describe describeit the function
}
func ExampleAppBashComplete() {
// set args for examples sake
os.Args = []string{"greet", "--generate-bash-completion"}
app := cli.NewApp()
app.Name = "greet"
app.EnableBashCompletion = true
app.Commands = []cli.Command{
{
Name: "describeit",
ShortName: "d",
Usage: "use it to see a description",
Description: "This is how we describe describeit the function",
Action: func(c *cli.Context) {
fmt.Printf("i like to describe things")
},
}, {
Name: "next",
Usage: "next example",
Description: "more stuff to see when generating bash completion",
Action: func(c *cli.Context) {
fmt.Printf("the next example")
},
},
}
app.Run(os.Args)
// Output:
// describeit
// d
// next
// help
// h
}
func TestApp_Run(t *testing.T) {
s := ""
app := cli.NewApp()
app.Action = func(c *cli.Context) {
s = s + c.Args().First()
}
err := app.Run([]string{"command", "foo"})
expect(t, err, nil)
err = app.Run([]string{"command", "bar"})
expect(t, err, nil)
expect(t, s, "foobar")
}
var commandAppTests = []struct {
name string
expected bool
}{
{"foobar", true},
{"batbaz", true},
{"b", true},
{"f", true},
{"bat", false},
{"nothing", false},
}
func TestApp_Command(t *testing.T) {
app := cli.NewApp()
fooCommand := cli.Command{Name: "foobar", ShortName: "f"}
batCommand := cli.Command{Name: "batbaz", ShortName: "b"}
app.Commands = []cli.Command{
fooCommand,
batCommand,
}
for _, test := range commandAppTests {
expect(t, app.Command(test.name) != nil, test.expected)
}
}
func TestApp_CommandWithArgBeforeFlags(t *testing.T) {
var parsedOption, firstArg string
app := cli.NewApp()
command := cli.Command{
Name: "cmd",
Flags: []cli.Flag{
cli.StringFlag{Name: "option", Value: "", Usage: "some option"},
},
Action: func(c *cli.Context) {
parsedOption = c.String("option")
firstArg = c.Args().First()
},
}
app.Commands = []cli.Command{command}
app.Run([]string{"", "cmd", "my-arg", "--option", "my-option"})
expect(t, parsedOption, "my-option")
expect(t, firstArg, "my-arg")
}
func TestApp_RunAsSubcommandParseFlags(t *testing.T) {
var context *cli.Context
a := cli.NewApp()
a.Commands = []cli.Command{
{
Name: "foo",
Action: func(c *cli.Context) {
context = c
},
Flags: []cli.Flag{
cli.StringFlag{
Name: "lang",
Value: "english",
Usage: "language for the greeting",
},
},
Before: func(_ *cli.Context) error { return nil },
},
}
a.Run([]string{"", "foo", "--lang", "spanish", "abcd"})
expect(t, context.Args().Get(0), "abcd")
expect(t, context.String("lang"), "spanish")
}
func TestApp_CommandWithFlagBeforeTerminator(t *testing.T) {
var parsedOption string
var args []string
app := cli.NewApp()
command := cli.Command{
Name: "cmd",
Flags: []cli.Flag{
cli.StringFlag{Name: "option", Value: "", Usage: "some option"},
},
Action: func(c *cli.Context) {
parsedOption = c.String("option")
args = c.Args()
},
}
app.Commands = []cli.Command{command}
app.Run([]string{"", "cmd", "my-arg", "--option", "my-option", "--", "--notARealFlag"})
expect(t, parsedOption, "my-option")
expect(t, args[0], "my-arg")
expect(t, args[1], "--")
expect(t, args[2], "--notARealFlag")
}
func TestApp_CommandWithNoFlagBeforeTerminator(t *testing.T) {
var args []string
app := cli.NewApp()
command := cli.Command{
Name: "cmd",
Action: func(c *cli.Context) {
args = c.Args()
},
}
app.Commands = []cli.Command{command}
app.Run([]string{"", "cmd", "my-arg", "--", "notAFlagAtAll"})
expect(t, args[0], "my-arg")
expect(t, args[1], "--")
expect(t, args[2], "notAFlagAtAll")
}
func TestApp_Float64Flag(t *testing.T) {
var meters float64
app := cli.NewApp()
app.Flags = []cli.Flag{
cli.Float64Flag{Name: "height", Value: 1.5, Usage: "Set the height, in meters"},
}
app.Action = func(c *cli.Context) {
meters = c.Float64("height")
}
app.Run([]string{"", "--height", "1.93"})
expect(t, meters, 1.93)
}
func TestApp_ParseSliceFlags(t *testing.T) {
var parsedOption, firstArg string
var parsedIntSlice []int
var parsedStringSlice []string
app := cli.NewApp()
command := cli.Command{
Name: "cmd",
Flags: []cli.Flag{
cli.IntSliceFlag{Name: "p", Value: &cli.IntSlice{}, Usage: "set one or more ip addr"},
cli.StringSliceFlag{Name: "ip", Value: &cli.StringSlice{}, Usage: "set one or more ports to open"},
},
Action: func(c *cli.Context) {
parsedIntSlice = c.IntSlice("p")
parsedStringSlice = c.StringSlice("ip")
parsedOption = c.String("option")
firstArg = c.Args().First()
},
}
app.Commands = []cli.Command{command}
app.Run([]string{"", "cmd", "my-arg", "-p", "22", "-p", "80", "-ip", "8.8.8.8", "-ip", "8.8.4.4"})
IntsEquals := func(a, b []int) bool {
if len(a) != len(b) {
return false
}
for i, v := range a {
if v != b[i] {
return false
}
}
return true
}
StrsEquals := func(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i, v := range a {
if v != b[i] {
return false
}
}
return true
}
var expectedIntSlice = []int{22, 80}
var expectedStringSlice = []string{"8.8.8.8", "8.8.4.4"}
if !IntsEquals(parsedIntSlice, expectedIntSlice) {
t.Errorf("%v does not match %v", parsedIntSlice, expectedIntSlice)
}
if !StrsEquals(parsedStringSlice, expectedStringSlice) {
t.Errorf("%v does not match %v", parsedStringSlice, expectedStringSlice)
}
}
func TestApp_DefaultStdout(t *testing.T) {
app := cli.NewApp()
if app.Writer != os.Stdout {
t.Error("Default output writer not set.")
}
}
type mockWriter struct {
written []byte
}
func (fw *mockWriter) Write(p []byte) (n int, err error) {
if fw.written == nil {
fw.written = p
} else {
fw.written = append(fw.written, p...)
}
return len(p), nil
}
func (fw *mockWriter) GetWritten() (b []byte) {
return fw.written
}
func TestApp_SetStdout(t *testing.T) {
w := &mockWriter{}
app := cli.NewApp()
app.Name = "test"
app.Writer = w
err := app.Run([]string{"help"})
if err != nil {
t.Fatalf("Run error: %s", err)
}
if len(w.written) == 0 {
t.Error("App did not write output to desired writer.")
}
}
func TestApp_BeforeFunc(t *testing.T) {
beforeRun, subcommandRun := false, false
beforeError := fmt.Errorf("fail")
var err error
app := cli.NewApp()
app.Before = func(c *cli.Context) error {
beforeRun = true
s := c.String("opt")
if s == "fail" {
return beforeError
}
return nil
}
app.Commands = []cli.Command{
cli.Command{
Name: "sub",
Action: func(c *cli.Context) {
subcommandRun = true
},
},
}
app.Flags = []cli.Flag{
cli.StringFlag{Name: "opt"},
}
// run with the Before() func succeeding
err = app.Run([]string{"command", "--opt", "succeed", "sub"})
if err != nil {
t.Fatalf("Run error: %s", err)
}
if beforeRun == false {
t.Errorf("Before() not executed when expected")
}
if subcommandRun == false {
t.Errorf("Subcommand not executed when expected")
}
// reset
beforeRun, subcommandRun = false, false
// run with the Before() func failing
err = app.Run([]string{"command", "--opt", "fail", "sub"})
// should be the same error produced by the Before func
if err != beforeError {
t.Errorf("Run error expected, but not received")
}
if beforeRun == false {
t.Errorf("Before() not executed when expected")
}
if subcommandRun == true {
t.Errorf("Subcommand executed when NOT expected")
}
}
func TestAppNoHelpFlag(t *testing.T) {
oldFlag := cli.HelpFlag
defer func() {
cli.HelpFlag = oldFlag
}()
cli.HelpFlag = cli.BoolFlag{}
app := cli.NewApp()
err := app.Run([]string{"test", "-h"})
if err != flag.ErrHelp {
t.Errorf("expected error about missing help flag, but got: %s (%T)", err, err)
}
}
func TestAppHelpPrinter(t *testing.T) {
oldPrinter := cli.HelpPrinter
defer func() {
cli.HelpPrinter = oldPrinter
}()
var wasCalled = false
cli.HelpPrinter = func(template string, data interface{}) {
wasCalled = true
}
app := cli.NewApp()
app.Run([]string{"-h"})
if wasCalled == false {
t.Errorf("Help printer expected to be called, but was not")
}
}
func TestAppVersionPrinter(t *testing.T) {
oldPrinter := cli.VersionPrinter
defer func() {
cli.VersionPrinter = oldPrinter
}()
var wasCalled = false
cli.VersionPrinter = func(c *cli.Context) {
wasCalled = true
}
app := cli.NewApp()
ctx := cli.NewContext(app, nil, nil)
cli.ShowVersion(ctx)
if wasCalled == false {
t.Errorf("Version printer expected to be called, but was not")
}
}
func TestAppCommandNotFound(t *testing.T) {
beforeRun, subcommandRun := false, false
app := cli.NewApp()
app.CommandNotFound = func(c *cli.Context, command string) {
beforeRun = true
}
app.Commands = []cli.Command{
cli.Command{
Name: "bar",
Action: func(c *cli.Context) {
subcommandRun = true
},
},
}
app.Run([]string{"command", "foo"})
expect(t, beforeRun, true)
expect(t, subcommandRun, false)
}
func TestGlobalFlagsInSubcommands(t *testing.T) {
subcommandRun := false
app := cli.NewApp()
app.Flags = []cli.Flag{
cli.BoolFlag{Name: "debug, d", Usage: "Enable debugging"},
}
app.Commands = []cli.Command{
cli.Command{
Name: "foo",
Subcommands: []cli.Command{
{
Name: "bar",
Action: func(c *cli.Context) {
if c.GlobalBool("debug") {
subcommandRun = true
}
},
},
},
},
}
app.Run([]string{"command", "-d", "foo", "bar"})
expect(t, subcommandRun, true)
}

View file

@ -1,13 +0,0 @@
#! /bin/bash
_cli_bash_autocomplete() {
local cur prev opts base
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
prev="${COMP_WORDS[COMP_CWORD-1]}"
opts=$( ${COMP_WORDS[@]:0:$COMP_CWORD} --generate-bash-completion )
COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
return 0
}
complete -F _cli_bash_autocomplete $PROG

Some files were not shown because too many files have changed in this diff Show more