234 lines
8.6 KiB
Markdown
234 lines
8.6 KiB
Markdown
# quay
|
|
|
|
|
|
|
|
**Note**: The `master` branch may be in an *unstable or even broken state* during development.
|
|
Please use [releases] instead of the `master` branch in order to get stable binaries.
|
|
|
|
|
|
|
|
Quay is project to build, store, and distribute container images.
|
|
|
|
High-level features include:
|
|
|
|
- Docker Registry Protocol [v1], [v2]
|
|
- Docker Manifest Schema [v2.1]
|
|
- [AppC Image Discovery] via on-demand transcoding
|
|
- Image Squashing via on-demand transcoding
|
|
- Authentication provided by [LDAP], [Keystone], [Dex], [Google], [GitHub]
|
|
- ACLs, team management, and auditability logs
|
|
- Geo-replicated storage provided by local filesystems, [S3], [GCS], [Swift], [Ceph]
|
|
- Continuous Integration integrated with [GitHub], [Bitbucket], [GitLab], and [git]
|
|
- Security Vulnerability Analysis via [Clair]
|
|
- [Swagger]-compliant HTTP API
|
|
|
|
[releases]: https://github.com/coreos-inc/quay/releases
|
|
[v1]: https://docs.docker.com/v1.6/reference/api/registry_api/
|
|
[v2]: https://docs.docker.com/v1.6/registry/
|
|
[v2.1]: https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-1.md
|
|
[AppC Image Discovery]: https://github.com/appc/spec/blob/master/spec/discovery.md
|
|
[LDAP]: https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol
|
|
[Keystone]: http://docs.openstack.org/developer/keystone
|
|
[Dex]: https://github.com/coreos/dex
|
|
[Google]: https://developers.google.com/identity/sign-in/web/sign-in
|
|
[GitHub]: https://developer.github.com/v3/oauth
|
|
[S3]: https://aws.amazon.com/s3
|
|
[GCS]: https://cloud.google.com/storage
|
|
[Swift]: http://swift.openstack.org
|
|
[Ceph]: http://docs.ceph.com/docs/master/radosgw/config
|
|
[GitHub]: https://github.com
|
|
[Bitbucket]: https://bitbucket.com
|
|
[GitLab]: https://gitlab.com
|
|
[git]: https://git-scm.com
|
|
[Clair]: https://github.com/coreos/clair
|
|
[Swagger]: http://swagger.io
|
|
|
|
## Getting Started
|
|
|
|
### macOS
|
|
|
|
macOS developers will need:
|
|
|
|
* [command line tools] or [xcode]
|
|
* [brew]
|
|
|
|
[command line tools]: https://developer.apple.com/downloads
|
|
[xcode]: https://developer.apple.com/downloads
|
|
[brew]: https://github.com/Homebrew/brew
|
|
|
|
```
|
|
# Download the code
|
|
git clone git@github.com:coreos-inc/quay.git && cd quay
|
|
|
|
# Install the system dependencies
|
|
brew install libevent libmagic postgresql gpgme pyenv pyenv-virtualenv docker docker-machine
|
|
|
|
# create a default virtualmachine for docker
|
|
docker-machine create -d virtualbox default
|
|
|
|
# setup brew dependencies
|
|
# note you may want to add these to your bashrc or zshrc file
|
|
eval "$(pyenv virtualenv-init -)"
|
|
eval "$(pyenv init -)"
|
|
eval $(/usr/local/bin/docker-machine env default)
|
|
|
|
# Some installs don't have /usr/include, required for finding SASL header files
|
|
# http://apple.stackexchange.com/questions/196224/unix-ln-s-command-not-permitted-in-osx-el-capitan-beta3
|
|
# note this command might fail because of new OSx write protections here above is a link that explains
|
|
# how to fix that
|
|
if [ ! -e /usr/include ]; then sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include; fi
|
|
|
|
# Install the Python dependencies
|
|
pyenv install 2.7.11
|
|
pyenv virtualenv 2.7.11 quay
|
|
pyenv activate quay
|
|
|
|
# Some packages may fail to build with clang (which now defaults to C11).
|
|
CFLAGS='-std=c99' pip install -r requirements.txt
|
|
pip install -r requirements-dev.txt
|
|
|
|
# Setup a local config
|
|
git clone git@github.com:coreos-inc/quay-config.git ../quay-config
|
|
ln -s ../../quay-config/local conf/stack
|
|
```
|
|
|
|
### Useful docs
|
|
|
|
[docker](https://beta.docker.com/docs/mac/getting-started://beta.docker.com/docs/mac/getting-started)
|
|
[docker-machine](https://docs.docker.com/machine/install-machine://docs.docker.com/machine/install-machine)
|
|
[pyenv](https://github.com/yyuu/pyenv)
|
|
[pyenv-virtualenv](https://github.com/yyuu/pyenv-virtualenv)
|
|
|
|
|
|
### Linux
|
|
|
|
TODO
|
|
|
|
## Running and Testing
|
|
|
|
### Test Data
|
|
|
|
A SQLite database full of test data is committed to this git repository at `test/data/test.db`.
|
|
This database is generated by executing `python initdb.py`.
|
|
The username and password of the admin test account is `devtable` and `password`, respectively.
|
|
|
|
### Local Scripts
|
|
|
|
* `local-run` runs the web server for testing
|
|
* `local-test` runs the unit test suite
|
|
|
|
### Development inside Docker
|
|
|
|
To build and run a development container, pass one argument to local-docker.sh:
|
|
|
|
- `buildman`: run the buildmanager
|
|
- `dev`: run web server on port 5000
|
|
- `initdb`: clear and initialize the test database
|
|
- `notifications`: run the notification worker
|
|
- `test`: run the unit test suite
|
|
|
|
### Adding a Python Dependency
|
|
|
|
```
|
|
# Create a new virtualenv and activate it
|
|
pyenv virtualenv 2.7.11 quay-deps
|
|
pyenv activate quay-deps
|
|
|
|
# Install unversioned dependencies with your changes
|
|
pip install -r requirements-nover.txt
|
|
|
|
# Run the unit test suite
|
|
./local-test.sh
|
|
|
|
# Freeze the versions of all of the dependencies
|
|
pip freeze > requirements.txt
|
|
```
|
|
|
|
### Running the Build System
|
|
|
|
TODO
|
|
|
|
```
|
|
# Run an instance of redis
|
|
docker run -d -p 6379:6379 quay.io/quay/redis
|
|
```
|
|
|
|
### To run individual tests
|
|
|
|
```
|
|
# To run a specific suite
|
|
TEST=true python -m test.test_api_usage -f
|
|
|
|
# To run a specific test in a suite
|
|
TEST=true python -m test.test_api_usage -f SuiteName
|
|
```
|
|
|
|
|
|
### Running migrations
|
|
|
|
```
|
|
# To create a new migration with this description.
|
|
# Note there might be some errors about unique id being to long
|
|
# That's okay as long as the migration file is created
|
|
./data/migrations/migration.sh "Description goes here"
|
|
|
|
# To test the up and down of the migration
|
|
./data/migrations/migration.sh # without params
|
|
|
|
# Migrations get run when you create a docker image or you can run them
|
|
# manually with the following command.
|
|
PYTHONPATH=. alembic upgrade head
|
|
|
|
# You can also rebuild your local sqlite db image from initdb.py using
|
|
# And once you have a migration you should do this and check in the
|
|
# changes to share your migration with others.
|
|
rm test/data/test.db
|
|
python initdb.py
|
|
```
|
|
|
|
## Documentation
|
|
|
|
* [Quay Enterprise Documentation](https://tectonic.com/quay-enterprise/docs/latest)
|
|
* [Quay.io Documentation](https://docs.quay.io)
|
|
|
|
### [Architecture at a Glance](https://docs.google.com/a/coreos.com/drawings/d/1J-YZs7aun1lLy-1wFwIZcBma5IJmZQ8WfgtEftHCKJ0/edit?usp=sharing)
|
|
|
|
### Terminology
|
|
|
|
#### Organizations
|
|
|
|
- **AppC**: a standards body responsible for a _Runtime_ and _Image Format_ superseded by the _Open Container Initiative_
|
|
- **Open Container Initiative**: a standards body responsible for a _Runtime_ specification and an _Image Format_
|
|
- **Docker**: a company that builds a platform that has its own _Image Formats_, _Build System_, _Container Runtime_, and _Container Orchestration_
|
|
|
|
#### Concepts
|
|
|
|
- **Image**: an archive containing all of the contents necessary to execute a container
|
|
- **Image Format**: a specification for the structure of an _Image_
|
|
- **Image Layer**: an _Image_ that may depend on being applied to other _Images_ to generate a final _Image_
|
|
- **Image Squashing**: the process of compressing an _Image_ into a single _Layer_
|
|
- **Manifest**: a text file containing metadata for a particular _Image_
|
|
- **Tag**: a human-friendly named, mutable pointer to a particular set of _Images_
|
|
- **Build System**: a program used to generate _Images_
|
|
- **Registry**: a program that speaks one or more standard protocols to store and receive _Images_
|
|
- **Repository**: a collection of related _Tags_ organized by a _Registry_
|
|
- **Push**: the act of uploading an _Image_ to a _Registry_
|
|
- **Pull**: the act of downloading an _Image_ from a _Registry_
|
|
- **Container**: an _Image_ and its execution environment
|
|
- **Container Runtime**: a program that can transform an _Image_ into a _Container_ by executing it
|
|
- **Container Orchestration**: a program or set of programs that provides a framework for deploying _Containers_
|
|
|
|
#### Software
|
|
|
|
- **Quay.io**: CoreOS's hosted _Registry_
|
|
- **Quay**: CoreOS's enterprise-grade _Registry_ product
|
|
- **quayctl**: an open source program that implements alternative methods for _pulling_ _Images_ from _Quay_
|
|
- **Clair**: an open source static analysis tool used to detect vulnerability in _Images_
|
|
- **Quay Security Scanning**: the integration between _Clair_ and _Quay_
|
|
- **Kubernetes**: an open source program implementing _Container Orchestration_
|
|
- **Docker Hub**: Docker's hosted _Registry_
|
|
- **Docker Trusted Registry**: Docker's enterprise-grade _Registry_ product
|
|
- **Notary**: an open source implementation of the TUF protocol used in _Docker Content Trust_
|
|
- **Docker Content Trust**: the integration between _Notary_ and _Docker Trusted Registry_
|
|
- **Docker Engine**: a program used to interact with all aspects of the Docker platform
|
|
- **Swarm**: a program implementing _Container Orchestration_ for the Docker platform
|