quay/README.md

# quay

![Docker Repository on Quay](https://quay.io/repository/quay/quay/status?token=7bffbc13-8bb0-4fb4-8a70-684a0cf485d3 "Docker Repository on Quay")

**Note**: The `master` branch may be in an *unstable or even broken state* during development.
Please use [releases] instead of the `master` branch in order to get stable binaries.

![Quay Logo](static/img/quay_preview.png)

Quay is project to build, store, and distribute container images.

High-level features include:

- Docker Registry Protocol [v1], [v2]
- Docker Manifest Schema [v2.1]
- [AppC Image Discovery] via on-demand transcoding
- Image Squashing via on-demand transcoding
- Authentication provided by [LDAP], [Keystone], [Dex], [Google], [GitHub]
- ACLs, team management, and auditability logs
- Geo-replicated storage provided by local filesystems, [S3], [GCS], [Swift], [Ceph]
- Continuous Integration integrated with [GitHub], [Bitbucket], [GitLab], and [git]
- Security Vulnerability Analysis via [Clair]
- [Swagger]-compliant HTTP API

[releases]: https://github.com/coreos-inc/quay/releases
[v1]: https://docs.docker.com/v1.6/reference/api/registry_api/
[v2]: https://docs.docker.com/v1.6/registry/
[v2.1]: https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-1.md
[AppC Image Discovery]: https://github.com/appc/spec/blob/master/spec/discovery.md
[LDAP]: https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol
[Keystone]: http://docs.openstack.org/developer/keystone
[Dex]: https://github.com/coreos/dex
[Google]: https://developers.google.com/identity/sign-in/web/sign-in
[GitHub]: https://developer.github.com/v3/oauth
[S3]: https://aws.amazon.com/s3
[GCS]: https://cloud.google.com/storage
[Swift]: http://swift.openstack.org
[Ceph]: http://docs.ceph.com/docs/master/radosgw/config
[GitHub]: https://github.com
[Bitbucket]: https://bitbucket.com
[GitLab]: https://gitlab.com
[git]: https://git-scm.com
[Clair]: https://github.com/coreos/clair
[Swagger]: http://swagger.io

## Getting Started

### macOS

macOS developers will need:

* [command line tools] or [xcode]
* [brew]

[command line tools]: https://developer.apple.com/downloads
[xcode]: https://developer.apple.com/downloads
[brew]: https://github.com/Homebrew/brew

```
# Download the code
git clone git@github.com:coreos-inc/quay.git && cd quay

# Install the system dependencies
brew install libevent libmagic postgresql gpgme pyenv pyenv-virtualenv docker docker-machine

# create a default virtualmachine for docker
docker-machine create -d virtualbox default

# setup brew dependencies
# note you may want to add these to your bashrc or zshrc file
eval "$(pyenv virtualenv-init -)"
eval "$(pyenv init -)"
eval $(/usr/local/bin/docker-machine env default)

# Some installs don't have /usr/include, required for finding SASL header files
# http://apple.stackexchange.com/questions/196224/unix-ln-s-command-not-permitted-in-osx-el-capitan-beta3
# note this command might fail because of new OSx write protections here above is a link that explains
# how to fix that
if [ ! -e /usr/include ]; then sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include; fi

# Install the Python dependencies
pyenv install 2.7.11
pyenv virtualenv 2.7.11 quay
pyenv activate quay

# Some packages may fail to build with clang (which now defaults to C11).
CFLAGS='-std=c99' pip install -r requirements.txt
pip install -r requirements-dev.txt

# Setup a local config
git clone git@github.com:coreos-inc/quay-config.git ../quay-config
ln -s ../../quay-config/local conf/stack
```

### Useful docs

[docker](https://beta.docker.com/docs/mac/getting-started://beta.docker.com/docs/mac/getting-started)
[docker-machine](https://docs.docker.com/machine/install-machine://docs.docker.com/machine/install-machine)
[pyenv](https://github.com/yyuu/pyenv)
[pyenv-virtualenv](https://github.com/yyuu/pyenv-virtualenv)


### Linux

TODO

## Running and Testing

### Test Data

A SQLite database full of test data is committed to this git repository at `test/data/test.db`.
This database is generated by executing `python initdb.py`.
The username and password of the admin test account is `devtable` and `password`, respectively.

### Local Scripts

* `local-run` runs the web server for testing
* `local-test` runs the unit test suite

### Development inside Docker

To build and run a development container, pass one argument to local-docker.sh:

- `buildman`: run the buildmanager
- `dev`: run web server on port 5000
- `initdb`: clear and initialize the test database
- `notifications`: run the notification worker
- `test`: run the unit test suite

### Adding a Python Dependency

```
# Create a new virtualenv and activate it
pyenv virtualenv 2.7.11 quay-deps
pyenv activate quay-deps

# Install unversioned dependencies with your changes
pip install -r requirements-nover.txt

# Run the unit test suite
./local-test.sh

# Freeze the versions of all of the dependencies
pip freeze > requirements.txt
```

### Running the Build System

TODO

```
# Run an instance of redis
docker run -d -p 6379:6379 quay.io/quay/redis
```

### To run individual tests

```
# To run a specific suite
TEST=true python -m test.test_api_usage -f

# To run a specific test in a suite
TEST=true python -m test.test_api_usage -f SuiteName
```


### Running migrations

```
# To create a new migration with this description.
# Note there might be some errors about unique id being to long
# That's okay as long as the migration file is created
./data/migrations/migration.sh "Description goes here"

# To test the up and down of the migration
./data/migrations/migration.sh # without params

# Migrations get run when you create a docker image or you can run them
# manually with the following command.
PYTHONPATH=. alembic upgrade head

# You can also rebuild your local sqlite db image from initdb.py using
rm test/data/test.db
python initdb.py
```

## Documentation

* [Quay Enterprise Documentation](https://tectonic.com/quay-enterprise/docs/latest)
* [Quay.io Documentation](https://docs.quay.io)

### Architecture at a Glance

TODO

### Terminology

#### Organizations

- **AppC**: a standards body responsible for a _Runtime_ and _Image Format_ superseded by the _Open Container Initiative_
- **Open Container Initiative**: a standards body responsible for a _Runtime_ specification and an _Image Format_
- **Docker**: a company that builds a platform that has its own _Image Formats_, _Build System_, _Container Runtime_, and _Container Orchestration_

#### Concepts

- **Image**: an archive containing all of the contents necessary to execute a container
- **Image Format**: a specification for the structure of an _Image_
- **Image Layer**: an _Image_ that may depend on being applied to other _Images_ to generate a final _Image_
- **Image Squashing**: the process of compressing an _Image_ into a single _Layer_
- **Manifest**: a text file containing metadata for a particular _Image_
- **Tag**: a human-friendly named, mutable pointer to a particular set of _Images_
- **Build System**: a program used to generate _Images_
- **Registry**: a program that speaks one or more standard protocols to store and receive _Images_
- **Repository**: a collection of related _Tags_ organized by a _Registry_
- **Push**: the act of uploading an _Image_ to a _Registry_
- **Pull**: the act of downloading an _Image_ from a _Registry_
- **Container**: an _Image_ and its execution environment
- **Container Runtime**: a program that can transform an _Image_ into a _Container_ by executing it
- **Container Orchestration**: a program or set of programs that provides a framework for deploying _Containers_

#### Software

- **Quay.io**: CoreOS's hosted _Registry_
- **Quay**: CoreOS's enterprise-grade _Registry_ product
- **quayctl**: an open source program that implements alternative methods for _pulling_ _Images_ from _Quay_
- **Clair**: an open source static analysis tool used to detect vulnerability in _Images_
- **Quay Security Scanning**: the integration between _Clair_ and _Quay_
- **Kubernetes**: an open source program implementing _Container Orchestration_
- **Docker Hub**: Docker's hosted _Registry_
- **Docker Trusted Registry**: Docker's enterprise-grade _Registry_ product
- **Notary**: an open source implementation of the TUF protocol used in _Docker Content Trust_
- **Docker Content Trust**: the integration between _Notary_ and _Docker Trusted Registry_
- **Docker Engine**: a program used to interact with all aspects of the Docker platform
- **Swarm**: a program implementing _Container Orchestration_ for the Docker platform