quay/README.md

# quay

[![Build Status](https://travis-ci.com/quay/quay.svg?token=pWvEz2TeyDsVn69Hkiwq&branch=master)](https://travis-ci.com/quay/quay)
![Docker Repository on Quay](https://quay.io/repository/quay/quay/status?token=7bffbc13-8bb0-4fb4-8a70-684a0cf485d3 "Docker Repository on Quay")

:warning: The `master` branch may be in an *unstable or even broken state* during development.
Please use [releases] instead of the `master` branch in order to get stable binaries.

![Quay Logo](static/img/quay_preview.png)

Quay is project to build, store, and distribute container images.

High-level features include:

- Docker Registry Protocol [v1], [v2]
- Docker Manifest Schema [v2.1]
- [AppC Image Discovery] via on-demand transcoding
- Image Squashing via on-demand transcoding
- Authentication provided by [LDAP], [Keystone], [Dex], [Google], [GitHub]
- ACLs, team management, and auditability logs
- Geo-replicated storage provided by local filesystems, [S3], [GCS], [Swift], [Ceph]
- Continuous Integration integrated with [GitHub], [Bitbucket], [GitLab], and [git]
- Security Vulnerability Analysis via [Clair]
- [Swagger]-compliant HTTP API

[releases]: https://github.com/coreos-inc/quay/releases
[v1]: https://docs.docker.com/v1.6/reference/api/registry_api/
[v2]: https://docs.docker.com/v1.6/registry/
[v2.1]: https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-1.md
[AppC Image Discovery]: https://github.com/appc/spec/blob/master/spec/discovery.md
[LDAP]: https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol
[Keystone]: http://docs.openstack.org/developer/keystone
[Dex]: https://github.com/coreos/dex
[Google]: https://developers.google.com/identity/sign-in/web/sign-in
[GitHub]: https://developer.github.com/v3/oauth
[S3]: https://aws.amazon.com/s3
[GCS]: https://cloud.google.com/storage
[Swift]: http://swift.openstack.org
[Ceph]: http://docs.ceph.com/docs/master/radosgw/config
[GitHub]: https://github.com
[Bitbucket]: https://bitbucket.com
[GitLab]: https://gitlab.com
[git]: https://git-scm.com
[Clair]: https://github.com/coreos/clair
[Swagger]: http://swagger.io

## Table of Contents

1. **[Getting Started](#getting-started)**
  1. [macOS](#macos)
  3. [Linux](#linux)
2. **[Development](#development)**
  1. [PivotalTracker Integration](#pivotaltracker-integration)
3. **[Running and Testing](#running-and-testing)**
  1. [Test Data](#test-data)
  2. [Local Scripts](#local-scripts)
  3. [Development inside Docker](#development-inside-docker)
  4. [Adding a Python Dependency](#adding-a-python-dependency)
  5. [Adding a Yarn Dependency](#adding-a-yarn-dependency)
  6. [Running the Build System](#running-the-build-system)
  7. [To run individual tests](#to-run-individual-tests)
    1. [Pytest](#pytest)
    2. [Tox](#tox)
  8. [Running Migrations](#running-migrations)
  9. [How to run a build with tests for a push or merge](#how-to-run-a-build-with-tests-for-a-push-or-merge)
4. **[Documentation](#documentation)**
  1. [Architecture at a Glance](#architecture-at-a-glance)
  2. [Terminology](#terminology)
    1. [Organizations](#organizations)
    2. [Concepts](#concepts)
    3. [Software](#software)

## Getting Started

### macOS

macOS developers will need:

* [command line tools] or [xcode]
* [brew]

[command line tools]: https://developer.apple.com/downloads
[xcode]: https://developer.apple.com/downloads
[brew]: https://github.com/Homebrew/brew
[docker]: https://www.docker.com/docker-mac

```sh
# Download the code
git clone git@github.com:coreos-inc/quay.git && cd quay

# Install the system dependencies
brew install libevent libmagic postgresql gpgme pyenv pyenv-virtualenv node yarn

# Create a default virtualmachine for docker
docker-machine create -d virtualbox default

# Add these to ~/.bashrc or ~/.zshrc
eval "$(pyenv virtualenv-init -)"
eval "$(pyenv init -)"
eval $(/usr/local/bin/docker-machine env default)
export PYTHONPATH="."

# Some installs don't have /usr/include, required for finding SASL header files
# This command might fail because of the rootfs is read-only. Refer to the following:
# http://apple.stackexchange.com/questions/196224/unix-ln-s-command-not-permitted-in-osx-el-capitan-beta3
if [ ! -e /usr/include ]; then sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include; fi

# Install the Python dependencies
pyenv install 2.7.12
pyenv virtualenv 2.7.12 quay
pyenv activate quay
pyenv local quay

# Some packages may fail to build with clang (which now defaults to C11).
# If you're getting errors trying running again with CFLAGS='std=c99'.
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -r requirements-test.txt

# Setup a local config
git clone git@github.com:coreos-inc/quay-config.git ../quay-config
ln -s ../../quay-config/local conf/stack

# Install Node Dependencies
yarn install

# Link Typescript
yarn link typescript
```

#### Third Party Docs

* [docker](https://beta.docker.com/docs/mac/getting-started)
* [docker-machine](https://docs.docker.com/machine/install-machine)
* [pyenv](https://github.com/yyuu/pyenv)
* [pyenv-virtualenv](https://github.com/yyuu/pyenv-virtualenv)

### Docker Compose

You'll need Docker and [Docker Compose](https://docs.docker.com/compose) installed.
If you're on macOS, [Docker for Mac](https://www.docker.com/docker-mac) should include
both tools.  Otherwise, follow the docs for your platform.

You'll also need Node.js and NPM if you want to interact with the
frontend code outside a container.

Finally, you'll need a recent [Go](https://golang.org) version for the
builder.

To start Quay locally:
```sh
# Clone the various repos you'll need:
git clone https://github.com/coreos-inc/quay.git
git clone https://github.com/coreos-inc/quay-config-local.git
git clone https://github.com/coreos-inc/quay-builder.git

# Build the builder:
cd quay-builder
make build GOOS=linux

# Install NPM modules:
cd ../quay
npm install

# Build or pull images and start all Quay components:
docker-compose up
```

#### Third Party Docs

* [Docker Compose](https://docs.docker.com/compose)
* [Docker for Mac](https://www.docker.com/docker-mac)

### Linux

Do you use Linux? Send us a PR! Or use docker-compose!

## Development

### JIRA

The Quay backlog can be found in JIRA: https://jira.coreos.com/projects/QUAY

## Running and Testing

### Test Data

A SQLite database full of test data is committed to this git repository at [test/data/test.db](quay/test/data/test.db).
This database is generated by executing `python initdb.py`.
The username and password of the admin test account is `devtable` and `password`, respectively.

### Local Scripts

Running the web server locally requires [goreman](https://github.com/mattn/goreman):

```sh
go get github.com/mattn/goreman
```

* `local-run` runs the web server for testing
* `local-test` runs the unit test suite
* `yarn run build` builds front-end dependencies
* `yarn run watch` builds and watches front-end dependencies
* `yarn test` runs front-end test suite

### Development inside Docker

To build and run a development container, pass one argument to [local-docker.sh](quay/local-docker.sh):

- `buildman`: run the buildmanager
- `dev`: run web server on port 5000
- `initdb`: clear and initialize the test database
- `notifications`: run the notification worker
- `test`: run the unit test suite

### Adding a Python Dependency

```sh
# Create a new virtualenv and activate it
pyenv virtualenv 2.7.12 quay-deps
pyenv activate quay-deps

# Install unversioned dependencies with your changes
pip install -r requirements-nover.txt

# Run the unit test suite
./local-test.sh

# Freeze the versions of all of the dependencies
pip freeze > requirements.txt

# Delete the virtualenv
pyenv uninstall quay-deps
```

### Adding a Yarn Dependency

We use [Yarn](https://yarnpkg.com/) for frontend dependency management. The `yarn.lock` file ensures
that we get consistant version installs using the `yarn install` command. However, new dependencies 
should be added using `yarn add <npm package>`. This will add an entry to `package.json` and `yarn.lock`.

Occassionally there will be merge conflicts with `yarn.lock`. To resolve them, use the following (taken 
from [here](https://github.com/yarnpkg/yarn/issues/1776#issuecomment-269539948)).

```sh
git rebase origin/master
git checkout origin/master -- yarn.lock
yarn install
git add yarn.lock
git rebase --continue
```

### Running the Build System

TODO

```sh
# Run an instance of redis
docker run -d -p 6379:6379 quay.io/quay/redis
```

### To run individual tests

```sh
# To run a specific suite
TEST=true python -m test.test_api_usage -f

# To run a specific test in a suite
TEST=true python -m test.test_api_usage -f SuiteName
```
#### Pytest

```sh
# To run all tests
TEST=true PYTHONPATH="." py.test --verbose

# To run a specific test module
TEST=true PYTHONPATH="." py.test --verbose test/registry/registry_tests.py

# To run a specific test unique test
TEST=true PYTHONPATH="." py.test --verbose test/test_api_usage.py::TestDeleteNamespace

# To retry only last failed (--lf):
TEST=true PYTHONPATH="." py.test --verbose --lf

# To start pdb on failure:
TEST=true PYTHONPATH="." py.test --verbose --pdb

# To run a coverage report (html pages in ./htmlcov):
TEST=true PYTHONPATH="." py.test --cov="." --cov-report=html --cov-report=term-missing  --cov-config=.coverage.ini --verbose

# Don't capture stdout (-s)
TEST=true PYTHONPATH="." py.test --verbose -s
```

#### Tox

To create a virtualenv to run the tests.
It allows to test the code on multiple env like python2.x and python3.x or different library versions

```sh
# Test all tox env:
tox

# Add extra parameters to the pytest command:
# tox -- [pytest ARGS]
tox -- -x

# build a single env with -e:
tox -e py27-api
```

### Running migrations

```sh
# To create a new migration with this description.
# Note there might be some errors about unique id being to long
# That's okay as long as the migration file is created
./data/migrations/migration.sh "Description goes here"

# To test the up and down of the migration
./data/migrations/migration.sh # without params

# Migrations get run when you create a docker image or you can run them
# manually with the following command.
PYTHONPATH=. alembic upgrade head

# You can also rebuild your local sqlite db image from initdb.py using
# And once you have a migration you should do this and check in the
# changes to share your migration with others.
rm test/data/test.db
python initdb.py
```


### Running tests for migrations

Use AWS/RDS to create a test image.
To create a new database from a snapshot to test against see
[this](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html).

Then point the migrations to the new instance using
quay-config/local/config.yaml

Remember to run this from the root of the quay directory and to set your
python environment first.

```sh
PYTHONPATH=. alembic upgrade head
```

### How to run a build with tests for a push or merge

```sh
# Inside the quay directory.
export QUAY_TAG=quay.io/quay/quay:localtest
docker build -t $QUAY_TAG --build-arg RUN_TESTS=true .
```

## Documentation

* [Quay Enterprise Documentation](https://tectonic.com/quay-enterprise/docs/latest)
* [Quay.io Documentation](https://docs.quay.io)

### Architecture at a Glance

<img src="https://docs.google.com/a/coreos.com/drawings/d/1J-YZs7aun1lLy-1wFwIZcBma5IJmZQ8WfgtEftHCKJ0/pub?w=640&amp;h=480">

Edit chart on Google Docs at [Architecture at a Glance](https://docs.google.com/a/coreos.com/drawings/d/1J-YZs7aun1lLy-1wFwIZcBma5IJmZQ8WfgtEftHCKJ0/edit?usp=sharing).

### Terminology

#### Organizations

- **AppC**: a standards body responsible for a _Runtime_ and _Image Format_ superseded by the _Open Container Initiative_
- **Open Container Initiative**: a standards body responsible for a _Runtime_ specification and an _Image Format_
- **Docker**: a company that builds a platform that has its own _Image Formats_, _Build System_, _Container Runtime_, and _Container Orchestration_

#### Concepts

- **Image**: an archive containing all of the contents necessary to execute a container
- **Image Format**: a specification for the structure of an _Image_
- **Image Layer**: an _Image_ that may depend on being applied to other _Images_ to generate a final _Image_
- **Image Squashing**: the process of compressing an _Image_ into a single _Layer_
- **Manifest**: a text file containing metadata for a particular _Image_
- **Tag**: a human-friendly named, mutable pointer to a particular set of _Images_
- **Build System**: a program used to generate _Images_
- **Registry**: a program that speaks one or more standard protocols to store and receive _Images_
- **Repository**: a collection of related _Tags_ organized by a _Registry_
- **Push**: the act of uploading an _Image_ to a _Registry_
- **Pull**: the act of downloading an _Image_ from a _Registry_
- **Container**: an _Image_ and its execution environment
- **Container Runtime**: a program that can transform an _Image_ into a _Container_ by executing it
- **Container Orchestration**: a program or set of programs that provides a framework for deploying _Containers_

#### Software

- **Quay.io**: CoreOS's hosted _Registry_
- **Quay**: CoreOS's enterprise-grade _Registry_ product
- **quayctl**: an open source program that implements alternative methods for _pulling_ _Images_ from _Quay_
- **Clair**: an open source static analysis tool used to detect vulnerability in _Images_
- **Quay Security Scanning**: the integration between _Clair_ and _Quay_
- **Kubernetes**: an open source program implementing _Container Orchestration_
- **Docker Hub**: Docker's hosted _Registry_
- **Docker Trusted Registry**: Docker's enterprise-grade _Registry_ product
- **Notary**: an open source implementation of the TUF protocol used in _Docker Content Trust_
- **Docker Content Trust**: the integration between _Notary_ and _Docker Trusted Registry_
- **Docker Engine**: a program used to interact with all aspects of the Docker platform
- **Swarm**: a program implementing _Container Orchestration_ for the Docker platform