# quay ![Docker Repository on Quay](https://quay.io/repository/quay/quay/status?token=7bffbc13-8bb0-4fb4-8a70-684a0cf485d3 "Docker Repository on Quay") :warning: The `master` branch may be in an *unstable or even broken state* during development. Please use [releases] instead of the `master` branch in order to get stable binaries. ![Quay Logo](static/img/quay_preview.png) Quay is project to build, store, and distribute container images. High-level features include: - Docker Registry Protocol [v1], [v2] - Docker Manifest Schema [v2.1] - [AppC Image Discovery] via on-demand transcoding - Image Squashing via on-demand transcoding - Authentication provided by [LDAP], [Keystone], [Dex], [Google], [GitHub] - ACLs, team management, and auditability logs - Geo-replicated storage provided by local filesystems, [S3], [GCS], [Swift], [Ceph] - Continuous Integration integrated with [GitHub], [Bitbucket], [GitLab], and [git] - Security Vulnerability Analysis via [Clair] - [Swagger]-compliant HTTP API [releases]: https://github.com/coreos-inc/quay/releases [v1]: https://docs.docker.com/v1.6/reference/api/registry_api/ [v2]: https://docs.docker.com/v1.6/registry/ [v2.1]: https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-1.md [AppC Image Discovery]: https://github.com/appc/spec/blob/master/spec/discovery.md [LDAP]: https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol [Keystone]: http://docs.openstack.org/developer/keystone [Dex]: https://github.com/coreos/dex [Google]: https://developers.google.com/identity/sign-in/web/sign-in [GitHub]: https://developer.github.com/v3/oauth [S3]: https://aws.amazon.com/s3 [GCS]: https://cloud.google.com/storage [Swift]: http://swift.openstack.org [Ceph]: http://docs.ceph.com/docs/master/radosgw/config [GitHub]: https://github.com [Bitbucket]: https://bitbucket.com [GitLab]: https://gitlab.com [git]: https://git-scm.com [Clair]: https://github.com/coreos/clair [Swagger]: http://swagger.io ## Table of Contents 1. **[Getting Started](#getting-started)** 1. [macOS](#macos) 3. [Linux](#linux) 2. **[Development](#development)** 1. [PivotalTracker Integration](#pivotaltracker-integration) 3. **[Running and Testing](#running-and-testing)** 1. [Test Data](#test-data) 2. [Local Scripts](#local-scripts) 3. [Development inside Docker](#development-inside-docker) 4. [Adding a Python Dependency](#adding-a-python-dependency) 5. [Running the Build System](#running-the-build-system) 6. [To run individual tests](#to-run-individual-tests) 1. [Pytest](#pytest) 2. [Tox](#tox) 7. [Running Migrations](#running-migrations) 8. [How to run a build with tests for a push or merge](#how-to-run-a-build-with-tests-for-a-push-or-merge) 4. **[Documentation](#documentation)** 1. [Architecture at a Glance](#architecture-at-a-glance) 2. [Terminology](#terminology) 1. [Organizations](#organizations) 2. [Concepts](#concepts) 3. [Software](#software) ## Getting Started ### macOS macOS developers will need: * [command line tools] or [xcode] * [brew] [command line tools]: https://developer.apple.com/downloads [xcode]: https://developer.apple.com/downloads [brew]: https://github.com/Homebrew/brew ``` # Download the code git clone git@github.com:coreos-inc/quay.git && cd quay # Install the system dependencies brew install libevent libmagic postgresql gpgme pyenv pyenv-virtualenv docker docker-machine node # create a default virtualmachine for docker docker-machine create -d virtualbox default # setup brew dependencies # note you may want to add these to your bashrc or zshrc file eval "$(pyenv virtualenv-init -)" eval "$(pyenv init -)" eval $(/usr/local/bin/docker-machine env default) # Some installs don't have /usr/include, required for finding SASL header files # http://apple.stackexchange.com/questions/196224/unix-ln-s-command-not-permitted-in-osx-el-capitan-beta3 # note this command might fail because of new OSx write protections here above is a link that explains # how to fix that if [ ! -e /usr/include ]; then sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include; fi # Install the Python dependencies pyenv install 2.7.11 pyenv virtualenv 2.7.11 quay pyenv activate quay # Some packages may fail to build with clang (which now defaults to C11). CFLAGS='-std=c99' pip install -r requirements.txt pip install -r requirements-dev.txt # Setup a local config git clone git@github.com:coreos-inc/quay-config.git ../quay-config ln -s ../../quay-config/local conf/stack # Install Node Dependencies npm install # Link Typescript npm link typescript ``` #### Useful docs * [docker](https://beta.docker.com/docs/mac/getting-started://beta.docker.com/docs/mac/getting-started) * [docker-machine](https://docs.docker.com/machine/install-machine://docs.docker.com/machine/install-machine) * [pyenv](https://github.com/yyuu/pyenv) * [pyenv-virtualenv](https://github.com/yyuu/pyenv-virtualenv) ### Linux TODO ## Development ### PivotalTracker Integration Tag a commit with the Tracker Story ID and GitHub automatically comments on your story with the commit message. Add the following at the end of your commit message: ``` [(Finishes|Fixes|Delivers) #TRACKER_STORY_ID] ``` When you push to GitHub, the post-receive hook will then call back to Tracker and put a comment on the story with a link to the commit on GitHub. When tagged with "Finishes", the trigger will also click Finish on the story. Reference: PivotalTracker blog - [A Guide to GitHub’s Service Hook for Tracker](https://www.pivotaltracker.com/blog/guide-githubs-service-hook-tracker/) ## Running and Testing ### Test Data A SQLite database full of test data is committed to this git repository at [test/data/test.db](quay/test/data/test.db). This database is generated by executing `python initdb.py`. The username and password of the admin test account is `devtable` and `password`, respectively. ### Local Scripts Running the web server locally requires [goreman](https://github.com/mattn/goreman): ``` go get github.com/mattn/goreman ``` * `local-run` runs the web server for testing * `local-test` runs the unit test suite * `npm run build` builds front end dependencies * `npm run watch` a watcher for webpack ### Development inside Docker To build and run a development container, pass one argument to [local-docker.sh](quay/local-docker.sh): - `buildman`: run the buildmanager - `dev`: run web server on port 5000 - `initdb`: clear and initialize the test database - `notifications`: run the notification worker - `test`: run the unit test suite ### Adding a Python Dependency ``` # Create a new virtualenv and activate it pyenv virtualenv 2.7.11 quay-deps pyenv activate quay-deps # Install unversioned dependencies with your changes pip install -r requirements-nover.txt # Run the unit test suite ./local-test.sh # Freeze the versions of all of the dependencies pip freeze > requirements.txt ``` ### Running the Build System TODO ``` # Run an instance of redis docker run -d -p 6379:6379 quay.io/quay/redis ``` ### To run individual tests ``` # To run a specific suite TEST=true python -m test.test_api_usage -f # To run a specific test in a suite TEST=true python -m test.test_api_usage -f SuiteName ``` #### Pytest ``` # To run all tests TEST=true PYTHONPATH="." py.test --verbose test/ # To run a specific test module TEST=true PYTHONPATH="." py.test --verbose test/registry_tests.py # To run a specific test unique test TEST=true PYTHONPATH="." py.test --verbose test/test_api_usage.py::TestDeleteNamespace # To retry only last failed (--lf): TEST=true PYTHONPATH="." py.test --verbose --lf # To start pdb on failure: TEST=true PYTHONPATH="." py.test --verbose --pdb # To run a coverage report (html pages in ./htmlcov): TEST=true PYTHONPATH="." py.test --cov="." --cov-report=html --cov-report=term-missing --cov-config=.coverage.ini --verbose # Don't capture stdout (-s) TEST=true PYTHONPATH="." py.test --verbose -s ``` #### Tox To create a virtualenv to run the tests. It allows to test the code on multiple env like python2.x and python3.x or different library versions ``` # Test all tox env: tox # Add extra parameters to the pytest command: # tox -- [pytest ARGS] tox -- -x # build a single env with -e: tox -e py27-api ``` ### Running migrations ``` # To create a new migration with this description. # Note there might be some errors about unique id being to long # That's okay as long as the migration file is created ./data/migrations/migration.sh "Description goes here" # To test the up and down of the migration ./data/migrations/migration.sh # without params # Migrations get run when you create a docker image or you can run them # manually with the following command. PYTHONPATH=. alembic upgrade head # You can also rebuild your local sqlite db image from initdb.py using # And once you have a migration you should do this and check in the # changes to share your migration with others. rm test/data/test.db python initdb.py ``` ### How to run a build with tests for a push or merge ``` # Inside the quay directory. export QUAY_TAG=quay.io/quay/quay:localtest docker build -t $QUAY_TAG --build-arg RUN_TESTS=true . ``` ## Documentation * [Quay Enterprise Documentation](https://tectonic.com/quay-enterprise/docs/latest) * [Quay.io Documentation](https://docs.quay.io) ### Architecture at a Glance Edit chart on Google Docs at [Architecture at a Glance](https://docs.google.com/a/coreos.com/drawings/d/1J-YZs7aun1lLy-1wFwIZcBma5IJmZQ8WfgtEftHCKJ0/edit?usp=sharing). ### Terminology #### Organizations - **AppC**: a standards body responsible for a _Runtime_ and _Image Format_ superseded by the _Open Container Initiative_ - **Open Container Initiative**: a standards body responsible for a _Runtime_ specification and an _Image Format_ - **Docker**: a company that builds a platform that has its own _Image Formats_, _Build System_, _Container Runtime_, and _Container Orchestration_ #### Concepts - **Image**: an archive containing all of the contents necessary to execute a container - **Image Format**: a specification for the structure of an _Image_ - **Image Layer**: an _Image_ that may depend on being applied to other _Images_ to generate a final _Image_ - **Image Squashing**: the process of compressing an _Image_ into a single _Layer_ - **Manifest**: a text file containing metadata for a particular _Image_ - **Tag**: a human-friendly named, mutable pointer to a particular set of _Images_ - **Build System**: a program used to generate _Images_ - **Registry**: a program that speaks one or more standard protocols to store and receive _Images_ - **Repository**: a collection of related _Tags_ organized by a _Registry_ - **Push**: the act of uploading an _Image_ to a _Registry_ - **Pull**: the act of downloading an _Image_ from a _Registry_ - **Container**: an _Image_ and its execution environment - **Container Runtime**: a program that can transform an _Image_ into a _Container_ by executing it - **Container Orchestration**: a program or set of programs that provides a framework for deploying _Containers_ #### Software - **Quay.io**: CoreOS's hosted _Registry_ - **Quay**: CoreOS's enterprise-grade _Registry_ product - **quayctl**: an open source program that implements alternative methods for _pulling_ _Images_ from _Quay_ - **Clair**: an open source static analysis tool used to detect vulnerability in _Images_ - **Quay Security Scanning**: the integration between _Clair_ and _Quay_ - **Kubernetes**: an open source program implementing _Container Orchestration_ - **Docker Hub**: Docker's hosted _Registry_ - **Docker Trusted Registry**: Docker's enterprise-grade _Registry_ product - **Notary**: an open source implementation of the TUF protocol used in _Docker Content Trust_ - **Docker Content Trust**: the integration between _Notary_ and _Docker Trusted Registry_ - **Docker Engine**: a program used to interact with all aspects of the Docker platform - **Swarm**: a program implementing _Container Orchestration_ for the Docker platform