also some misc cleanup
12 KiB
quay
⚠️ The master
branch may be in an unstable or even broken state during development.
Please use releases instead of the master
branch in order to get stable binaries.
Quay is project to build, store, and distribute container images.
High-level features include:
- Docker Registry Protocol v1, v2
- Docker Manifest Schema v2.1
- AppC Image Discovery via on-demand transcoding
- Image Squashing via on-demand transcoding
- Authentication provided by LDAP, Keystone, Dex, Google, GitHub
- ACLs, team management, and auditability logs
- Geo-replicated storage provided by local filesystems, S3, GCS, Swift, Ceph
- Continuous Integration integrated with GitHub, Bitbucket, GitLab, and git
- Security Vulnerability Analysis via Clair
- Swagger-compliant HTTP API
Table of Contents
- Getting Started
- macOS
- Linux
- Development
- PivotalTracker Integration
- Running and Testing
- Test Data
- Local Scripts
- Development inside Docker
- Adding a Python Dependency
- Running the Build System
- To run individual tests 1. Pytest 2. Tox
- Running Migrations
- How to run a build with tests for a push or merge
- Documentation
- Architecture at a Glance
- Terminology 1. Organizations 2. Concepts 3. Software
Getting Started
macOS
macOS developers will need:
# Download the code
git clone git@github.com:coreos-inc/quay.git && cd quay
# Install the system dependencies
brew install libevent libmagic postgresql gpgme pyenv pyenv-virtualenv pyenv-pip-rehash docker docker-machine node
# Create a default virtualmachine for docker
docker-machine create -d virtualbox default
# Add these to ~/.bashrc or ~/.zshrc
eval "$(pyenv virtualenv-init -)"
eval "$(pyenv init -)"
eval $(/usr/local/bin/docker-machine env default)
# Some installs don't have /usr/include, required for finding SASL header files
# This command might fail because of the rootfs is read-only. Refer to the following:
# http://apple.stackexchange.com/questions/196224/unix-ln-s-command-not-permitted-in-osx-el-capitan-beta3
if [ ! -e /usr/include ]; then sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include; fi
# Install the Python dependencies
pyenv install 2.7.12
pyenv virtualenv 2.7.12 quay
pyenv activate quay
pyenv local quay
# Some packages may fail to build with clang (which now defaults to C11).
# If you're getting errors trying running again with CFLAGS='std=c99'.
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -r requirements-test.txt
# Setup a local config
git clone git@github.com:coreos-inc/quay-config.git ../quay-config
ln -s ../../quay-config/local conf/stack
# Install Node Dependencies
npm install
# Link Typescript
npm link typescript
Third Party Docs
Linux
Do you use Linux? Send us a PR!
Development
PivotalTracker Integration
Tag a commit with the Tracker Story ID and GitHub automatically comments on your story with the commit message.
Add the following at the end of your commit message:
[(Finishes|Fixes|Delivers) #TRACKER_STORY_ID]
When you push to GitHub, the post-receive hook will then call back to Tracker and put a comment on the story with a link to the commit on GitHub. When tagged with "Finishes", the trigger will also click Finish on the story.
Reference: PivotalTracker blog - A Guide to GitHub’s Service Hook for Tracker
Running and Testing
Test Data
A SQLite database full of test data is committed to this git repository at test/data/test.db.
This database is generated by executing python initdb.py
.
The username and password of the admin test account is devtable
and password
, respectively.
Local Scripts
Running the web server locally requires goreman:
go get github.com/mattn/goreman
local-run
runs the web server for testinglocal-test
runs the unit test suitenpm run build
builds front end dependenciesnpm run watch
a watcher for webpack
Development inside Docker
To build and run a development container, pass one argument to local-docker.sh:
buildman
: run the buildmanagerdev
: run web server on port 5000initdb
: clear and initialize the test databasenotifications
: run the notification workertest
: run the unit test suite
Adding a Python Dependency
# Create a new virtualenv and activate it
pyenv virtualenv 2.7.11 quay-deps
pyenv activate quay-deps
# Install unversioned dependencies with your changes
pip install -r requirements-nover.txt
# Run the unit test suite
./local-test.sh
# Freeze the versions of all of the dependencies
pip freeze > requirements.txt
# Delete the virtualenv
pyenv uninstall quay-deps
Running the Build System
TODO
# Run an instance of redis
docker run -d -p 6379:6379 quay.io/quay/redis
To run individual tests
# To run a specific suite
TEST=true python -m test.test_api_usage -f
# To run a specific test in a suite
TEST=true python -m test.test_api_usage -f SuiteName
Pytest
# To run all tests
TEST=true PYTHONPATH="." py.test --verbose
# To run a specific test module
TEST=true PYTHONPATH="." py.test --verbose test/registry_tests.py
# To run a specific test unique test
TEST=true PYTHONPATH="." py.test --verbose test/test_api_usage.py::TestDeleteNamespace
# To retry only last failed (--lf):
TEST=true PYTHONPATH="." py.test --verbose --lf
# To start pdb on failure:
TEST=true PYTHONPATH="." py.test --verbose --pdb
# To run a coverage report (html pages in ./htmlcov):
TEST=true PYTHONPATH="." py.test --cov="." --cov-report=html --cov-report=term-missing --cov-config=.coverage.ini --verbose
# Don't capture stdout (-s)
TEST=true PYTHONPATH="." py.test --verbose -s
Tox
To create a virtualenv to run the tests. It allows to test the code on multiple env like python2.x and python3.x or different library versions
# Test all tox env:
tox
# Add extra parameters to the pytest command:
# tox -- [pytest ARGS]
tox -- -x
# build a single env with -e:
tox -e py27-api
Running migrations
# To create a new migration with this description.
# Note there might be some errors about unique id being to long
# That's okay as long as the migration file is created
./data/migrations/migration.sh "Description goes here"
# To test the up and down of the migration
./data/migrations/migration.sh # without params
# Migrations get run when you create a docker image or you can run them
# manually with the following command.
PYTHONPATH=. alembic upgrade head
# You can also rebuild your local sqlite db image from initdb.py using
# And once you have a migration you should do this and check in the
# changes to share your migration with others.
rm test/data/test.db
python initdb.py
Running tests for migrations
Use AWS/RDS to create a test image. To create a new database from a snapshot to test against see this.
Then point the migrations to the new instance using quay-config/local/config.yaml
Remember to run this from the root of the quay directory and to set your python environment first.
PYTHONPATH=. alembic upgrade head
How to run a build with tests for a push or merge
# Inside the quay directory.
export QUAY_TAG=quay.io/quay/quay:localtest
docker build -t $QUAY_TAG --build-arg RUN_TESTS=true .
Documentation
Architecture at a Glance
Edit chart on Google Docs at Architecture at a Glance.
Terminology
Organizations
- AppC: a standards body responsible for a Runtime and Image Format superseded by the Open Container Initiative
- Open Container Initiative: a standards body responsible for a Runtime specification and an Image Format
- Docker: a company that builds a platform that has its own Image Formats, Build System, Container Runtime, and Container Orchestration
Concepts
- Image: an archive containing all of the contents necessary to execute a container
- Image Format: a specification for the structure of an Image
- Image Layer: an Image that may depend on being applied to other Images to generate a final Image
- Image Squashing: the process of compressing an Image into a single Layer
- Manifest: a text file containing metadata for a particular Image
- Tag: a human-friendly named, mutable pointer to a particular set of Images
- Build System: a program used to generate Images
- Registry: a program that speaks one or more standard protocols to store and receive Images
- Repository: a collection of related Tags organized by a Registry
- Push: the act of uploading an Image to a Registry
- Pull: the act of downloading an Image from a Registry
- Container: an Image and its execution environment
- Container Runtime: a program that can transform an Image into a Container by executing it
- Container Orchestration: a program or set of programs that provides a framework for deploying Containers
Software
- Quay.io: CoreOS's hosted Registry
- Quay: CoreOS's enterprise-grade Registry product
- quayctl: an open source program that implements alternative methods for pulling Images from Quay
- Clair: an open source static analysis tool used to detect vulnerability in Images
- Quay Security Scanning: the integration between Clair and Quay
- Kubernetes: an open source program implementing Container Orchestration
- Docker Hub: Docker's hosted Registry
- Docker Trusted Registry: Docker's enterprise-grade Registry product
- Notary: an open source implementation of the TUF protocol used in Docker Content Trust
- Docker Content Trust: the integration between Notary and Docker Trusted Registry
- Docker Engine: a program used to interact with all aspects of the Docker platform
- Swarm: a program implementing Container Orchestration for the Docker platform