No description
This repository has been archived on 2020-03-24. You can view files and clone it, but cannot push or open issues or pull requests.
Find a file
josephschorr 4a90ebb901 Merge pull request #2397 from coreos-inc/qss-initial-notify
Batch QSS notifications after initial scan
2017-03-01 16:32:45 -05:00
.github feat(.github): add PR template 2017-02-01 15:59:32 -05:00
auth registry auth tests: test more access types 2016-11-28 14:02:08 -05:00
avatars fix(136521333): Handle None email_or_id in avatar code 2016-12-21 15:00:55 -05:00
buildman Fix removal of component from session factory 2017-02-23 17:15:47 -05:00
buildstatus Adding in UI for cancel anytime. 2016-11-21 10:58:32 -05:00
buildtrigger Catch SSL errors due to timeouts in Github calls 2017-02-22 13:20:04 -05:00
ci ConcourseCI: allow builds from forks 2017-02-06 15:36:58 -08:00
conf fix typo 2017-02-23 19:03:26 -05:00
data Fix health check 2017-02-24 12:23:18 -05:00
digest Merge branch 'newchanges' into python-registry-v2 2015-11-06 18:24:32 -05:00
emails emails: add missing periods to sentences in emails 2016-08-09 15:36:47 -07:00
endpoints Make QSS multiple notification messaging nicer 2017-03-01 16:11:11 -05:00
events Less verbose notifications for QSS 2016-10-10 15:18:49 -04:00
features Strip whitespace from ALL the things. 2014-11-24 16:07:38 -05:00
grunt retrieve if needed 2017-01-25 16:11:19 -08:00
health Add a configuration flag for external TLS termination 2016-09-22 18:28:57 -04:00
image rearrange(appc/test): use test_*.py naming convention 2017-01-12 11:33:47 -05:00
oauth Move OAuth login into its own endpoints module 2017-02-16 16:27:54 -05:00
static Merge pull request #2371 from coreos-inc/entity-search-fix 2017-02-24 11:27:51 -05:00
storage feature(app.py): adding queue_metrics to queues 2017-02-14 16:01:28 -05:00
templates moved Angular routes to separate module; load Webpack bundle before other main scripts 2017-01-20 16:24:55 -08:00
test Make QSS multiple notification messaging nicer 2017-03-01 16:11:11 -05:00
tools Read the number of unscanned clair images from the block allocator 2017-02-21 19:13:51 -05:00
util Batch QSS notifications after initial scan 2017-03-01 15:42:49 -05:00
workers Make the security scanning worker period configurable 2017-02-27 15:02:29 -05:00
.dockerignore Merge pull request #2327 from coreos-inc/FEAT-github-templates 2017-02-27 21:05:50 -05:00
.gitignore moved Webpack bundle directory out of /static/js because it contains more than just JS files 2017-01-24 14:05:06 -08:00
alembic.ini Print the time when logging from migrations 2015-11-09 22:29:17 -05:00
app.py feature(app.py): adding queue_metrics to queues 2017-02-14 16:01:28 -05:00
application.py Split secscan endpoints into a new process 2016-05-02 11:38:00 -04:00
boot.py Use the instance service key for registry JWT signing 2016-06-07 11:58:10 -04:00
build.sh Update build.sh to work with GIT_HEAD 2015-11-04 16:55:34 -05:00
CHANGELOG.md Update 2.1.0 changelog with reference to new doc 2017-02-22 14:13:26 -05:00
config.py Make the security scanning worker period configurable 2017-02-27 15:02:29 -05:00
dev.df Refactor prometheus integration 2016-07-01 14:16:50 -04:00
Dockerfile alphabetized Dockerfile 2017-02-27 22:02:40 -08:00
external_libraries.py Add proper and tested OIDC support on the server 2017-01-23 17:53:34 -05:00
initdb.py chore(initdb): allow overriding logging level with env var 2017-02-23 15:01:37 -05:00
karma.conf.js fixes based on requested changes 2017-02-01 00:29:56 -08:00
local-docker.sh local-docker.sh now accepts env vars 2016-07-08 15:50:51 -04:00
local-run.sh improvements to startup logo 2017-02-13 00:48:17 -08:00
local-test.sh use kwargs for parse_repository_name 2016-03-09 16:20:28 -05:00
MAINTAINERS MAINTAINERS: initial commit 2015-08-04 21:39:33 -07:00
Makefile feat(Makefile): add makefile to repo (#2264) 2017-02-15 08:39:19 -08:00
package.json removed all traces of npm 2017-02-27 17:02:30 -08:00
path_converters.py update(security_test.py): moving tests to new framework 2017-02-02 13:40:00 -05:00
Procfile Run gunicorn and webpack through Procfile 2016-10-20 13:50:57 -04:00
pylintrc database: initial manifestlist schema changes 2016-09-26 14:47:06 -04:00
README.md updated readme 2017-02-27 19:37:12 -08:00
registry.py Merge remote-tracking branch 'upstream/v2-phase4' into python-registry-v2 2015-10-22 16:59:28 -04:00
release.py superuser: add storage replication config 2015-11-09 17:34:22 -05:00
requirements-dev.txt reqs: add ipdb to dev 2016-05-27 14:44:30 -04:00
requirements-nover.txt Pull out redis validation into validator class 2017-02-24 12:23:15 -05:00
requirements-tests.txt fix(Dockerfile): add timeout, remove auto distmode 2016-12-28 14:31:28 -05:00
requirements.txt Pull out redis validation into validator class 2017-02-24 12:23:15 -05:00
ROADMAP.md Update the roadmap for current sprint and strip from QE 2016-03-04 11:29:51 -05:00
secscan.py Split secscan endpoints into a new process 2016-05-02 11:38:00 -04:00
setup.cfg fix(setup.cfg): ignore src dir for tests 2017-02-09 16:10:31 -08:00
tox.ini Fix tox definition file for registry tests 2017-01-18 14:31:43 -05:00
tsconfig.json @Inject decorator for easily annotating dependency-injected class constructors 2017-01-21 02:56:29 -08:00
tslint.json Namespace page directories 2016-10-31 16:35:11 -04:00
typings.json Setup skeleton files for new front end build tool (Node, Webpack, React, Typescript) 2016-10-18 17:13:56 -04:00
verbs.py Add DEBUGLOG flag for enabling full debug logs in all gunicorn components 2015-10-13 10:45:49 -04:00
web.py Move OAuth login into its own endpoints module 2017-02-16 16:27:54 -05:00
webpack.config.js update Webpack to 2.2 2017-02-06 14:05:19 -08:00
yarn.lock Replace 'npm install' with 'yarn install' 2017-02-27 14:36:17 -08:00

quay

Docker Repository on Quay

⚠️ The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order to get stable binaries.

Quay Logo

Quay is project to build, store, and distribute container images.

High-level features include:

Table of Contents

  1. Getting Started
  2. macOS
  3. Linux
  4. Development
  5. PivotalTracker Integration
  6. Running and Testing
  7. Test Data
  8. Local Scripts
  9. Development inside Docker
  10. Adding a Python Dependency
  11. Running the Build System
  12. To run individual tests 1. Pytest 2. Tox
  13. Running Migrations
  14. How to run a build with tests for a push or merge
  15. Documentation
  16. Architecture at a Glance
  17. Terminology 1. Organizations 2. Concepts 3. Software

Getting Started

macOS

macOS developers will need:

# Download the code
git clone git@github.com:coreos-inc/quay.git && cd quay

# Install the system dependencies
brew install libevent libmagic postgresql gpgme pyenv pyenv-virtualenv pyenv-pip-rehash docker docker-machine node yarn

# Create a default virtualmachine for docker
docker-machine create -d virtualbox default

# Add these to ~/.bashrc or ~/.zshrc
eval "$(pyenv virtualenv-init -)"
eval "$(pyenv init -)"
eval $(/usr/local/bin/docker-machine env default)

# Some installs don't have /usr/include, required for finding SASL header files
# This command might fail because of the rootfs is read-only. Refer to the following:
# http://apple.stackexchange.com/questions/196224/unix-ln-s-command-not-permitted-in-osx-el-capitan-beta3
if [ ! -e /usr/include ]; then sudo ln -s `xcrun --show-sdk-path`/usr/include /usr/include; fi

# Install the Python dependencies
pyenv install 2.7.12
pyenv virtualenv 2.7.12 quay
pyenv activate quay
pyenv local quay

# Some packages may fail to build with clang (which now defaults to C11).
# If you're getting errors trying running again with CFLAGS='std=c99'.
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -r requirements-test.txt

# Setup a local config
git clone git@github.com:coreos-inc/quay-config.git ../quay-config
ln -s ../../quay-config/local conf/stack

# Install Node Dependencies
yarn install

# Link Typescript
yarn link typescript

Third Party Docs

Linux

Do you use Linux? Send us a PR!

Development

PivotalTracker Integration

Tag a commit with the Tracker Story ID and GitHub automatically comments on your story with the commit message.

Add the following at the end of your commit message:

[(Finishes|Fixes|Delivers) #TRACKER_STORY_ID]

When you push to GitHub, the post-receive hook will then call back to Tracker and put a comment on the story with a link to the commit on GitHub. When tagged with "Finishes", the trigger will also click Finish on the story.

Reference: PivotalTracker blog - A Guide to GitHubs Service Hook for Tracker

Running and Testing

Test Data

A SQLite database full of test data is committed to this git repository at test/data/test.db. This database is generated by executing python initdb.py. The username and password of the admin test account is devtable and password, respectively.

Local Scripts

Running the web server locally requires goreman:

go get github.com/mattn/goreman
  • local-run runs the web server for testing
  • local-test runs the unit test suite
  • yarn run build builds front-end dependencies
  • yarn run watch builds and watches front-end dependencies
  • yarn test runs front-end test suite

Development inside Docker

To build and run a development container, pass one argument to local-docker.sh:

  • buildman: run the buildmanager
  • dev: run web server on port 5000
  • initdb: clear and initialize the test database
  • notifications: run the notification worker
  • test: run the unit test suite

Adding a Python Dependency

# Create a new virtualenv and activate it
pyenv virtualenv 2.7.12 quay-deps
pyenv activate quay-deps

# Install unversioned dependencies with your changes
pip install -r requirements-nover.txt

# Run the unit test suite
./local-test.sh

# Freeze the versions of all of the dependencies
pip freeze > requirements.txt

# Delete the virtualenv
pyenv uninstall quay-deps

Running the Build System

TODO

# Run an instance of redis
docker run -d -p 6379:6379 quay.io/quay/redis

To run individual tests

# To run a specific suite
TEST=true python -m test.test_api_usage -f

# To run a specific test in a suite
TEST=true python -m test.test_api_usage -f SuiteName

Pytest

# To run all tests
TEST=true PYTHONPATH="." py.test --verbose

# To run a specific test module
TEST=true PYTHONPATH="." py.test --verbose test/registry_tests.py

# To run a specific test unique test
TEST=true PYTHONPATH="." py.test --verbose test/test_api_usage.py::TestDeleteNamespace

# To retry only last failed (--lf):
TEST=true PYTHONPATH="." py.test --verbose --lf

# To start pdb on failure:
TEST=true PYTHONPATH="." py.test --verbose --pdb

# To run a coverage report (html pages in ./htmlcov):
TEST=true PYTHONPATH="." py.test --cov="." --cov-report=html --cov-report=term-missing  --cov-config=.coverage.ini --verbose

# Don't capture stdout (-s)
TEST=true PYTHONPATH="." py.test --verbose -s

Tox

To create a virtualenv to run the tests. It allows to test the code on multiple env like python2.x and python3.x or different library versions

# Test all tox env:
tox

# Add extra parameters to the pytest command:
# tox -- [pytest ARGS]
tox -- -x

# build a single env with -e:
tox -e py27-api

Running migrations

# To create a new migration with this description.
# Note there might be some errors about unique id being to long
# That's okay as long as the migration file is created
./data/migrations/migration.sh "Description goes here"

# To test the up and down of the migration
./data/migrations/migration.sh # without params

# Migrations get run when you create a docker image or you can run them
# manually with the following command.
PYTHONPATH=. alembic upgrade head

# You can also rebuild your local sqlite db image from initdb.py using
# And once you have a migration you should do this and check in the
# changes to share your migration with others.
rm test/data/test.db
python initdb.py

Running tests for migrations

Use AWS/RDS to create a test image. To create a new database from a snapshot to test against see this.

Then point the migrations to the new instance using quay-config/local/config.yaml

Remember to run this from the root of the quay directory and to set your python environment first.

PYTHONPATH=. alembic upgrade head

How to run a build with tests for a push or merge

# Inside the quay directory.
export QUAY_TAG=quay.io/quay/quay:localtest
docker build -t $QUAY_TAG --build-arg RUN_TESTS=true .

Documentation

Architecture at a Glance

Edit chart on Google Docs at Architecture at a Glance.

Terminology

Organizations

  • AppC: a standards body responsible for a Runtime and Image Format superseded by the Open Container Initiative
  • Open Container Initiative: a standards body responsible for a Runtime specification and an Image Format
  • Docker: a company that builds a platform that has its own Image Formats, Build System, Container Runtime, and Container Orchestration

Concepts

  • Image: an archive containing all of the contents necessary to execute a container
  • Image Format: a specification for the structure of an Image
  • Image Layer: an Image that may depend on being applied to other Images to generate a final Image
  • Image Squashing: the process of compressing an Image into a single Layer
  • Manifest: a text file containing metadata for a particular Image
  • Tag: a human-friendly named, mutable pointer to a particular set of Images
  • Build System: a program used to generate Images
  • Registry: a program that speaks one or more standard protocols to store and receive Images
  • Repository: a collection of related Tags organized by a Registry
  • Push: the act of uploading an Image to a Registry
  • Pull: the act of downloading an Image from a Registry
  • Container: an Image and its execution environment
  • Container Runtime: a program that can transform an Image into a Container by executing it
  • Container Orchestration: a program or set of programs that provides a framework for deploying Containers

Software

  • Quay.io: CoreOS's hosted Registry
  • Quay: CoreOS's enterprise-grade Registry product
  • quayctl: an open source program that implements alternative methods for pulling Images from Quay
  • Clair: an open source static analysis tool used to detect vulnerability in Images
  • Quay Security Scanning: the integration between Clair and Quay
  • Kubernetes: an open source program implementing Container Orchestration
  • Docker Hub: Docker's hosted Registry
  • Docker Trusted Registry: Docker's enterprise-grade Registry product
  • Notary: an open source implementation of the TUF protocol used in Docker Content Trust
  • Docker Content Trust: the integration between Notary and Docker Trusted Registry
  • Docker Engine: a program used to interact with all aspects of the Docker platform
  • Swarm: a program implementing Container Orchestration for the Docker platform