vbatts/quay
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

Get staging to run under docker on an EC2 host.

Browse source
This commit is contained in:
jakedt 2014-04-10 18:30:09 -04:00
parent 0fd5da172e
commit 8fac0474b5
8 changed files with 40 additions and 41 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
README.md
View file

@ -1,45 +1,35 @@
to prepare a new host: to prepare a new host:
``` ```
curl -s https://get.docker.io/ubuntu/ | sudo sh
sudo apt-get update sudo apt-get update
sudo apt-get install -y git python-virtualenv python-dev phantomjs libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev libevent-dev gdebi-core sudo apt-get install -y git
``` ```
check out the code: start the quay processes:
``` ```
git clone https://bitbucket.org/yackob03/quay.git git clone git clone https://bitbucket.org/yackob03/quayconfig.git
cd quay sudo docker pull quay.io/quay/quay
virtualenv --distribute venv sudo mkdir -p /mnt/logs/
source venv/bin/activate sudo docker run -d -p 80:80 -p 443:443 -v /mnt/logs:/mnt/logs -v `pwd`/quayconfig/production:/conf/stack quay.io/quay/quay
pip install -r requirements.txt
sudo gdebi --n binary_dependencies/*.deb
sudo cp conf/logrotate/* /etc/logrotate.d/
```
running:
```
sudo mkdir -p /mnt/logs/ && sudo chown $USER /mnt/logs/ && sudo /usr/local/nginx/sbin/nginx -c `pwd`/conf/nginx.conf
sudo mkdir -p /mnt/logs/ && sudo chown $USER /mnt/logs/ && STACK=prod gunicorn -c conf/gunicorn_config.py application:application
``` ```
start the log shipper: start the log shipper:
``` ```
curl -s https://get.docker.io/ubuntu/ | sudo sh
sudo docker pull quay.io/quay/logstash sudo docker pull quay.io/quay/logstash
sudo docker run -d -e REDIS_PORT_6379_TCP_ADDR=logs.quay.io -v /mnt/logs:/mnt/logs quay.io/quay/logstash quay.conf sudo docker run -d -e REDIS_PORT_6379_TCP_ADDR=logs.quay.io -v /mnt/logs:/mnt/logs quay.io/quay/logstash quay.conf
``` ```
start the workers: start the workers (FIXME):
``` ```
STACK=prod python -m workers.diffsworker -D STACK=prod python -m workers.diffsworker -D
STACK=prod python -m workers.webhookworker -D STACK=prod python -m workers.webhookworker -D
``` ```
bouncing the servers: bouncing the servers (FIXME):
``` ```
sudo kill -HUP `cat /mnt/logs/nginx.pid` sudo kill -HUP `cat /mnt/logs/nginx.pid`

3
application.py
View file

@ -1,11 +1,12 @@
import logging import logging
from app import app as application from app import app as application
from data.model import db as model_db
# Initialize logging # Initialize logging
application.config['LOGGING_CONFIG']() application.config['LOGGING_CONFIG']()
from data.model import db as model_db
# Turn off debug logging for boto # Turn off debug logging for boto
logging.getLogger('boto').setLevel(logging.CRITICAL) logging.getLogger('boto').setLevel(logging.CRITICAL)

4
conf/nginx-enterprise.conf
View file

@ -15,8 +15,8 @@ http {
listen 443 default; listen 443 default;
ssl on; ssl on;
ssl_certificate ./ssl.cert; ssl_certificate ./stack/ssl.cert;
ssl_certificate_key ./ssl.key; ssl_certificate_key ./stack/ssl.key;
ssl_session_timeout 5m; ssl_session_timeout 5m;
ssl_protocols SSLv3 TLSv1; ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP; ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP;

22
data/database.py
View file

@ -5,6 +5,7 @@ import uuid
from random import SystemRandom from random import SystemRandom
from datetime import datetime from datetime import datetime
from peewee import * from peewee import *
from sqlalchemy.engine.url import make_url
from urlparse import urlparse from urlparse import urlparse
from app import app from app import app
@ -21,19 +22,18 @@ SCHEME_DRIVERS = {
def generate_db(config_object): def generate_db(config_object):
db_kwargs = dict(config_object['DB_CONNECTION_ARGS']) db_kwargs = dict(config_object['DB_CONNECTION_ARGS'])
connection_string = config_object['DB_URI'] parsed_url = make_url(config_object['DB_URI'])
scheme, auth_and_host, dbname = urlparse(connection_string)[:3] if parsed_url.host:
if auth_and_host: db_kwargs['host'] = parsed_url.host
if '@' in auth_and_host: if parsed_url.port:
auth, db_kwargs['host'] = auth_and_host.split('@') db_kwargs['port'] = parsed_url.port
if ':' in auth: if parsed_url.username:
db_kwargs['user'], db_kwargs['passwd'] = auth.split(':') db_kwargs['user'] = parsed_url.username
if parsed_url.password:
db_kwargs['passwd'] = parsed_url.password
if scheme == 'sqlite': return SCHEME_DRIVERS[parsed_url.drivername](parsed_url.database, **db_kwargs)
dbname = dbname[1:]
return SCHEME_DRIVERS[scheme](dbname, **db_kwargs)
db = generate_db(app.config) db = generate_db(app.config)

4
requirements-nover.txt
View file

@ -26,4 +26,6 @@ loremipsum
pygithub pygithub
flask-restful flask-restful
jsonschema jsonschema
git+https://github.com/NateFerrero/oauth2lib.git git+https://github.com/NateFerrero/oauth2lib.git
alembic
sqlalchemy

9
requirements.enterprise
View file

@ -5,11 +5,14 @@ Flask-Mail==0.9.0
Flask-Principal==0.4.0 Flask-Principal==0.4.0
Flask-RESTful==0.2.12 Flask-RESTful==0.2.12
Jinja2==2.7.2 Jinja2==2.7.2
Mako==0.9.1
MarkupSafe==0.19 MarkupSafe==0.19
Pillow==2.3.1 Pillow==2.4.0
PyGithub==1.24.1 PyGithub==1.24.1
PyMySQL==0.6.1 PyMySQL==0.6.1
SQLAlchemy==0.9.4
Werkzeug==0.9.4 Werkzeug==0.9.4
alembic==0.6.4
aniso8601==0.82 aniso8601==0.82
argparse==1.2.1 argparse==1.2.1
beautifulsoup4==4.3.2 beautifulsoup4==4.3.2
@ -23,7 +26,7 @@ greenlet==0.4.2
gunicorn==18.0 gunicorn==18.0
hiredis==0.1.2 hiredis==0.1.2
html5lib==1.0b3 html5lib==1.0b3
itsdangerous==0.23 itsdangerous==0.24
jsonschema==2.3.0 jsonschema==2.3.0
lockfile==0.9.1 lockfile==0.9.1
logstash-formatter==0.5.8 logstash-formatter==0.5.8
@ -45,7 +48,7 @@ redis==2.9.1
reportlab==2.7 reportlab==2.7
requests==2.2.1 requests==2.2.1
six==1.6.1 six==1.6.1
stripe==1.12.2 stripe==1.14.0
websocket-client==0.11.0 websocket-client==0.11.0
wsgiref==0.1.2 wsgiref==0.1.2
xhtml2pdf==0.0.5 xhtml2pdf==0.0.5

9
requirements.txt
View file

@ -5,11 +5,14 @@ Flask-Mail==0.9.0
Flask-Principal==0.4.0 Flask-Principal==0.4.0
Flask-RESTful==0.2.12 Flask-RESTful==0.2.12
Jinja2==2.7.2 Jinja2==2.7.2
Mako==0.9.1
MarkupSafe==0.19 MarkupSafe==0.19
Pillow==2.3.1 Pillow==2.4.0
PyGithub==1.24.1 PyGithub==1.24.1
PyMySQL==0.6.1 PyMySQL==0.6.1
SQLAlchemy==0.9.4
Werkzeug==0.9.4 Werkzeug==0.9.4
alembic==0.6.4
aniso8601==0.82 aniso8601==0.82
argparse==1.2.1 argparse==1.2.1
beautifulsoup4==4.3.2 beautifulsoup4==4.3.2
@ -23,7 +26,7 @@ greenlet==0.4.2
gunicorn==18.0 gunicorn==18.0
hiredis==0.1.2 hiredis==0.1.2
html5lib==1.0b3 html5lib==1.0b3
itsdangerous==0.23 itsdangerous==0.24
jsonschema==2.3.0 jsonschema==2.3.0
lockfile==0.9.1 lockfile==0.9.1
logstash-formatter==0.5.8 logstash-formatter==0.5.8
@ -45,7 +48,7 @@ redis==2.9.1
reportlab==2.7 reportlab==2.7
requests==2.2.1 requests==2.2.1
six==1.6.1 six==1.6.1
stripe==1.12.2 stripe==1.14.0
websocket-client==0.11.0 websocket-client==0.11.0
wsgiref==0.1.2 wsgiref==0.1.2
xhtml2pdf==0.0.5 xhtml2pdf==0.0.5

2
workers/README.md
View file

@ -2,7 +2,7 @@ to prepare a new build node host:
``` ```
sudo apt-get update sudo apt-get update
sudo apt-get install -y git python-virtualenv python-dev phantomjs libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev libevent-dev gdebi-core sudo apt-get install -y git python-virtualenv python-dev phantomjs libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev libevent-dev gdebi-core lxc
``` ```
check out the code, install the kernel, custom docker, nsexec, and reboot: check out the code, install the kernel, custom docker, nsexec, and reboot: