Joseph Schorr
|
b8e9f2d1fa
|
Disable the lxc stability check. LXC is so broken that this was causing the build fleet to thrash when it encountered real issues with LXC/user namespacing.
|
2014-11-25 04:18:50 -05:00 |
|
Joseph Schorr
|
b2a0e58756
|
Use the new kwargs_from_env so that we can test with boot2docker and fix the issue with the .history call.
|
2014-11-24 20:28:48 -05:00 |
|
Jimmy Zelinskie
|
716d7a737b
|
Strip whitespace from ALL the things.
|
2014-11-24 16:07:38 -05:00 |
|
Jake Moshenko
|
f4681f2c18
|
Merge branch 'master' into nomenclature
Conflicts:
test/data/test.db
|
2014-11-17 17:59:59 -05:00 |
|
Joseph Schorr
|
c06f57a6e7
|
Make sure builders close the db handle when no work comes in and make the metrics transaction smaller in scope
|
2014-10-24 11:40:02 -04:00 |
|
Jake Moshenko
|
1461310ab8
|
Merge remote-tracking branch 'origin/master' into nomenclature
Conflicts:
endpoints/common.py
endpoints/notificationhelper.py
test/data/test.db
workers/dockerfilebuild.py
|
2014-10-23 13:25:37 -04:00 |
|
Jake Moshenko
|
1ccd6a9c5d
|
Change the max_instances for the workers to only allow one parallel job execution.
|
2014-10-22 18:09:00 -04:00 |
|
Joseph Schorr
|
07f3bd6f8c
|
Add a synthetic .git directory containing the commit sha so that 'git rev-parse HEAD' works from inside builds
|
2014-10-10 17:20:07 -04:00 |
|
Jake Moshenko
|
ed8bcff39e
|
Merge remote-tracking branch 'origin/master' into nomenclature
Conflicts:
test/data/test.db
workers/dockerfilebuild.py
|
2014-10-06 10:29:39 -04:00 |
|
Joseph Schorr
|
ec42303750
|
image_and_tag must be before we use it
|
2014-10-03 13:00:41 -04:00 |
|
Jake Moshenko
|
e8b3d1cc4a
|
Phase 4 of the namespace to user migration: actually remove the column from the db and remove the dependence on serialized namespaces in the workers and queues
|
2014-10-01 14:23:46 -04:00 |
|
Joseph Schorr
|
9c88ca16b5
|
Add the docker version to the build logs
|
2014-09-23 14:45:22 -04:00 |
|
Joseph Schorr
|
f23038c6ee
|
Update the worker code to better handle exceptions, fix the utcdate issue and make sure we send the proper retry. Also updates notification workers to send JobExceptions rather than returning true or false
|
2014-09-22 12:52:57 -04:00 |
|
Jake Moshenko
|
15a2f0d56f
|
Allow most builds to be archived without writing a file to the disk.
|
2014-09-16 00:03:04 -04:00 |
|
Jake Moshenko
|
efb66f7c1e
|
Select the random row function based on DB driver.
|
2014-09-15 15:58:56 -04:00 |
|
Jake Moshenko
|
c01de4a916
|
Set redis logs entries to expire rather than to immediately delete them to make the logs archiver idempotent.
|
2014-09-12 13:13:14 -04:00 |
|
Jake Moshenko
|
353da13b16
|
Set up the service monitoring for the log migration script. Add a database migration to add the logs_archived column.
|
2014-09-12 11:57:47 -04:00 |
|
Jake Moshenko
|
b55e79e5d3
|
Merge remote-tracking branch 'origin/master' into waltermitty
|
2014-09-12 11:45:07 -04:00 |
|
Jake Moshenko
|
5388633f9a
|
Merge remote-tracking branch 'origin/master' into pullfail
|
2014-09-12 10:36:38 -04:00 |
|
Jake Moshenko
|
8b3a3178b0
|
Finish the build logs archiver, add handlers for cloud and local that handle gzip encoded archived content.
|
2014-09-11 15:33:10 -04:00 |
|
Jake Moshenko
|
2455c17f96
|
Merge remote-tracking branch 'origin/master' into waltermitty
Conflicts:
app.py
data/userfiles.py
|
2014-09-11 11:18:28 -04:00 |
|
Jake Moshenko
|
29d40db5ea
|
Add a new RadosGW storage engine. Allow engines to distinguish not only between those that can support direct uploads and downloads, but those that support doing it through the browser. Rename resumeable->resumable.
|
2014-09-09 15:54:03 -04:00 |
|
Jake Moshenko
|
451e034ca1
|
Archived logs commit 1. Squash me.
|
2014-09-08 16:43:17 -04:00 |
|
Joseph Schorr
|
07aab4274c
|
Fix parameters for logging the extra data needed
|
2014-08-28 19:19:20 -04:00 |
|
Joseph Schorr
|
5744f0f888
|
Make the dockerfilebuild error checking less harsh
|
2014-08-28 16:07:56 -04:00 |
|
Joseph Schorr
|
463a3c55c3
|
Make worker error messages more descriptive
|
2014-08-27 19:02:53 -04:00 |
|
Joseph Schorr
|
510bbe7889
|
Add more check conditions for unhealthy workers and make the messaging better.
|
2014-08-26 12:41:43 -04:00 |
|
Joseph Schorr
|
67905c277e
|
Remove webhook worker
|
2014-08-25 19:13:40 -04:00 |
|
Joseph Schorr
|
daa43c3bb9
|
Add better messaging around pulling of base images when they fail due to invalid or missing credentials
|
2014-08-18 20:34:39 -04:00 |
|
Joseph Schorr
|
736af3165b
|
Add a default message if the build pack MIME processor fails
|
2014-08-15 18:23:43 -04:00 |
|
Joseph Schorr
|
8681dd9cb9
|
Add a new exposed 'unpacking' phase to the build and make sure that the unzip/untar/etc always occurs under a try-except
|
2014-08-15 17:58:11 -04:00 |
|
Joseph Schorr
|
728af56384
|
Make the watchdog in the build worker also requeue the current item if the worker has gone bad
|
2014-08-13 19:04:51 -04:00 |
|
Joseph Schorr
|
b9e9064af2
|
Only retry on unhealthy exceptions, not JobException's.
|
2014-08-10 18:28:20 -04:00 |
|
Joseph Schorr
|
1b7379df29
|
Fix workers to not always be marked as unhealthy
|
2014-08-08 15:24:19 -04:00 |
|
Jake Moshenko
|
0372013f70
|
Merge remote-tracking branch 'origin/redalert'
Conflicts:
app.py
|
2014-08-04 16:56:34 -04:00 |
|
Jake Moshenko
|
0aa6e92b02
|
Finish porting the workers over to apscheduler 3.0
|
2014-08-01 18:38:02 -04:00 |
|
Jake Moshenko
|
6b38ddb9b6
|
Remove the gpled loremipsum module.
|
2014-07-31 16:46:02 -04:00 |
|
Joseph Schorr
|
49801bc2c4
|
- Add web hook queue code back in. We'll remove it and turn it off after this CL goes to prod
- Make notification lookup always be by repo and its UUID, rather than the internal DB ID
- Add the init script for the notification worker
|
2014-07-31 13:30:54 -04:00 |
|
Joseph Schorr
|
bab3a0949c
|
Make sure completion marking is also under the lock
|
2014-07-30 18:45:40 -04:00 |
|
Joseph Schorr
|
4aec422e24
|
Add a lock around accessing the current queue item and make sure to report it as incomplete whenever the worker becomes unhealthy
|
2014-07-30 18:30:54 -04:00 |
|
Joseph Schorr
|
7e935f5a8c
|
Make build workers report that they are unhealthy when we get an LXC error or a Docker connection issue
|
2014-07-30 17:54:58 -04:00 |
|
Joseph Schorr
|
752efb9e0f
|
Fix the spawn_notification to work in all cases and clean up some of the remaining code
|
2014-07-18 16:34:52 -04:00 |
|
Joseph Schorr
|
591cd020b8
|
Merge branch 'master' into redalert
|
2014-07-18 15:58:56 -04:00 |
|
Joseph Schorr
|
af31bde997
|
Add support for the remaining events to the frontend and the backend
|
2014-07-18 15:58:18 -04:00 |
|
Jake Moshenko
|
74d1c4e6b0
|
Update the worker status endpoint to be ELB friendly.
|
2014-07-18 15:04:20 -04:00 |
|
Joseph Schorr
|
8d7493cb86
|
Convert over to notifications system. Note this is incomplete
|
2014-07-17 22:51:58 -04:00 |
|
Jake Moshenko
|
cceb09d4f6
|
Remove some unused dependencies and update the rest.
|
2014-07-17 12:08:07 -04:00 |
|
Joseph Schorr
|
8b3659fefa
|
Dockerfile build worker should not report inner JobException's twice
|
2014-07-11 12:05:52 -04:00 |
|
Joseph Schorr
|
9d1ae8ba87
|
FROM line check needs to be on the tuple result, not the join
|
2014-06-16 14:01:17 -04:00 |
|
Joseph Schorr
|
f795868b5b
|
Handle the case where there is no FROM command in the Dockerfile
|
2014-06-13 16:56:48 -04:00 |
|
Jake Moshenko
|
e8355f301e
|
Remove our deploy key from the workers/Readme which gets included in the Docker image.
|
2014-05-27 15:19:23 -04:00 |
|
Jake Moshenko
|
0b6552d6cc
|
Fix the metrics so they are usable for scaling the workers down and up. Switch all datetimes which touch the database from now to utcnow. Fix the worker Dockerfile.
|
2014-05-23 14:16:26 -04:00 |
|
Jake Moshenko
|
d14798de1d
|
Add a queue capacity reporter plugin to the queue. Move the queue definitions to app. Add a cloudwatch reporter to the dockerfile build queue.
|
2014-05-21 19:50:37 -04:00 |
|
Jake Moshenko
|
b8466169ac
|
Integrate sentry with the build worker.
|
2014-05-19 13:50:45 -04:00 |
|
Jake Moshenko
|
212a4650f4
|
Rework the config to use runit logging.
|
2014-05-18 17:19:14 -04:00 |
|
Jake Moshenko
|
cc47e77156
|
Upgrade to the 0.11.1 tutum version of docker. Package it as a Dockerfile using Docker in Docker. Add a status server option to the workers to utilize the new termination signal and status features of gantry.
|
2014-05-16 18:31:24 -04:00 |
|
Jake Moshenko
|
c92ce54a37
|
Reduce a step in the worker bootstrap.
|
2014-05-13 17:44:45 -04:00 |
|
Jake Moshenko
|
bcb993a914
|
Set up the build logs to use our fake build logs on test and local.
|
2014-05-09 18:45:11 -04:00 |
|
Jake Moshenko
|
8a3af93b8c
|
Improve the builder response to being terminated or dying.
|
2014-05-06 18:46:19 -04:00 |
|
Jake Moshenko
|
55f18a2ecf
|
Add the missing uid translation range to the root user.
|
2014-05-01 17:54:59 -04:00 |
|
Jake Moshenko
|
ec282999bf
|
Use the docker version which works with 14.04 lxc.
|
2014-05-01 17:24:58 -04:00 |
|
Jake Moshenko
|
32583a5675
|
First steps toward running the builder on trusty.
|
2014-05-01 15:39:33 -04:00 |
|
Jake Moshenko
|
b888c05bc4
|
Change the version of our docker binary because the public registry is blocking the tutum agent name.
|
2014-05-01 11:44:59 -04:00 |
|
Jake Moshenko
|
450928674b
|
Use a new caching algorithm which can limit the size for the build nodes. Stop treating public images as special. Add a new phase to the builder for pulling.
|
2014-04-30 18:48:36 -04:00 |
|
jakedt
|
58dbb540a1
|
Run a worker task immediately when it starts.
|
2014-04-22 13:55:54 -04:00 |
|
jakedt
|
2bc3d24543
|
Update the build worker to remove all tags from expired images.
|
2014-04-18 18:36:11 -04:00 |
|
jakedt
|
0d8725e778
|
Update the instructions for starting and running the workers.
|
2014-04-17 16:18:53 -04:00 |
|
jakedt
|
0a9ee6c49f
|
Bust the dockerfile build cache across repository lines.
|
2014-04-16 15:45:41 -04:00 |
|
jakedt
|
0827e0fbac
|
Merge remote-tracking branch 'origin/master' into ncc1701
Conflicts:
endpoints/web.py
static/directives/signup-form.html
static/js/app.js
static/js/controllers.js
static/partials/landing.html
static/partials/view-repo.html
test/data/test.db
|
2014-04-14 19:37:22 -04:00 |
|
jakedt
|
724fec1b74
|
Test third party repo images for public-ness in the builder. Always clean up private images that we dont know about before build. Pull the base image to refresh before every build.
|
2014-04-14 18:54:39 -04:00 |
|
jakedt
|
40f82a9d16
|
Work harder to reset the state of the docker env on the build worker.
|
2014-04-14 15:59:57 -04:00 |
|
jakedt
|
de18236358
|
Allow for caching of previous docker builds for 24 hours.
|
2014-04-14 15:21:05 -04:00 |
|
jakedt
|
61a6db236f
|
Finish the implementation of local userfiles. Strip charsets from mimetypes in the build worker. Add canonical name ordering to the build queue. Port all queues to the canonical naming version.
|
2014-04-11 18:34:47 -04:00 |
|
jakedt
|
576fbe4f0d
|
Switch over to phusion baseimage. Prevent everything from daemonizing and start it with runit under phusion. Make workers trap and handle sigint and sigterm. Extend the reservation to 1hr for dockerfilebuild. Update nginx to remove the dependency on libgd. Merge the requirements and requirements enterprise files.
|
2014-04-11 13:32:45 -04:00 |
|
jakedt
|
8fac0474b5
|
Get staging to run under docker on an EC2 host.
|
2014-04-10 18:30:09 -04:00 |
|
jakedt
|
8076ad0a20
|
Add x-gzip as a mimetype for tarballs.
|
2014-04-07 10:58:53 -04:00 |
|
jakedt
|
37ad8a414a
|
Revert a change to the way zip files are extracted.
|
2014-04-04 16:06:41 -04:00 |
|
jakedt
|
e87ffa20cf
|
First attempt at making config loadable through string config overrides in an env variable.
|
2014-04-03 17:31:46 -04:00 |
|
jakedt
|
b95d3ec329
|
Add a watchdog timer to the build worker to kill a build step that takes more than 20 minutes.
|
2014-04-02 19:32:41 -04:00 |
|
Joseph Schorr
|
9a79d1562a
|
Change to store the pull robot on the repository build and only add the credentials to the queue item. This prevents the credentials from being exposed to the end user. Also fixes the restart build option
|
2014-04-01 21:49:06 -04:00 |
|
jakedt
|
ca1970a2f4
|
Use the real registry endpoint in the login command.
|
2014-04-01 19:00:11 -04:00 |
|
jakedt
|
3525e383df
|
Merge remote-tracking branch 'origin/master' into pullinprivate
Conflicts:
test/data/test.db
|
2014-04-01 18:28:48 -04:00 |
|
jakedt
|
7c44932c87
|
Use safer tar extraction. Handle error messages in the build process more intelligently.
|
2014-04-01 13:46:41 -04:00 |
|
jakedt
|
d67a1cddc2
|
Merge remote-tracking branch 'origin/master' into pullinprivate
Conflicts:
workers/dockerfilebuild.py
|
2014-03-31 18:10:34 -04:00 |
|
jakedt
|
e7c20e1052
|
Add tarball support to the builder and pull github code as a tarball.
|
2014-03-31 15:40:24 -04:00 |
|
jakedt
|
7c14190d2a
|
Freeze the version of docker-py to 0.3.0 and create a fancy streaming json decoder to work around the lack of newlines in push statuses.
|
2014-03-28 17:53:33 -04:00 |
|
Joseph Schorr
|
2006917e03
|
Add support for pull credentials on builds and build triggers
|
2014-03-27 18:33:13 -04:00 |
|
Jake Moshenko
|
40e81f478f
|
If JSON decoding of a stream response fails, just ignore it since it's probably a blank line.
|
2014-03-27 22:15:59 +00:00 |
|
Jake Moshenko
|
2767c7e1f7
|
Change the docker version to one that starts using the lxc driver.
|
2014-03-27 21:11:51 +00:00 |
|
Jake Moshenko
|
d6f1ea36d4
|
Upgrade the worker docker to the patched version of 0.9.
|
2014-03-27 20:39:29 +00:00 |
|
jakedt
|
536a91cbb8
|
Handle messages with the word message in them.
|
2014-03-07 12:27:30 -05:00 |
|
jakedt
|
4566d76e45
|
Add a mime type for x-zip-compressed. Report unknown mime types as errors.
|
2014-03-06 23:40:21 -05:00 |
|
jakedt
|
9a733a7320
|
Sometimes status messages have an extra level of wrapping for some reason.
|
2014-03-05 17:44:22 -05:00 |
|
jakedt
|
270a62b8c1
|
Fix the build worker to use the latest docker-py.
|
2014-03-05 10:32:24 -05:00 |
|
jakedt
|
c13f7cd9df
|
Add the resource_key back to the repository build.
|
2014-02-25 18:22:02 -05:00 |
|
jakedt
|
13dea98499
|
Prepare the build worker to support multiple tags and subdirectories. Change the build database config to accept a job config object instead of breaking out the parameters into independent blocks.
|
2014-02-24 16:11:23 -05:00 |
|
yackob03
|
91da26e317
|
Improve the instructions for starting the worker on the build node.
|
2014-02-13 12:46:51 -05:00 |
|
yackob03
|
1e8ec634e2
|
Some small fixes to the dockerfile build worker.
|
2014-02-13 11:44:39 -05:00 |
|
yackob03
|
b920a0cb1f
|
Really try to emulate the logs format with the test logs. Switch to a simplified flat logs format.
|
2014-02-12 18:58:40 -05:00 |
|
yackob03
|
83fb806e51
|
Apparently gdebi doesn't support multiple packages in one command? lame.
|
2014-02-12 10:07:10 -05:00 |
|
yackob03
|
d8df243a0a
|
Add the binary dependencies for the build worker and some instructions on how to get it running on a host.
|
2014-02-11 19:55:22 -05:00 |
|
yackob03
|
6fd343741b
|
Change to the new paging format with the commands available at the top.
|
2014-02-10 19:12:43 -05:00 |
|
yackob03
|
52d2be7953
|
Identify build commands separately from their output.
|
2014-02-10 15:03:55 -05:00 |
|
yackob03
|
fab699530c
|
Check for errors before delegating to the cleanup function.
|
2014-02-05 19:59:26 -05:00 |
|
yackob03
|
67e0736fc6
|
Separate out the dockerfile build stuff to a separate class. Add a dependency on docker-py master.
|
2014-02-05 11:07:54 -05:00 |
|
yackob03
|
5270066d6d
|
Switch to the redis backed build logs and status.
|
2014-02-03 19:08:37 -05:00 |
|
yackob03
|
aa58b840a5
|
Fix the dockerfile build login to work on http and https registries.
|
2014-01-24 16:25:25 -05:00 |
|
yackob03
|
72559fb948
|
First stab at the new builder.
|
2014-01-24 14:40:36 -05:00 |
|
yackob03
|
8981f576fc
|
Fixes to the build server.
|
2013-12-31 16:22:58 -05:00 |
|
yackob03
|
bf60862d8e
|
Switch to the new Digital Ocean Docker image.
|
2013-12-23 15:07:10 -05:00 |
|
yackob03
|
e787d8b2d8
|
Make the worker post json. Add a retry timeout after an incompletel queue item is processed. Submit webhook jobs to the queue on a successful push.
|
2013-11-16 15:05:26 -05:00 |
|
yackob03
|
c1ea6263e1
|
Flesh out the webworkers a bit.
|
2013-11-15 16:45:02 -05:00 |
|
yackob03
|
14263de7f8
|
Extract some boilerplate from the worker and create a base class. Port the diffs worker over to the base.
|
2013-11-15 15:50:20 -05:00 |
|
yackob03
|
0f3d942b5e
|
Re-add the missing method for getting repository builds. Add more logging to try to see why some builds are not getting removed from the queue.
|
2013-11-13 17:29:26 -05:00 |
|
yackob03
|
d7592fd133
|
Update the diffs worker to not fail if the repository is removed before diffs are computed.
|
2013-11-13 14:41:20 -05:00 |
|
yackob03
|
2cd98fc58e
|
Make the app config more powerful in terms of injecting fake dependencies. Refactor the tests to use metaclasses and to actually all run.
|
2013-11-06 23:21:12 -05:00 |
|
yackob03
|
f8f4c55f89
|
Close the db connections on the workers occasionally.
|
2013-10-30 13:22:00 -04:00 |
|
yackob03
|
1bbcd91ae0
|
Make sure the daemon starts up in the right directory to access the certs file.
|
2013-10-29 19:59:29 -04:00 |
|
yackob03
|
7adf37e6b5
|
Move the parmiko import to work around a bug with python daemonization and paramiko.
|
2013-10-29 18:56:57 -04:00 |
|
yackob03
|
5e81f999d1
|
Make the build server and build worker slightly more robust to errors.
|
2013-10-29 18:15:12 -04:00 |
|
yackob03
|
6f105326aa
|
Return unfinished items to the queue when they can be retried.
|
2013-10-29 15:42:19 -04:00 |
|
yackob03
|
798cf78172
|
Re-add cleaning up the DO node.
|
2013-10-29 15:39:44 -04:00 |
|
yackob03
|
c91b40f356
|
Switch to using environment variables to pass information to the build node, this closes down a security loophole with the admin endpoint of the server.
|
2013-10-29 13:42:26 -04:00 |
|
yackob03
|
1c7398e920
|
Allow build node logs to flow to docker. Clean up the pull key before executing user code. Actually clean up the DO node when we are done. Run the status server threaded just in case.
|
2013-10-28 14:49:23 -04:00 |
|
yackob03
|
d7f51fb764
|
Bug fixes and associated changes involved with spinning up build nodes and sending jobs to them.
|
2013-10-27 19:06:20 -04:00 |
|
yackob03
|
78d2d6cad0
|
Add a test endpoint that allows us to manually create builds. Add access tokens to the build data.
|
2013-10-25 18:17:43 -04:00 |
|
yackob03
|
684ce83058
|
Add a test api to test repositories which are currenlty building.
|
2013-10-25 15:47:34 -04:00 |
|
yackob03
|
ffc33e454c
|
Fixes and refinements to the dockerfile build system.
|
2013-10-25 15:13:11 -04:00 |
|
yackob03
|
9b9a29c310
|
Flesh out the create API and wire everything up together. Next up, testing.
|
2013-10-25 01:14:38 -04:00 |
|
yackob03
|
32ccc599fb
|
Revert some fabric changes to the build worker.
|
2013-10-24 16:39:25 -04:00 |
|
yackob03
|
847e91b696
|
Add the first part of the build worker that runs on quay.
|
2013-10-24 16:37:03 -04:00 |
|
yackob03
|
0c979cd79f
|
Fix the diffsworker to run as a daemon.
|
2013-10-20 14:52:56 -04:00 |
|
yackob03
|
abb6efda40
|
Process as many jobs as the queue will give us in a row, sleep for longer in between checks.
|
2013-10-20 03:26:06 -04:00 |
|
yackob03
|
607bf3ecc4
|
Daemonize the worker.
|
2013-10-18 17:52:55 -04:00 |
|
yackob03
|
2fcd8df42b
|
Get the worker and the registry playing nice together.
|
2013-10-18 17:27:09 -04:00 |
|
yackob03
|
4514f5a969
|
Fix some typos and bugs in the worker.
|
2013-10-18 15:28:16 -04:00 |
|
yackob03
|
93b856bdb3
|
First few changes for the image diffs feature.
|
2013-10-17 18:25:19 -04:00 |
|