Commit graph

191 commits

Author SHA1 Message Date
Joseph Schorr
d973f9df45 Reenable metrics until we know they are the problem 2015-02-25 16:00:46 -05:00
Joseph Schorr
bdb84f1c20 Merge branch 'master' of github.com:coreos-inc/quay 2015-02-25 16:00:17 -05:00
Joseph Schorr
4551b3a957 Remove the boto timeout set (doesn't work anyway) and add some better logging to the scheduler 2015-02-25 16:00:14 -05:00
Jimmy Zelinskie
090a198afc temporarily comment out metrics 2015-02-25 15:29:35 -05:00
Jimmy Zelinskie
db79ad2dde unused import 2015-02-25 15:26:36 -05:00
Joseph Schorr
5dd78f76c7 Add additional logging, timeouts, and exception checks 2015-02-25 15:15:22 -05:00
Jimmy Zelinskie
328de0201f Merge branch 'master' of github.com:coreos-inc/quay 2015-02-25 13:56:05 -05:00
Jimmy Zelinskie
346d6b933a buildman: initialize queuemetrics asynchronously 2015-02-25 13:55:18 -05:00
Joseph Schorr
2eaec092f0 Handle the case where we cannot write the tags on the build nodes 2015-02-25 13:47:36 -05:00
Joseph Schorr
390f8df4ad Make sure the build manager dies on an unhandled schedule exception 2015-02-25 12:19:21 -05:00
Joseph Schorr
afe7e14254 Add better exception handling and logging to the ephemeral build manager 2015-02-25 12:09:14 -05:00
Joseph Schorr
b7901d2adb Add trigger metadata (which includes the SHA) and the built image_id to the event data 2015-02-24 15:13:51 -05:00
Jimmy Zelinskie
47f8cb77c4 Merge pull request #11 from coreos-inc/nimbus
CloudWatch for build job status
2015-02-18 17:17:28 -05:00
Jimmy Zelinskie
9ab3554226 buildreporter: does not execute in a coroutine! 2015-02-18 17:11:45 -05:00
Jimmy Zelinskie
0d38e0b00b metrics: use config['name'] to get metric conf 2015-02-18 16:05:36 -05:00
Jimmy Zelinskie
f53dea46b7 buildman: address PR #11 comments 2015-02-18 14:13:36 -05:00
Joseph Schorr
524705b88c Get dashboard working and upgrade bootstrap. Note: the bootstrap fixes will be coming in the followup CL 2015-02-17 19:15:54 -05:00
Jimmy Zelinskie
5790d7d8cc buildman: build_metrics call correct method 2015-02-17 17:03:12 -05:00
Jimmy Zelinskie
1a71925125 buildreporter: remove unused logging 2015-02-17 17:02:37 -05:00
Jimmy Zelinskie
85edb651e2 buildserver: remove pylint comments 2015-02-17 15:32:25 -05:00
Jimmy Zelinskie
d70c95e42e buildreporter: move reporting into server callback 2015-02-17 15:31:53 -05:00
Jimmy Zelinskie
25fc999d50 buildreporter: handle app=None 2015-02-17 15:30:09 -05:00
Jimmy Zelinskie
b8d9ef0fe9 buildman: remove old create_task for queue metrics 2015-02-17 14:18:32 -05:00
Jimmy Zelinskie
935db5c766 buildman: clarify queue metrics from job state metrics 2015-02-17 12:23:08 -05:00
Jimmy Zelinskie
ffb897dfe6 buildman: add job status logging to managers 2015-02-17 12:22:23 -05:00
Jimmy Zelinskie
ca0d2b1721 buildreporter: getattr method 2015-02-17 12:21:22 -05:00
Jimmy Zelinskie
0a00453024 buildreporter: rm pylint comments 2015-02-17 12:20:46 -05:00
Jimmy Zelinskie
0e7418ffce buildman: add BuildMetrics and BuildReporter 2015-02-17 10:56:09 -05:00
Joseph Schorr
fbdbc21eb1 Merge branch 'master' into quark 2015-02-13 16:24:53 -05:00
Jimmy Zelinskie
6a3d269574 buildman: update metrics task 2015-02-13 11:25:29 -05:00
Joseph Schorr
ae8bb5fc13 Add preparing build node status item and change the build status colors to be variations on a blue color 2015-02-12 16:38:43 -05:00
Joseph Schorr
f84d1bad45 Handle internal errors in a better fashion: If a build would be marked as internal error, only do so if there are retries remaining. Otherwise, we mark it as failed (since it won't be rebuilt anyway) 2015-02-12 16:19:44 -05:00
Joseph Schorr
f107b50a46 Merge branch 'master' into ackbar 2015-02-12 12:04:45 -05:00
Joseph Schorr
f796c281d5 Remove support for v0.2 2015-02-11 17:12:53 -05:00
Joseph Schorr
e1a15464a1 Fix typo, add some logging and fix command comparison 2015-02-11 16:02:36 -05:00
Joseph Schorr
893ae46dec Add an ImageTree class and change to searching *all applicable* branches when looking for the best cache tag. 2015-02-10 21:46:58 -05:00
Joseph Schorr
98b4f62ef7 Switch to using a squashed image for the build workers 2015-02-10 15:43:01 -05:00
Joseph Schorr
045614c6c8 Merge branch 'master' into ackbar 2015-02-09 17:16:42 -05:00
Joseph Schorr
6b9464c999 Add support for 0.3 (the new builder version) 2015-02-09 16:59:21 -05:00
Joseph Schorr
9f1ec9d47d Fix loading of partial caching under a tag 2015-02-09 16:29:15 -05:00
Joseph Schorr
b0e315c332 Fix issues in cache comment comparison 2015-02-09 15:48:36 -05:00
Joseph Schorr
9b0e43514b Fix typos 2015-02-09 14:53:18 -05:00
Joseph Schorr
384d0eba6f Fix cache command argument 2015-02-09 14:12:24 -05:00
Joseph Schorr
6cb1212da6 Add logging 2015-02-09 13:54:14 -05:00
Joseph Schorr
4310f47dee Some code cleanup in the cached tag determination code 2015-02-09 12:16:43 -05:00
Joseph Schorr
0065ac8503 Add back in the cache checking code and remove the old 0.1 build pack code 2015-02-09 12:13:40 -05:00
Joseph Schorr
48949627e0 Merge master in delta 2015-02-09 12:07:43 -05:00
Joseph Schorr
9dfe523615 Merge master changes 2015-02-05 13:11:16 -05:00
Jimmy Zelinskie
c7c5377285 Add my key back to the ephemeral builder machines. 2015-02-05 12:51:02 -05:00
Joseph Schorr
5fedd74399 Remove Jake's key 2015-02-04 21:31:26 -05:00
Jake Moshenko
a952d0b1ce Merge branch 'master' of github.com:coreos-inc/quay 2015-02-04 11:59:27 -05:00
Jake Moshenko
5b8d65991e Update the space on the builder nodes because its cheap. 2015-02-04 11:58:58 -05:00
Joseph Schorr
9ffb53cd47 Add support for v2 of the build worker, which performs the Dockerfile parsing on its own. Note that this version is backwards compatible with v1-beta of the build worker, so it should be pushed first. Also note that this version is temporary until such time as we get the caching branches merged. 2015-02-03 21:05:18 -05:00
Joseph Schorr
a1938593a9 Better handling of retries on build errors 2015-02-03 16:29:47 -05:00
Joseph Schorr
3bf5e93f06 Remove log statement 2015-02-03 16:06:23 -05:00
Joseph Schorr
d709e0f64a Fix the new notifications code to work 2015-02-03 13:08:38 -05:00
Joseph Schorr
07e85324e9 - Add build notifications back in
- Fix spelling mistake
- Add the sha output as part of the build script
2015-02-03 13:01:42 -05:00
Joseph Schorr
361fb33574 - Add a small build script
- Take in the build worker branch name from config
- Add additional logging (to be removed after we figure out the problem)
2015-02-03 12:48:41 -05:00
Jake Moshenko
2215ec6669 Associate a public IP with the network interfaces on our VPC instances. 2015-02-02 15:28:40 -05:00
Jake Moshenko
db8493f254 update the executor template to use VPC instances. 2015-02-02 14:55:34 -05:00
Jake Moshenko
3687419ab3 Change a typo to an enum 2015-02-02 12:24:32 -05:00
Jake Moshenko
a4b0c8698d Allow the key prefixes in etcd to be configurable. 2015-02-02 12:00:19 -05:00
Joseph Schorr
0875d3dce1 Merge branch 'master' of https://github.com/coreos-inc/quay 2015-01-29 18:40:49 -05:00
Joseph Schorr
3872d29de9 Add a transaction around the extend_processing call 2015-01-29 18:40:41 -05:00
Jake Moshenko
fb533a1f4c Merge branch 'master' of github.com:coreos-inc/quay 2015-01-29 18:40:24 -05:00
Jake Moshenko
8e85ff63f1 Add everyones ssh keys to the ephemeral build workers. 2015-01-29 18:40:17 -05:00
Jake Moshenko
63d23a04c0 Make the loop pause when we run out of builder capacity. 2015-01-29 18:40:01 -05:00
Joseph Schorr
838bfe23b1 Remove retries update in the extend processing call and make sure it is under a transaction 2015-01-29 18:33:17 -05:00
Joseph Schorr
a6fa08c19c Change returns to trollius returns 2015-01-29 18:21:32 -05:00
Joseph Schorr
0e5f6dc17d Fix typo in timed out 2015-01-29 18:13:31 -05:00
Joseph Schorr
60eae43ae4 Add the date time to the log entries 2015-01-29 18:05:05 -05:00
Joseph Schorr
ce3f8b438c Fix pull credentials bug, fix job details parse bug and add some better logging 2015-01-29 18:01:42 -05:00
Joseph Schorr
7ee00b83cb Switch to using a CloseForLongOperation around the sleep 2015-01-29 14:50:07 -05:00
Joseph Schorr
cf35da30bc Make sure to not hold DB connections open in the new build manager 2015-01-29 14:40:24 -05:00
Jake Moshenko
2e86417329 Allow the buildman server to die if an uncaught exception terminates the scheduler process. 2015-01-29 10:56:57 -05:00
Jake Moshenko
c308794063 Fix the enterprise manager to use the new coroutine based interface. 2015-01-29 10:56:18 -05:00
Joseph Schorr
d359c849cd Add the build worker and job count information to the charts 2015-01-28 17:12:33 -05:00
Jake Moshenko
0ddfd07749 Use the tiny registry-build-worker image. Bind mount in the root certificates so that Quay SSL certificates can be calidated. 2015-01-27 14:12:47 -05:00
Jake Moshenko
ef0806bd9d Make the logs for the build manager more bearable. 2015-01-26 15:27:39 -05:00
Joseph Schorr
be6701b310 Have the builder not start and stop, over and over, if not enabled 2015-01-26 14:13:55 -05:00
Jake Moshenko
86852da4ba Catch exceptions when ELB times out a connection to etcd. 2015-01-23 11:29:38 -05:00
Jake Moshenko
725808a4f8 Make the logs from the build manager more useful. 2015-01-23 11:29:15 -05:00
Jake Moshenko
265aeabf60 We need to tell the etcd client which protocol to use. 2015-01-22 16:59:04 -05:00
Jake Moshenko
f2471a86f6 Fix the python requirements. Add the ability to map in etcd client certs and ca. 2015-01-22 10:53:23 -05:00
Jake Moshenko
fc757fecad Tag the EC2 instances with the build uuid. 2015-01-05 15:35:14 -05:00
Jake Moshenko
dd7664328c Make the build manager ports configurable. 2015-01-05 15:09:03 -05:00
Jake Moshenko
8037962716 Change the severity of a log message which is actually expected in the happy case. 2015-01-05 14:44:54 -05:00
Jake Moshenko
f58b09a064 Remove the loop argument from the call to build_component_ready. 2015-01-05 13:08:25 -05:00
Jake Moshenko
320ae63ccd Handle the case where there are no realms registered. 2015-01-05 12:23:54 -05:00
Jake Moshenko
b33ee1a474 Register existing builders to watch their expirations. 2015-01-05 11:21:36 -05:00
Jake Moshenko
a9839021af When the etcd key tracking realms is first created the action is create, not set. 2014-12-31 11:46:02 -05:00
Jake Moshenko
cc70225043 Generalize the ephemeral build managers so that any manager may manage a builder spawned by any other manager. 2014-12-31 11:33:56 -05:00
Jake Moshenko
ccb19571d6 Try lowering the sleep on the shutdown timeout to avoid the service dispatch timeout built into systemd. 2014-12-23 17:42:47 -05:00
Jake Moshenko
ec87e37d8c EC2 terminate_instances does not take a force flag. 2014-12-23 17:17:53 -05:00
Jake Moshenko
1005c29b6b Fix the shutdown command for when the builder terminates itself. 2014-12-23 17:08:16 -05:00
Jake Moshenko
cece94e1da We want to terminate instances, not stop them. 2014-12-23 16:20:42 -05:00
Jake Moshenko
3ce64b4a7f We must yield from stop_builder. 2014-12-23 16:12:10 -05:00
Jake Moshenko
ef70432b11 We need to call build_finished async. 2014-12-23 16:04:10 -05:00
Jake Moshenko
8e16fbf59b The root device on CoreOS is /dev/xvda. 2014-12-23 15:41:58 -05:00
Jake Moshenko
2f2a88825d Try using SSD for root volumes. 2014-12-23 15:35:21 -05:00