Evan Cordell
42ebb0a6c3
Record metrics in a separate etcd record
2016-10-03 16:11:37 -04:00
Joseph Schorr
f50bb8a1ce
Add missing call to set_phase when a build doesn't start
...
This change fixes the build manager ephemeral executor to tell the overall build server to call set_phase when a build never starts. Before this change, we'd properly adjust the queue item, but not the repo build row or the logs, which is why users just saw "Preparing Build Node", with no indicating the node failed to start.
Fixes #1904
2016-09-30 14:54:49 +02:00
Evan Cordell
832ee89923
Add duration metric collector decorator ( #1885 )
...
Track time-to-start for builders
Track time-to-build for builders
Track ec2 builder fallbacks
Track build time
2016-09-29 15:44:06 -04:00
Joseph Schorr
de990253bc
Fix the build manager tests for recent change
2016-09-26 17:28:09 +02:00
Joseph Schorr
742e153133
Fix watch of the jobs key in the build manager
2016-08-16 15:43:09 -04:00
Joseph Schorr
d78361b041
Cleanup old executions that never start
...
Fixes #1727
2016-08-15 16:54:02 -04:00
Joseph Schorr
c29f9ccc7f
Fix TTL on heartbeat in etcd
...
Until now, once the heartbeat has expired, we would issue a TTL that is negative, which causes etcd to either raise an exception or simply ignore the expiration (depending on the version of etcd). This change ensures that once the key is expired, it is removed immediately via a set of a TTL of 0. Also adds tests for this case and the normal expiration case.
2016-08-03 11:15:03 -04:00
Joseph Schorr
392242d20b
Another fix for the record keeping in buildman
...
Adds some more mocked tests as well
2016-07-22 12:01:30 -04:00
Joseph Schorr
2c1880b944
Bug fixes, refactoring and "new" tests for the build manager
...
- Fixes various bugs introduced in the most recent build system commit
- Refactors state management in the build manager to be cleaner and more contained
- Adds back in the mock-based tests, fixed to not use threads and adjusted for the refactoring
- Adds some more simplified unit tests around non-etch related flows
2016-07-18 13:46:48 -04:00
Joseph Schorr
74b87fa813
Build manager cleanup and more logging
2016-07-14 14:33:14 -04:00
Joseph Schorr
811413fe9c
Add multiple executor and whitelist support to build manager
2016-07-08 15:50:51 -04:00
Colin Hom
bc13333f20
Kubernetes build worker
2016-07-08 15:50:51 -04:00
Joseph Schorr
2f771304fa
Disable flaky mock-based tests
2016-06-24 16:04:34 -04:00
Jimmy Zelinskie
ca65df68ba
skip flaky buildman test
2015-10-27 17:02:15 -04:00
Joseph Schorr
e0d715024c
Fix typo in test
2015-10-23 16:39:40 -04:00
Joseph Schorr
75b36c0f33
Update test
2015-06-25 23:13:33 -04:00
Joseph Schorr
b4c39e8ec0
Fix ephemeral build manager to ask for watches in index order with no gaps
2015-06-23 17:11:46 -04:00
Jake Moshenko
f767fc4d03
Track whether builders ever came online in etcd. Mark builds which never successfully heartbeated as incomplete.
2015-06-10 16:19:51 -04:00
Jake Moshenko
adce277d49
Fix the tests to use the default realm and building config.
2015-02-02 12:05:36 -05:00
Joseph Schorr
3872d29de9
Add a transaction around the extend_processing call
2015-01-29 18:40:41 -05:00
Jake Moshenko
b33ee1a474
Register existing builders to watch their expirations.
2015-01-05 11:21:36 -05:00
Jake Moshenko
a9839021af
When the etcd key tracking realms is first created the action is create, not set.
2014-12-31 11:46:02 -05:00
Jake Moshenko
cc70225043
Generalize the ephemeral build managers so that any manager may manage a builder spawned by any other manager.
2014-12-31 11:33:56 -05:00
Jake Moshenko
4a2295373f
Fix tests for no timeout watches.
2014-12-23 15:35:34 -05:00
Jake Moshenko
4e22e22ba1
We have to serialize our build data before sending it to etc.
2014-12-23 14:09:04 -05:00
Jake Moshenko
709e571b78
Handle read timeouts from etcd when watching a key.
2014-12-23 12:13:49 -05:00
Jake Moshenko
055a6b0c37
Add a total maximum time that a machine is allowed to stick around before we terminate it more forcefully.
2014-12-23 11:18:10 -05:00
Jake Moshenko
34bf92673b
Add support for adjusting etcd ttl on job_heartbeat. Switch the heartbeat method to a coroutine.
2014-12-22 17:24:44 -05:00
Jake Moshenko
2b6c2a2a50
Improve tests for the ephemeral build manager.
2014-12-22 16:22:07 -05:00
Jake Moshenko
12ee8e0fc0
Switch a few of the buildman methods to coroutines in order to support network calls in methods. Add a test for the ephemeral build manager.
2014-12-22 12:14:16 -05:00
