Joseph Schorr
f50bb8a1ce
Add missing call to set_phase when a build doesn't start
...
This change fixes the build manager ephemeral executor to tell the overall build server to call set_phase when a build never starts. Before this change, we'd properly adjust the queue item, but not the repo build row or the logs, which is why users just saw "Preparing Build Node", with no indicating the node failed to start.
Fixes #1904
2016-09-30 14:54:49 +02:00
Evan Cordell
832ee89923
Add duration metric collector decorator ( #1885 )
...
Track time-to-start for builders
Track time-to-build for builders
Track ec2 builder fallbacks
Track build time
2016-09-29 15:44:06 -04:00
josephschorr
ad4efba802
Merge pull request #1830 from coreos-inc/superuser-dashboard
...
Add prometheus stats to enable better dashboarding
2016-09-26 17:19:22 +02:00
Joseph Schorr
1571b2867a
Add executor name to the build metric
2016-09-16 16:26:04 -04:00
Joseph Schorr
f9f60b9faf
Fix some issues around state in the build managers
...
- Make sure to cleanup the job if the executor could not be started
- Change the setup leeway to further ensure there isn't any crossover between the queue item timing out and the cleanup of the jobs
- Make the lock used for marking jobs as internal error extremely long, but also based on the execution ID. This should ensure we don't get duplicates while allowing different executions to be handled properly.
- Make sure to invoke the callback update for the queue before we run off to etcd; should reduce certain timeouts
Hopefully Fixes #1836
2016-09-15 14:37:45 -04:00
Joseph Schorr
e67b95ae04
Change log level of an expected log message
2016-08-31 17:25:54 -04:00
Joseph Schorr
e17e0e4172
Add log for when the job key is written
2016-08-30 14:08:56 -04:00
Joseph Schorr
292abb5395
Better handling and logging of exceptions in build manager
...
Also increases the setup timeout for EC2
2016-08-30 13:52:36 -04:00
Joseph Schorr
bc670611ef
Increase the timeout on the atomic lock
...
Some nodes were still performing the action twice when falling outside of the 30s window
2016-08-23 12:50:38 -04:00
Joseph Schorr
3112388004
Fix multiple reporting of incomplete
2016-08-17 16:01:28 -04:00
Joseph Schorr
742e153133
Fix watch of the jobs key in the build manager
2016-08-16 15:43:09 -04:00
Joseph Schorr
313d65a6a4
Make sure the etcd watch coroutines get called
2016-08-16 13:02:27 -04:00
Joseph Schorr
d78361b041
Cleanup old executions that never start
...
Fixes #1727
2016-08-15 16:54:02 -04:00
Joseph Schorr
c29f9ccc7f
Fix TTL on heartbeat in etcd
...
Until now, once the heartbeat has expired, we would issue a TTL that is negative, which causes etcd to either raise an exception or simply ignore the expiration (depending on the version of etcd). This change ensures that once the key is expired, it is removed immediately via a set of a TTL of 0. Also adds tests for this case and the normal expiration case.
2016-08-03 11:15:03 -04:00
Joseph Schorr
428a7cb435
Fix decreased setup timeout on ephemeral build manager
2016-07-22 13:35:38 -04:00
Joseph Schorr
392242d20b
Another fix for the record keeping in buildman
...
Adds some more mocked tests as well
2016-07-22 12:01:30 -04:00
Joseph Schorr
68baa51d55
Fix cross-manager handling of realm components
2016-07-21 15:47:25 -04:00
Joseph Schorr
4420b1bac9
Add temporary back-compat shims for the build manager
2016-07-20 13:41:01 -04:00
Joseph Schorr
2c1880b944
Bug fixes, refactoring and "new" tests for the build manager
...
- Fixes various bugs introduced in the most recent build system commit
- Refactors state management in the build manager to be cleaner and more contained
- Adds back in the mock-based tests, fixed to not use threads and adjusted for the refactoring
- Adds some more simplified unit tests around non-etch related flows
2016-07-18 13:46:48 -04:00
Joseph Schorr
74b87fa813
Build manager cleanup and more logging
2016-07-14 14:33:14 -04:00
Joseph Schorr
811413fe9c
Add multiple executor and whitelist support to build manager
2016-07-08 15:50:51 -04:00
Joseph Schorr
7471d0e35f
Small code cleanup before whitelist addition
2016-07-08 15:50:51 -04:00
Colin Hom
bc13333f20
Kubernetes build worker
2016-07-08 15:50:51 -04:00
Joseph Schorr
713ba3abaf
Further updates to the Prometheus client code
2016-07-01 14:16:51 -04:00
Joseph Schorr
773e73861f
Change error into info in build manager
...
Fixes #1046
2015-12-09 14:30:14 -05:00
Joseph Schorr
edd9a03af5
Catch additional key not found exception
...
Fixes #806
2015-12-01 12:29:58 -05:00
Joseph Schorr
0f37e66cc8
Better error handling for the build manager
...
Fixes #604
2015-10-13 11:40:07 -04:00
Matt Jibson
bba1557437
Monitor queue adds and EC2 node starts
...
fixes #157
see #304
2015-09-18 16:21:16 -04:00
Joseph Schorr
6655c7f745
Add exception handling that doesn't log the read-timeout exception
...
Note: This is a *hack* and needs to be replaced with proper code ASAP
2015-06-25 23:35:29 -04:00
Joseph Schorr
6e6610f31a
Switch to a 30s maximum timeout
2015-06-25 23:08:49 -04:00
Joseph Schorr
ecebc06343
Update comment now that restarter is abstracted
2015-06-25 21:53:42 -04:00
Joseph Schorr
9f5f71398c
Abstract out the concept of a restart function
2015-06-25 21:40:50 -04:00
Joseph Schorr
52fa9aad5b
Fix etcd watching
...
Etcd can miss events on watches if they are occurring fast enough, so if we can get an exception indicating that we've missed an index, we reset the state of our local tracking structures by re-reading the *full* list and starting a new watch at HEAD
2015-06-25 21:22:39 -04:00
Joseph Schorr
b4c39e8ec0
Fix ephemeral build manager to ask for watches in index order with no gaps
2015-06-23 17:11:46 -04:00
Joseph Schorr
c2dc1c9b75
Handle case where etcd key is already removed on job complete
2015-06-17 15:02:58 -04:00
Jake Moshenko
c435f5c127
Add a comment about why we are taking a lock when terminating a builder machine.
2015-06-10 16:19:51 -04:00
Jake Moshenko
f767fc4d03
Track whether builders ever came online in etcd. Mark builds which never successfully heartbeated as incomplete.
2015-06-10 16:19:51 -04:00
Jake Moshenko
884fedd229
Improve the log messages in the buildman.
2015-06-10 16:19:51 -04:00
Jake Moshenko
d31e25d5cd
Allow the individual build manager types to specify how long the queue should wait before retring a job that fails to schedule.
2015-06-10 16:19:50 -04:00
Joseph Schorr
f82831bff6
Log the etcd exception so we can debug this issue
2015-06-09 20:33:55 -04:00
Jake Moshenko
0349f3f1a3
Handle the case where YAML config returns a list not a tuple.
2015-03-26 14:53:56 -04:00
Jimmy Zelinskie
8589871f43
buildman: rm unused imports
2015-03-09 13:04:16 -04:00
Jake Moshenko
5c68e52fce
Really really fix the exception handling.
2015-02-27 17:33:46 -05:00
Jake Moshenko
cf5bc6f0be
Properly catch multiple exceptions.
2015-02-27 17:32:10 -05:00
Jake Moshenko
857c3e2959
Start catching etcd key errors as well.
2015-02-27 17:10:15 -05:00
Joseph Schorr
4551b3a957
Remove the boto timeout set (doesn't work anyway) and add some better logging to the scheduler
2015-02-25 16:00:14 -05:00
Joseph Schorr
5dd78f76c7
Add additional logging, timeouts, and exception checks
2015-02-25 15:15:22 -05:00
Joseph Schorr
afe7e14254
Add better exception handling and logging to the ephemeral build manager
2015-02-25 12:09:14 -05:00
Joseph Schorr
524705b88c
Get dashboard working and upgrade bootstrap. Note: the bootstrap fixes will be coming in the followup CL
2015-02-17 19:15:54 -05:00
Jake Moshenko
a4b0c8698d
Allow the key prefixes in etcd to be configurable.
2015-02-02 12:00:19 -05:00
