Commit graph

269 commits

Author SHA1 Message Date
Joseph Schorr
818ea38dac Add repo-specific reporting of repository builds 2016-09-09 15:36:54 -04:00
Brad Ison
2a1cf2bfd1 Always pull latest image in k8s builds 2016-09-08 15:00:12 -04:00
Joseph Schorr
e67b95ae04 Change log level of an expected log message 2016-08-31 17:25:54 -04:00
Brad Ison
6365b6dbfb Set defaults in qemu-coreos entrypoint 2016-08-31 15:49:21 -04:00
Joseph Schorr
9e6e3a6c94 Remove our names from the checked in keys
This means they won't go out in the QE binary, nor will be viewable on the ephemeral build nodes

Longer term we should probably move these into the config dir
2016-08-30 18:02:05 -04:00
Joseph Schorr
e17e0e4172 Add log for when the job key is written 2016-08-30 14:08:56 -04:00
Joseph Schorr
2fe896ba6a Restore retries of jobs not started and add some leeway to the processing time 2016-08-30 13:57:26 -04:00
Joseph Schorr
292abb5395 Better handling and logging of exceptions in build manager
Also increases the setup timeout for EC2
2016-08-30 13:52:36 -04:00
Brad Ison
e5cc97d462 Time qemu-img resize in qemu-coreos startup 2016-08-29 15:23:30 -04:00
Joseph Schorr
cd2d0341a7 Fix k8s builder to use the declared volume size
Fixes #1773
2016-08-29 15:16:28 -04:00
Joseph Schorr
bc670611ef Increase the timeout on the atomic lock
Some nodes were still performing the action twice when falling outside of the 30s window
2016-08-23 12:50:38 -04:00
Joseph Schorr
3112388004 Fix multiple reporting of incomplete 2016-08-17 16:01:28 -04:00
Joseph Schorr
0b50928900 Fix build start check for the ephemeral case 2016-08-16 17:18:57 -04:00
Joseph Schorr
433b157531 Add extra check to ensure a build cannot be started without on_ready called 2016-08-16 16:38:48 -04:00
Joseph Schorr
5e1a117ff3 Delete the job first to prevent Kubernetes from starting another pod 2016-08-16 16:33:43 -04:00
Joseph Schorr
742e153133 Fix watch of the jobs key in the build manager 2016-08-16 15:43:09 -04:00
Joseph Schorr
313d65a6a4 Make sure the etcd watch coroutines get called 2016-08-16 13:02:27 -04:00
josephschorr
cddba20ffe Merge pull request #1731 from coreos-inc/k8s-cleanup
Cleanup old executions that never start
2016-08-15 17:00:13 -04:00
Joseph Schorr
d78361b041 Cleanup old executions that never start
Fixes #1727
2016-08-15 16:54:02 -04:00
Brad Ison
d37f32b9c7 Add bison's SSH key to builders 2016-08-15 15:53:26 -04:00
Joseph Schorr
acdfc9369d Allow the version of CoreOS to be specified when building QEMU image 2016-08-05 16:46:11 -04:00
Joseph Schorr
c29f9ccc7f Fix TTL on heartbeat in etcd
Until now, once the heartbeat has expired, we would issue a TTL that is negative, which causes etcd to either raise an exception or simply ignore the expiration (depending on the version of etcd). This change ensures that once the key is expired, it is removed immediately via a set of a TTL of 0. Also adds tests for this case and the normal expiration case.
2016-08-03 11:15:03 -04:00
Joseph Schorr
428a7cb435 Fix decreased setup timeout on ephemeral build manager 2016-07-22 13:35:38 -04:00
Joseph Schorr
392242d20b Another fix for the record keeping in buildman
Adds some more mocked tests as well
2016-07-22 12:01:30 -04:00
Joseph Schorr
68baa51d55 Fix cross-manager handling of realm components 2016-07-21 15:47:25 -04:00
Joseph Schorr
4420b1bac9 Add temporary back-compat shims for the build manager 2016-07-20 13:41:01 -04:00
Joseph Schorr
2c1880b944 Bug fixes, refactoring and "new" tests for the build manager
- Fixes various bugs introduced in the most recent build system commit
- Refactors state management in the build manager to be cleaner and more contained
- Adds back in the mock-based tests, fixed to not use threads and adjusted for the refactoring
- Adds some more simplified unit tests around non-etch related flows
2016-07-18 13:46:48 -04:00
Joseph Schorr
74b87fa813 Build manager cleanup and more logging 2016-07-14 14:33:14 -04:00
Joseph Schorr
d8b72e8503 Switch to using a defined branch and not always pulling the VM image 2016-07-08 17:53:25 -04:00
Joseph Schorr
3d4af78f01 Fix label to never allow a space (which breaks Kubernetes) 2016-07-08 17:09:06 -04:00
Joseph Schorr
811413fe9c Add multiple executor and whitelist support to build manager 2016-07-08 15:50:51 -04:00
Joseph Schorr
7471d0e35f Small code cleanup before whitelist addition 2016-07-08 15:50:51 -04:00
Colin Hom
1e3351f3f4 local-docker.sh now accepts env vars 2016-07-08 15:50:51 -04:00
Colin Hom
bc13333f20 Kubernetes build worker 2016-07-08 15:50:51 -04:00
Joseph Schorr
713ba3abaf Further updates to the Prometheus client code 2016-07-01 14:16:51 -04:00
Matt Jibson
3d9acf2fff Use prometheus as a metric backend
This entails writing a metric aggregation program since each worker has its
own memory, and thus own metrics because of python gunicorn. The python
client is a simple wrapper that makes web requests to it.
2016-07-01 14:16:50 -04:00
Joseph Schorr
1173192739 Move channel back, as it is referenced by generate_cloud_config 2016-06-22 17:25:06 -04:00
Joseph Schorr
61695eb439 Allow the build node AMI to be overridden in config 2016-06-22 15:13:54 -04:00
josephschorr
20a6fdc73f Merge pull request #1557 from jzelinskie/buildargs
buildman: mark missing buildargs as failure
2016-06-20 14:40:17 -04:00
Jimmy Zelinskie
871c1634ed buildman: mark missing buildargs as failure 2016-06-17 18:33:54 -04:00
Joseph Schorr
7292524d69 Add a cloud watch metric when we fail to start a build via EC2
Fixes #1555
2016-06-17 16:19:57 -04:00
Jimmy Zelinskie
5298452fa7 builder cloudconfig: shutdown server after 3 hours (#1554) 2016-06-17 16:03:40 -04:00
Joseph Schorr
f9469a84b3 Make the size of the build node HDD configurable
Fixes #1520
2016-06-06 11:35:10 -04:00
Jimmy Zelinskie
7d356c451b buildman: fix misspell 2016-06-03 15:42:14 -04:00
Jimmy Zelinskie
44b56ae2cf queue: explicitly declare ordering requirement
This change defaults the ordering requirement of queue items to be off
and only enables it for the build manager. This should make the queries
for getting queueitems significantly faster for every other use case.
2016-05-27 14:44:30 -04:00
Jimmy Zelinskie
79aa78906a buildman: refresh and add Evan's key to builders 2016-05-24 14:05:39 -04:00
Joseph Schorr
5262535945 Boto error_code is a string, not the HTTP status code 2015-12-23 15:12:01 -05:00
Jimmy Zelinskie
601b99a083 buildman: add git checkout failure 2015-12-16 14:49:37 -05:00
Joseph Schorr
773e73861f Change error into info in build manager
Fixes #1046
2015-12-09 14:30:14 -05:00
josephschorr
c06e5cc9c7 Merge pull request #1002 from coreos-inc/buildertagexc
Add timeout and failure if an EC2 instance could not be found when ta…
2015-12-09 14:28:31 -05:00