Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abort unsupported build steps #727

Merged
merged 4 commits into from Mar 27, 2020
Merged

Abort unsupported build steps #727

merged 4 commits into from Mar 27, 2020

Conversation

edolstra
Copy link
Member

If we don't see machine that supports a build step for max_unsupported_time seconds, the step is aborted. The default is 0, which is appropriate for Hydra installations that don't provision missing machines dynamically.

If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.
@grahamc
Copy link
Member

grahamc commented Mar 26, 2020

Seems like a good idea. I wonder how we can setup alerting on hydra.n.o to alert us before they get aborted. We might have all the metrics we need already.

@edolstra
Copy link
Member Author

I've added a metric to keep track of the current number of unsupported runnable steps.

@grahamc
Copy link
Member

grahamc commented Mar 26, 2020

Looking ...

@grahamc
Copy link
Member

grahamc commented Mar 26, 2020

I wonder what it would take to export this information at the prometheus endpoint as well

sub prometheus :Local Args(0) {
though a method which got all of the gauge information by default would be nicest.

@grahamc
Copy link
Member

grahamc commented Mar 26, 2020

(btw I don't consider this blocking, we can re-export it elsewhere like most of these metrics)

@edolstra edolstra merged commit a7540b1 into flake Mar 27, 2020
@grahamc grahamc deleted the fail-unsupported branch February 24, 2021 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants