NixOS tests: Wait for shell for 10x longer (50m) #49441
Conversation
|
Another test to try, independent of this, might be reverting the new backdoor. |
I agree, though I'm waiting for Domen and co. to convince me that this is superstitious. :P |
|
It seems to be your best lead, so why wait? :) |
srhb
force-pushed
the
debug-hydra-failures
branch
from
e8a3b96
to
9bc10e1
Compare
|
Because it's a lead based solely on the timing of the commits. I can't reproduce any change in the behaviour locally with any timeout with/without the backdoor commit. Of course, it's really difficult to replicate the conditions on the builder machines realistically, but I'd like to have something slightly more solid than hand-waving. Mind, whether we do the timeout test first or the backdoor test makes little difference to me. I just want more data, and either helps. :) |
|
Let's give this a try first. I also had a hunch that the failures could be related to the new VM backdoor, but no evidence at all. |
|
@srhb should we backport this to 18.09? Might be easier to see the effects there, and it shouldn't break anything. |
|
@xeji I actually prefer having it on one but not the other. Though maybe it should have been on release-18.09 instead... |
|
Unfortunately/thankfully a lot of other changes happened at about the same time to hopefully unblock the channels. Just got the first indication that this is still a problem, and not really a timeout but some other issue: https://hydra.nixos.org/build/83438035/nixlog/19/tail |
See NixOS#49441 for an earlier attempt, which was subsequently reverted. I am assuming that doubling the time will be sufficient if the machine is overloaded since so many of the tests already pass at 5 minutes, while still not holding back failures for needlessly long.
See #49441 for an earlier attempt, which was subsequently reverted. I am assuming that doubling the time will be sufficient if the machine is overloaded since so many of the tests already pass at 5 minutes, while still not holding back failures for needlessly long. (cherry picked from commit b28b37e)
See NixOS#49441 for an earlier attempt, which was subsequently reverted. I am assuming that doubling the time will be sufficient if the machine is overloaded since so many of the tests already pass at 5 minutes, while still not holding back failures for needlessly long.
Motivation for this change
#49384
This increases the timeout waiting for the shell to the virtual machine to connect by 10 times, to 50 minutes, in order to investigate the failures that frequently plague and block our tested set, especially the tests that have a restart incorporated (installer.*)
I have very low confidence that this is the actual issue, so it is mostly intended as a check. I find it more likely that the shell will never connect, even though the machine has finished booting, and that somehow we recently (~last month) introduced a single change that caused this, or caused this to occur much more frequently.
Things done
sandboxinnix.confon non-NixOS)nix-shell -p nox --run "nox-review wip"./result/bin/)nix path-info -Sbefore and after)