Add a delay before killing the builder process #2201
Conversation
Sometimes, the builder process is killed while the build succeeded. It seems this is because the signal is sent after file descriptors are closed (in `do_exit`) and before the process reaches the terminated state. This leads to a build failure (failed due to signal 9). To mitigate this issue, a delay (~10s) is introduced before sending the kill signal to the build process if it doesn't reach the terminated state. Fixes NixOS#2176
|
I'm not sure about the root cause of the issue #2176 but with this patch, I'm no longer able to reproduce it (30000 iterations). |
|
Adding arbitrary timeouts doesn't seem like a good idea. For instance, why is 10 seconds an appropriate value? Doesn't this mean that the build can still fail randomly under high load? It would probably be better to fix the shell not to close stdout/stderr before exiting. |
|
I agree arbitrary timeouts are not a good idea but I don't know how to "fix" this bug otherwise. Moreover we currently have this timeout but we don't know its value! I think the problem comes from the way the worker knows the builder finished: we wait for closed file descriptors. But to terminate a process, the Linux kernel seems to first close file descriptors and then, it changes the status of the process. It is still possible the build fails under high load. But from my basic experiments, I was no longer able to hit this issue with 10sec. Do you suggest to prevent user build code to close stdout/stderr in order to remove the kill expression? I'll investigate this way. |
|
Maybe it would be better to switch 21948de to using something non-blocking (like catching SIGCHLD + self-pipe trick or signalfd). |
|
Ran into this problem when running Nix on AWS Lambda. (https://github.com/tomberek/lambdanix). For what it's worth, this patch fixed it. Not sure why. I was also completely abusing the pseud-terminal interface (AWS does not give you access to one), so this could be other issues interacting. |
|
I marked this as stale due to inactivity. → More info |
|
I closed this issue due to inactivity. → More info |
Sometimes, the builder process is killed while the build succeeded. It
seems this is because the signal is sent after file descriptors are
closed (in
do_exit) and before the process reaches the terminatedstate. This leads to a build failure (failed due to signal 9).
To mitigate this issue, a delay (~10s) is introduced before sending the kill
signal to the build process if it doesn't reach the terminated state.
Fixes #2176