Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: NixOS/hydra
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 63188667b848
Choose a base ref
...
head repository: NixOS/hydra
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 3e73a2f7098e
Choose a head ref
  • 1 commit
  • 1 file changed
  • 1 contributor

Commits on Jul 27, 2020

  1. Fix issue #614: restart queue/evaluator on sufficient disk space avai… ( 

    #777)

* Fix issue #614: restart queue/evaluator on sufficient disk space available.

* Only try to stop the service if it is currently running.

* Use named variables and added restarting message.
    @kquick
    kquick authored Jul 27, 2020
    Copy the full SHA
    3e73a2f View commit details
Showing with 24 additions and 8 deletions.
  1. +24 −8 hydra-module.nix
32 changes: 24 additions & 8 deletions hydra-module.nix
View file
Original file line number Diff line number Diff line change
@@ -395,17 +395,33 @@ in

# If there is less than a certain amount of free disk space, stop
# the queue/evaluator to prevent builds from failing or aborting.
# Leaves a tag file indicating this reason; if the tag file exists
# and disk space is above the threshold + 10GB, the queue/evaluator will be
# restarted; starting it if it is already started is not harmful.
systemd.services.hydra-check-space =
{ script =
''
if [ $(($(stat -f -c '%a' /nix/store) * $(stat -f -c '%S' /nix/store))) -lt $((${toString cfg.minimumDiskFree} * 1024**3)) ]; then
echo "stopping Hydra queue runner due to lack of free space..."
systemctl stop hydra-queue-runner
fi
if [ $(($(stat -f -c '%a' /nix/store) * $(stat -f -c '%S' /nix/store))) -lt $((${toString cfg.minimumDiskFreeEvaluator} * 1024**3)) ]; then
echo "stopping Hydra evaluator due to lack of free space..."
systemctl stop hydra-evaluator
fi
spaceleft=$(($(stat -f -c '%a' /nix/store) * $(stat -f -c '%S' /nix/store)))
spacestopstart() {
service=$1
minFreeGB=$2
if [ $spaceleft -lt $(($minFreeGB * 1024**3)) ]; then
if [ $(systemctl is-active $service) == active ]; then
echo "stopping $service due to lack of free space..."
systemctl stop $service
date > /var/lib/hydra/.$service-stopped-minspace
fi
else
if [ $spaceleft -gt $(( ($minFreeGB + 10) * 1024**3)) -a \
-r /var/lib/hydra/.$service-stopped-minspace ] ; then
rm /var/lib/hydra/.$service-stopped-minspace
echo "restarting $service due to newly available free space..."
systemctl start $service
fi
fi
}
spacestopstart hydra-queue-runner ${toString cfg.minimumDiskFree}
spacestopstart hydra-evaluator ${toString cfg.minimumDiskFreeEvaluator}
'';
startAt = "*:0/5";
};