Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nixos/slurm: set slurmd KillMode and add extraConfigPaths #50862

Merged
merged 4 commits into from Dec 10, 2018
Merged

nixos/slurm: set slurmd KillMode and add extraConfigPaths #50862

merged 4 commits into from Dec 10, 2018

Conversation

markuskowa
Copy link
Member

Motivation for this change
  • set slurmd systemd service KillMode to process: In the standard configuration all jobs are killed when slurmd is restarted (e.g. when the configuration is updated). This is quite disruptive in a production environment. Setting KillMode=process kills only slurmd but not slurmstepd and thus not the running job. This is also the standard behavior of the systemd files delivered from slurm.
  • add new option extraConfigPaths: This allows to add custom config files to slurm configuration. All plugins that come with config files need to be in the same directory as slurm.conf.
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
    nixos/tests/slurm.nix succeeds.
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@markuskowa
nixos/slurm: set slurmd KillMode to process
ae93ed0 
The default of systemd is to kill the
the whole cgroup of a service. For slurmd
this means that all running jobs get killed
as well whenever the configuration is updated (and activated).

To avoid this behaviour we set "KillMode=process"
to kill only slurmd on reload. This is how
slurm configures the systemd service.

See:
https://bugs.schedmd.com/show_bug.cgi?id=2095#c24
SchedMD/slurm@508f866
@markuskowa
Copy link
Member Author

@GrahamcOfBorg test slurm

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.slurm

Partial log (click to expand)

Cannot nix-instantiate `tests.slurm' because:
error: while evaluating 'recursiveUpdate' at /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:415:26, called from /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:148:28:
while evaluating 'recursiveUpdateUntil' at /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:384:37, called from /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:416:5:
while evaluating 'zipAttrsWith' at /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:347:21, called from /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:394:8:
while evaluating 'zipAttrsWithNames' at /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:332:33, called from /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/lib/attrsets.nix:347:27:
while evaluating the attribute 'slurm' at /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/nixos/tests/all-tests.nix:189:3:
while evaluating 'handleTest' at /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/nixos/tests/all-tests.nix:189:11:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/nix-test-rs-15/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-15/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/06pf7xk36qi3r90kkj60pspr8sww4cpi-grahamc-aarch64-community-15' is forbidden in restricted mode

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.slurm

Partial log (click to expand)

error: while evaluating 'recursiveUpdate' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:415:26, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:148:28:
while evaluating 'recursiveUpdateUntil' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:384:37, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:416:5:
while evaluating 'zipAttrsWith' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:347:21, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:394:8:
while evaluating 'zipAttrsWithNames' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:332:33, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:347:27:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/lists.nix:113:41, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:347:46:
while evaluating the attribute 'slurm' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:189:3:
while evaluating 'handleTest' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:189:11:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/41dzsyb0a8lagqklasc5xb5nphqb8f8p-gleber-bastion' is forbidden in restricted mode

@markuskowa
Copy link
Member Author

@GrahamcOfBorg test slurm

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.slurm

Partial log (click to expand)

Cannot nix-instantiate `tests.slurm' because:
error: while evaluating 'recursiveUpdate' at /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:415:26, called from /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:148:28:
while evaluating 'recursiveUpdateUntil' at /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:384:37, called from /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:416:5:
while evaluating 'zipAttrsWith' at /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:347:21, called from /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:394:8:
while evaluating 'zipAttrsWithNames' at /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:332:33, called from /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/lib/attrsets.nix:347:27:
while evaluating the attribute 'slurm' at /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/nixos/tests/all-tests.nix:189:3:
while evaluating 'handleTest' at /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/nixos/tests/all-tests.nix:189:11:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/nix-test-rs-17/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-17/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/mib2g2wj623bqgpn3rg137vg8g52s4y1-grahamc-aarch64-community-17' is forbidden in restricted mode

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.slurm

Partial log (click to expand)

error: while evaluating 'recursiveUpdate' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:415:26, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:148:28:
while evaluating 'recursiveUpdateUntil' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:384:37, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:416:5:
while evaluating 'zipAttrsWith' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:347:21, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:394:8:
while evaluating 'zipAttrsWithNames' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:332:33, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:347:27:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/lists.nix:113:41, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/attrsets.nix:347:46:
while evaluating the attribute 'slurm' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:189:3:
while evaluating 'handleTest' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:17:22, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:189:11:
while evaluating 'discoverTests' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:13:19, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/tests/all-tests.nix:18:5:
access to path '/nix/store/sg8f9b749gym8bsg2khbxfhqwkci27ma-gleber-bastion' is forbidden in restricted mode

@markuskowa markuskowa mentioned this pull request Nov 21, 2018
Closed
@Mic92
Copy link
Member

Mic92 commented Nov 21, 2018

@GrahamcOfBorg build nixosTests.slurm

this is a workaround

@GrahamcOfBorg
Copy link

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: nixosTests.slurm

Partial log (click to expand)

while evaluating the attribute 'linux_4_14' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14334:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14334:16:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/linux-4.14.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:69:12:
while evaluating 'buildLinux' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14632:16, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/linux-4.14.nix:5:1:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14632:23:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/generic.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:69:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/generic.nix:51:1

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: nixosTests.slurm

Partial log (click to expand)

while evaluating the attribute 'values' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/lib/modules.nix:458:7:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/lib/modules.nix:348:28, called from /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/lib/modules.nix:348:17:
while evaluating 'dischargeProperties' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/lib/modules.nix:416:25, called from /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/lib/modules.nix:349:62:
while evaluating the attribute 'value' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/lib/types.nix:288:55:
while evaluating the attribute 'ExecStart' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/nixos/modules/services/computing/slurm/slurm.nix:340:9:
while evaluating the attribute 'args' of the derivation 'wrappedSlurm' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/pkgs/stdenv/generic/make-derivation.nix:180:11:
while evaluating the attribute 'text' of the derivation 'builder.sh' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/pkgs/stdenv/generic/make-derivation.nix:180:11:
while evaluating the attribute 'paths' of the derivation 'etc-slurm' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/pkgs/stdenv/generic/make-derivation.nix:180:11:
undefined variable 'extraConfigPaths' at /var/lib/gc-of-borg/nix-test-rs-26/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-26/nixos/modules/services/computing/slurm/slurm.nix:49:60

@GrahamcOfBorg
Copy link

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: nixosTests.slurm

Partial log (click to expand)

while evaluating anonymous function at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/lists.nix:113:41, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/modules.nix:348:17:
while evaluating anonymous function at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/modules.nix:348:28, called from undefined position:
while evaluating 'dischargeProperties' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/modules.nix:416:25, called from /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/modules.nix:349:62:
while evaluating the attribute 'value' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/lib/types.nix:288:55:
while evaluating the attribute 'ExecStart' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/modules/services/computing/slurm/slurm.nix:340:9:
while evaluating the attribute 'args' of the derivation 'wrappedSlurm' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/pkgs/stdenv/generic/make-derivation.nix:180:11:
while evaluating the attribute 'text' of the derivation 'builder.sh' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/pkgs/stdenv/generic/make-derivation.nix:180:11:
while evaluating the attribute 'paths' of the derivation 'etc-slurm' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/pkgs/stdenv/generic/make-derivation.nix:180:11:
undefined variable 'extraConfigPaths' at /var/lib/gc-of-borg/nix-root/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/gleber-bastion/nixos/modules/services/computing/slurm/slurm.nix:49:60

@Mic92
Copy link
Member

Mic92 commented Nov 21, 2018

The evaluation error you see is now legitimate the same test evaluates locally.

@markuskowa
Copy link
Member Author

Sorry, looks like I forgot to push the latest version.

@Mic92
Copy link
Member

Mic92 commented Nov 22, 2018

@GrahamcOfBorg build nixosTests.slurm

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: nixosTests.slurm

Partial log (click to expand)

killing node2 (pid 597)
killing node1 (pid 609)
killing node3 (pid 621)
killing control (pid 633)
killing dbd (pid 645)
killing submit (pid 657)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/4sj5qcykcmxg6m1nyv6hk6s4jbicaxmy-vm-test-run-slurm.drv' failed with exit code 1
error: build of '/nix/store/4sj5qcykcmxg6m1nyv6hk6s4jbicaxmy-vm-test-run-slurm.drv' failed

@GrahamcOfBorg
Copy link

Unexpected error: unexpected build failure on aarch64-linux (full log)

Attempted: nixosTests.slurm

Partial log (click to expand)

/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/libexec/zfs/zpool.d/encdev: interpreter directive changed from "/bin/sh" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/sh"
  CC      util/id.o
/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/libexec/zfs/zpool.d/enc: interpreter directive changed from "/bin/sh" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/sh"
/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/sbin/fsck.zfs: interpreter directive changed from "/bin/sh" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/sh"
  CC      net/wireless/wext-proc.o
/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/share/initramfs-tools/scripts/local-top/zfs: interpreter directive changed from "/bin/sh" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/sh"
/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/share/initramfs-tools/hooks/zfs: interpreter directive changed from "/bin/sh" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/sh"
/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/share/zfs/test-runner/bin/test-runner.py: interpreter directive changed from "/usr/bin/python" to "/nix/store/27l1iai8hqchv3iv6k4k1fah4j424ik2-python-2.7.15/bin/python"
/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/share/zfs/zfs-helpers.sh: interpreter directive changed from "/bin/bash" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/bash"
/nix/store/x8rywycnfhlq5ikr5rki37k4hwk2h2nj-zfs-user-0.7.12/share/zfs/smb.sh: interpreter directive changed from "/bin/bash" to "/nix/store/n1kfdl37qpzh3xn6klbym1ay6xpxvmw1-bash-4.4-p23/bin/bash"

@GrahamcOfBorg
Copy link

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: nixosTests.slurm

Partial log (click to expand)

while evaluating the attribute 'linux_4_14' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14327:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14327:16:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/linux-4.14.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:69:12:
while evaluating 'buildLinux' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14625:16, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/linux-4.14.nix:5:1:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14625:23:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/generic.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:69:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/generic.nix:51:1

@markuskowa
Copy link
Member Author

This seems to be a timing issue now. The test passes just fine locally.
@GrahamcOfBorg build nixosTests.slurm

@GrahamcOfBorg
Copy link

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: nixosTests.slurm

Partial log (click to expand)

while evaluating the attribute 'linux_4_14' at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:14327:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:14327:16:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.14.nix:1:1, called from /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:69:12:
while evaluating 'buildLinux' at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:14625:16, called from /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.14.nix:5:1:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:14625:23:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:1:1, called from /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:69:12:
assertion failed at /private/var/lib/ofborg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:51:1

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: nixosTests.slurm

Partial log (click to expand)

killing control (pid 597)
killing node1 (pid 609)
killing dbd (pid 621)
killing node3 (pid 633)
killing node2 (pid 645)
killing submit (pid 657)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/bhh3gwhq483a4gzx5xxh677lvk121rip-vm-test-run-slurm.drv' failed with exit code 1
error: build of '/nix/store/bhh3gwhq483a4gzx5xxh677lvk121rip-vm-test-run-slurm.drv' failed

@markuskowa
Copy link
Member Author

@GrahamcOfBorg build nixosTests.slurm

@GrahamcOfBorg
Copy link

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: nixosTests.slurm

Partial log (click to expand)

while evaluating the attribute 'linux_4_14' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14327:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14327:16:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/linux-4.14.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:69:12:
while evaluating 'buildLinux' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14625:16, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/linux-4.14.nix:5:1:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:108:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/top-level/all-packages.nix:14625:23:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:67:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:112:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/generic.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/lib/customisation.nix:69:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndndx-vm/pkgs/os-specific/linux/kernel/generic.nix:51:1

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nixosTests.slurm

Partial log (click to expand)

cleaning up
killing node3 (pid 631)
killing submit (pid 643)
killing node2 (pid 657)
killing node1 (pid 670)
killing dbd (pid 683)
killing control (pid 696)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/bmmi3r6d1pl94z20na2xx2nldq49f2jb-vm-test-run-slurm

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: nixosTests.slurm

Partial log (click to expand)

killing dbd (pid 597)
killing node2 (pid 609)
killing control (pid 621)
killing node3 (pid 633)
killing submit (pid 645)
killing node1 (pid 657)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/lb9a6x4r9gfi9qzwswwwlr8vi0m1zbmw-vm-test-run-slurm.drv' failed with exit code 1
error: build of '/nix/store/lb9a6x4r9gfi9qzwswwwlr8vi0m1zbmw-vm-test-run-slurm.drv' failed

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nixosTests.slurm

Partial log (click to expand)

cleaning up
killing dbd (pid 631)
killing submit (pid 643)
killing node2 (pid 657)
killing control (pid 670)
killing node3 (pid 683)
killing node1 (pid 696)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/5dfwrdvhps8ackbahkdskiy83g4nn1q9-vm-test-run-slurm

@markuskowa
Copy link
Member Author

markuskowa commented Nov 22, 2018 •
edited

I am not sure how to fix this. The test behaves different on my local (NixOS) machine.

@markuskowa
Copy link
Member Author

@GrahamcOfBorg build nixosTests.slurm

@Mic92
Copy link
Member

Mic92 commented Dec 9, 2018

some sort of timing issue?

@markuskowa
tests/slurm: wait for open DBD port
8eee1ec 
This makes tests more reliable. It seems
that waitForUnit(slurmdbd.service) is not sufficient
on some systems.
@markuskowa
Copy link
Member Author

It looks like some weird timing issue. The test runs just fine locally. Let's see if waiting for the TCP port to appear helps.

@markuskowa
Copy link
Member Author

@GrahamcOfBorg build nixosTests.slurm

@markuskowa
Copy link
Member Author

OK, that worked 🎊. The aarch64 failure is now due to a broken libjpeg-turbo.

@markuskowa
Copy link
Member Author

markuskowa commented Dec 10, 2018 •
edited

If there are no further comments, this PR would be ready to merge.

@markuskowa markuskowa merged commit 9fba490 into NixOS:master Dec 10, 2018
@markuskowa markuskowa deleted the fix-slurm-module branch December 10, 2018 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@infinisil infinisil Awaiting requested review from infinisil

Assignees
No one assigned
Labels
6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@markuskowa @GrahamcOfBorg @Mic92