singularity: add checkPhase #42374

bbarker · 2018-06-22T13:59:31Z

added correct checkPhase, but fails to run correctly in nixpkgs

Motivation for this change

Currently singularity checks aren't run when doCheck=true is specified since we need to run make test, and because some aspects of singularity require many system tools, SUID, etc, it will be good to test.

Things done

Currently, there is an issue; when checkPhase is run, it appears to be trying to look into a directory in the /nix/store that may not exist yet. Can I have confirmation on if this is the likely problem, and suggestions for possible workarounds? It may require fixes on the Singularity code base, but i'll try to get them in.

nix-build -I nixpkgs=/home/brandon/workspace/nixpkgs -E '(import <nixpkgs>{}).singularity.overrideAttrs (oldAttrs: { doCheck=true; })' --pure

eventually resulted in:

 -Wl,-rpath -Wl,/nix/store/fjwqxfxqjnva7880vwisd3780vnaaz5n-singularity-2.5.1/lib -Wl,-z -Wl,relro -Wl,-z -Wl,now -o .libs/start-suid 
start_suid-start.o util/start_suid-util.o util/start_suid-file.o util/start_suid-registry.o util/start_suid-privilege.o util/start_sui
d-sessiondir.o util/start_suid-suid.o util/start_suid-cleanupd.o util/start_suid-fork.o util/start_suid-daemon.o util/start_suid-signa
l.o util/start_suid-mount.o  lib/image/.libs/libsingularity-image.so lib/runtime/.libs/libsingularity-runtime.so action-lib/.libs/libi
nternal.a -Wl,-rpath -Wl,/nix/store/fjwqxfxqjnva7880vwisd3780vnaaz5n-singularity-2.5.1/lib/singularity                               
libtool: link: gcc -Wall -fpie -fPIC -g -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -fstack-protector --param ssp-buffer-size=4
 -Wl,-rpath -Wl,/nix/store/fjwqxfxqjnva7880vwisd3780vnaaz5n-singularity-2.5.1/lib -Wl,-z -Wl,relro -Wl,-z -Wl,now -o .libs/mount-suid 
mount_suid-mount.o util/mount_suid-util.o util/mount_suid-file.o util/mount_suid-registry.o util/mount_suid-suid.o util/mount_suid-pri
vilege.o util/mount_suid-mount.o  lib/image/.libs/libsingularity-image.so lib/runtime/.libs/libsingularity-runtime.so -Wl,-rpath -Wl,/
nix/store/fjwqxfxqjnva7880vwisd3780vnaaz5n-singularity-2.5.1/lib/singularity                                                         
make[3]: Leaving directory '/tmp/nix-build-singularity-2.5.1.drv-0/source/src'
make[2]: Leaving directory '/tmp/nix-build-singularity-2.5.1.drv-0/source/src'
make[1]: Leaving directory '/tmp/nix-build-singularity-2.5.1.drv-0/source/src'
make[1]: Entering directory '/tmp/nix-build-singularity-2.5.1.drv-0/source'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/tmp/nix-build-singularity-2.5.1.drv-0/source'
running tests
sh ./test.sh
Building/Installing Singularity to temporary directory
Reinvoking in a clean shell
ERROR: Could not locate singularity program at: /nix/store/fjwqxfxqjnva7880vwisd3780vnaaz5n-singularity-2.5.1/bin/singularity
make: *** [Makefile:832: test] Error 1
builder for '/nix/store/wy87hm0vrfzh20gv0ai5pz24n5alxqxv-singularity-2.5.1.drv' failed with exit code 2
error: build of '/nix/store/wy87hm0vrfzh20gv0ai5pz24n5alxqxv-singularity-2.5.1.drv' failed

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

jtojnar · 2018-06-22T14:55:34Z

You can use checkTarget = "test". Also since it looks like it requires the files to be installed, checkPhase will not work since it runs before installPhase. You can use doInstallCheck = true, though. Then you will have to use installCheckTarget = "test".

added correct checkPhase, but fails to run correctly in nixpkgs

bbarker · 2018-06-22T21:15:58Z

@jtojnar excellent, but I am getting some somewhat expected problems do to some suid issues:

running install tests
installcheck flags: -j8 -l8 SHELL=/nix/store/0aa1ngxygf1iqanfbpdb10klc7r343c0-bash-4.4-p19/bin/bash test
sh ./test.sh
Building/Installing Singularity to temporary directory
Reinvoking in a clean shell

################################################################################
Checking environment (script: 01-sanity.sh, testdir: /tmp/stest.kh2LCh)

 + sudo true                                                                                     (retval=1) ERROR
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?
Full output in: /tmp/stest.kh2LCh
ERROR: Failed running test: 01-sanity.sh
make: *** [Makefile:832: test] Error 1
builder for '/nix/store/bgfgdji4hz5fp22p0pwp829hrmibd1hv-singularity-2.5.1.drv' failed with exit code 2

And output of the tmp directory:

brandon@beb82dell0-DevContainer:/tmp/stest.kh2LCh
$ grep "" *    
cmd:sudo true
output:sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?
pwd:/tmp/nix-build-singularity-2.5.1.drv-0/source/tests
brandon@beb82dell0-DevContainer:/tmp/stest.kh2LCh

But, I'm running this in a slightly odd environment (a container that uses sudo from the base OS image), so maybe it works for others on NixOS...

bbarkerro · 2018-07-02T19:38:20Z

@jtojnar I'm at a bit of a loss still - can you easily confirm if the issue exists on NixOS, or can we ask @GrahamcOfBorg to do a build with { doInstallCheck=true; })?

jtojnar · 2018-07-02T20:36:24Z

I do not think builders should ever run sudo. Not even sure what would the expected behaviour be.

bbarkerro · 2018-07-02T21:10:03Z

How does one run tests that require the suid bit set, then? I wonder if there is a precedent.

…

On Mon, Jul 2, 2018, 4:36 PM Jan Tojnar ***@***.***> wrote: I do not think builders should ever run sudo. Not even sure what would the expected behaviour be. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#42374 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AgI-Cwt_yUIUGQytSGNhv7nW2MTS6jB9ks5uCoRMgaJpZM4Uz17H> .

jtojnar · 2018-07-03T06:57:27Z

You should be able to do so in NixOS tests, see nixos/tests directory. And the security.wrappers option.

matthewbauer · 2018-07-17T23:52:41Z

@GrahamcOfBorg build singularity

GrahamcOfBorg · 2018-07-17T23:52:55Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: singularity

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-17T23:53:31Z

Success on x86_64-linux (full log)

Attempted: singularity

Partial log (click to expand)

/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/handlers/archive-cpio.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/handlers/image-http.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/handlers/archive-tar.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/helpers/image.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/helpers/apps/list.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/helpers/help.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/helpers/check.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/libexec/singularity/helpers/inspect.sh: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/bin/run-singularity: interpreter directive changed from "/bin/sh" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/sh"
/nix/store/aadxa9vy592ysmdx3h846019xgkn4znj-singularity-2.5.2/bin/singularity: interpreter directive changed from "/bin/bash" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/bash"

GrahamcOfBorg · 2018-07-17T23:56:26Z

Success on aarch64-linux (full log)

Attempted: singularity

Partial log (click to expand)

/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/libexec/singularity/bootstrap-scripts/deffile-driver-localimage.sh: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/libexec/singularity/bootstrap-scripts/deffile-driver-docker.sh: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/libexec/singularity/bootstrap-scripts/deffile-driver-debootstrap.sh: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/libexec/singularity/bootstrap-scripts/deffile-driver-busybox.sh: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/libexec/singularity/bootstrap-scripts/deffile-driver-arch.sh: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/libexec/singularity/bootstrap-scripts/checks.sh: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/libexec/singularity/bootstrap-scripts/functions: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/bin/run-singularity: interpreter directive changed from "/bin/sh" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/sh"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2/bin/singularity: interpreter directive changed from "/bin/bash" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/bash"
/nix/store/c8vw4nz59xncyvrriap8k7i1accy7sys-singularity-2.5.2

veprbl · 2018-11-28T10:10:41Z

@GrahamcOfBorg build singularity

GrahamcOfBorg · 2018-11-28T10:14:51Z

Success on aarch64-linux (full log)

Attempted: singularity

Partial log (click to expand)

/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/libexec/singularity/bootstrap-scripts/deffile-driver-localimage.sh: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/libexec/singularity/bootstrap-scripts/deffile-driver-docker.sh: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/libexec/singularity/bootstrap-scripts/deffile-driver-debootstrap.sh: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/libexec/singularity/bootstrap-scripts/deffile-driver-busybox.sh: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/libexec/singularity/bootstrap-scripts/deffile-driver-arch.sh: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/libexec/singularity/bootstrap-scripts/checks.sh: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/libexec/singularity/bootstrap-scripts/functions: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/bin/run-singularity: interpreter directive changed from "/bin/sh" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/sh"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0/bin/singularity: interpreter directive changed from "/bin/bash" to "/nix/store/bpspdmsl0yys24gs70flsvcw3wcnl7lx-bash-4.4-p23/bin/bash"
/nix/store/gjqxp8h1r33mkb83hfwx0yw6ls0h5043-singularity-2.6.0

Mic92 · 2018-11-28T10:17:00Z

Our build sandbox does not allow to create setuid binaries.

GrahamcOfBorg · 2018-11-28T10:41:26Z

Success on x86_64-linux (full log)

Attempted: singularity

Partial log (click to expand)

/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/handlers/archive-cpio.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/handlers/image-http.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/handlers/archive-tar.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/helpers/image.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/helpers/apps/list.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/helpers/help.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/helpers/check.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/libexec/singularity/helpers/inspect.sh: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/bin/run-singularity: interpreter directive changed from "/bin/sh" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/sh"
/nix/store/6h4x7nx6jm98nfadzbzbr9ab7b9kgqx4-singularity-2.6.0/bin/singularity: interpreter directive changed from "/bin/bash" to "/nix/store/yal8k8rgbaj8p9vi08q1kh21ls0ndq95-bash-4.4-p23/bin/bash"

GrahamcOfBorg · 2018-11-28T11:09:22Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: singularity

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

veprbl · 2019-05-17T16:52:43Z

Approach with checkPhase seems to not work, this might need to become a nixos test instead. We also have singularity updated to 3.x branch by now. Closing this unless anyone has objections.

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Jun 22, 2018

singularity: add checkPhase

eb148c5

added correct checkPhase, but fails to run correctly in nixpkgs

bbarker force-pushed the singularity-checks branch from 9d56526 to eb148c5 Compare June 22, 2018 21:08

veprbl added the 2.status: work-in-progress label Nov 28, 2018

veprbl added the 2.status: merge conflict label Feb 18, 2019

veprbl closed this May 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

singularity: add checkPhase #42374

singularity: add checkPhase #42374

bbarker commented Jun 22, 2018

jtojnar commented Jun 22, 2018

bbarker commented Jun 22, 2018

bbarkerro commented Jul 2, 2018

jtojnar commented Jul 2, 2018

bbarkerro commented Jul 2, 2018 via email

jtojnar commented Jul 3, 2018 •

edited

matthewbauer commented Jul 17, 2018

GrahamcOfBorg commented Jul 17, 2018

GrahamcOfBorg commented Jul 17, 2018

GrahamcOfBorg commented Jul 17, 2018

veprbl commented Nov 28, 2018

GrahamcOfBorg commented Nov 28, 2018

Mic92 commented Nov 28, 2018

GrahamcOfBorg commented Nov 28, 2018

GrahamcOfBorg commented Nov 28, 2018

veprbl commented May 17, 2019

singularity: add checkPhase #42374

singularity: add checkPhase #42374

Conversation

bbarker commented Jun 22, 2018

Motivation for this change

Things done

jtojnar commented Jun 22, 2018

bbarker commented Jun 22, 2018

bbarkerro commented Jul 2, 2018

jtojnar commented Jul 2, 2018

bbarkerro commented Jul 2, 2018 via email

jtojnar commented Jul 3, 2018 • edited

matthewbauer commented Jul 17, 2018

GrahamcOfBorg commented Jul 17, 2018

GrahamcOfBorg commented Jul 17, 2018

GrahamcOfBorg commented Jul 17, 2018

veprbl commented Nov 28, 2018

GrahamcOfBorg commented Nov 28, 2018

Mic92 commented Nov 28, 2018

GrahamcOfBorg commented Nov 28, 2018

GrahamcOfBorg commented Nov 28, 2018

veprbl commented May 17, 2019

jtojnar commented Jul 3, 2018 •

edited