Update simgrid 3.19.1 -> 3.20 + parallel tests + enable darwin #42721

mickours · 2018-06-28T09:09:10Z

Motivation for this change

Enable the build on Darwin.
Speed up testing

Things done

I use this update to remove elfutils from the nativeBuildInputs because it was only needed for Model Checking. This should enables the build on Darwin without this option which is activated by default.

Also I add a falg to Ctest command to enable parallel run of the tests

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/simgrid/versions. These checks were done: - built on NixOS - /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/smpicc passed the binary check. - /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/smpicxx passed the binary check. - /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/smpirun passed the binary check. - Warning: no invocation of /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/tesh had a zero exit code or showed the expected version - /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/simgrid-colorizer passed the binary check. - Warning: no invocation of /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/simgrid_update_xml had a zero exit code or showed the expected version - /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/simgrid_convert_TI_traces passed the binary check. - Warning: no invocation of /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/smpimain had a zero exit code or showed the expected version - /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20/bin/graphicator passed the binary check. - 6 of 9 passed binary check by having a zero exit code. - 4 of 9 passed binary check by having the new version present in output. - found 3.20 with grep in /nix/store/cvyi6hvgc9rvgrnp7c028xrlzbl7jzb9-simgrid-3.20 - directory tree listing: https://gist.github.com/edefc2f1a2e81412484edc2e45986e03 - du listing: https://gist.github.com/a44f7d57537e93152a8e6c569f8ed1ae

- Fix dependencies to enable build on darwin - Add ctest flag to enable parallel testing

LnL7 · 2018-06-28T09:22:40Z

@GrahamcOfBorg build simgrid

LnL7 · 2018-06-28T09:20:56Z

pkgs/applications/science/misc/simgrid/default.nix

+        NB_CORES="$NIX_BUILD_CORES"
+      fi
+    fi
+    ctest -j $NB_CORES --output-on-failure -E smpi-replay-multiple


Doesn't -j0 work here, like with make?

I'm not sure, nothing is specified for "0" value in the ctest documentation (https://cmake.org/cmake/help/v3.0/manual/ctest.1.html), so I did it this way.

GrahamcOfBorg · 2018-06-28T09:24:45Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: simgrid

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-06-28T09:27:27Z

Success on x86_64-linux (full log)

Attempted: simgrid

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20
shrinking /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20/bin/graphicator
shrinking /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20/bin/smpimain
shrinking /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20/lib/libsimgrid.so.3.20
gzipping man pages under /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20/share/man/
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20/lib  /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20/bin
/nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20
patching script interpreter paths in /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20
checking for references to /build in /nix/store/ibvq958giaw8f5wfmvr03vdf7yvv7ax0-simgrid-3.20...

GrahamcOfBorg · 2018-06-28T09:34:28Z

Failure on x86_64-darwin (full log)

Attempted: simgrid

Partial log (click to expand)


Total Test time (real) = 100.29 sec

The following tests FAILED:
        519 - tesh-smpi-privatization-mmap-thread (Failed)
        520 - tesh-smpi-privatization-mmap-raw (Failed)
        521 - tesh-smpi-privatization-mmap-boost (Failed)
Errors while running CTest
builder for '/nix/store/ikc06nzamv2m4q0klrgg439ar00y2xvi-simgrid-3.20.drv' failed with exit code 8
�[31;1merror:�[0m build of '/nix/store/ikc06nzamv2m4q0klrgg439ar00y2xvi-simgrid-3.20.drv' failed

mickours · 2018-06-28T11:24:55Z

Darwin build has failed, but it seems from the CI logs that the osx version used by the CI is xnu-osx-10.11.6.

With our internal CI that runs on 10.13 the tests are all green.

matthewbauer

Avoid setting custom NB_CORES values. NIX_BUILD_CORES should just work everywhere (and if it doesn't we should handle that in CMake not here).

mickours · 2018-08-02T08:23:22Z

Thanks @matthewbauer for the review.

I agree that it seems odd to set this here, but I'm trying to parallelize the tests (and not the build) withctest, which is not used by default by the checkPhase even if cmake is setup.

So, because the use of ctest is already a custom behavior, I don't see any generic way to do that, and even if we can do it, I don't think that testing in parallel is really wanted for most of the project.

Also, it should not be switch on by the enableParallelBuilding variable, but by something like enableParallelChecking, but still I don't see where we can set this variable.

matthewbauer · 2018-08-02T18:43:47Z

Yeah I think the ctest stuff is good just that it shouldn't need the if statement. Just like this is what we do in other places:

     ctest -j $NIX_BUILD_CORES --output-on-failure -E smpi-replay-multiple

mickours · 2018-08-03T13:17:40Z

I desagree because ctest -j 0 is not understand by cmake so I need to compute the number of cores to get the intended behavior that is documented in the manual (https://nixos.org/nix/manual/#sec-common-options)

The value 0 means that the builder should use all available CPU cores in the system.

Also, If the enableParallelBuilding is disable I want it to be disabled for the tests too, which is not the case with the simple one-liner.

By the way, this way of doing things is not coming from me, I get this implementation from here:

nixpkgs/pkgs/build-support/vm/default.nix

Lines 218 to 222 in caccc40

    
               if [ "$enableParallelBuilding" = 1 ]; then 
        
                 if [ ''${NIX_BUILD_CORES:-0} = 0 ]; then 
        
                   QEMU_OPTS+=" -smp cpus=$(nproc)" 
        
                 else 
        
                   QEMU_OPTS+=" -smp cpus=$NIX_BUILD_CORES"

matthewbauer · 2018-08-03T16:30:24Z

We handle this logic already on setup.sh:

nixpkgs/pkgs/stdenv/generic/setup.sh

Lines 624 to 631 in e46e7f4

    
           elif [ "$NIX_BUILD_CORES" -le 0 ]; then 
        
             NIX_BUILD_CORES=$(nproc 2>/dev/null || true) 
        
             if expr >/dev/null 2>&1 "$NIX_BUILD_CORES" : "^[0-9][0-9]*$"; then 
        
               : 
        
             else 
        
               NIX_BUILD_CORES="1" 
        
             fi 
        
           fi

because it is already done in nixpkgs/pkgs/stdenv/generic/setup.sh

mickours · 2018-08-06T09:43:08Z

Thanks for the explanation, I've simplify the ctest command as you suggests.

And sorry for being so insistent, I'm sometimes a bit stubborn when I don't understand ;)

mpickering · 2018-08-14T19:01:38Z

Seems like your concerns were dealt with @matthewbauer ?

matthewbauer · 2018-08-16T00:54:22Z

@GrahamcOfBorg build simgrid

GrahamcOfBorg · 2018-08-16T00:55:16Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: simgrid

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-08-16T00:58:39Z

Success on x86_64-linux (full log)

Attempted: simgrid

Partial log (click to expand)

moving /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/doc to /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/share/doc
shrinking RPATHs of ELF executables and libraries in /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20
shrinking /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/bin/smpimain
shrinking /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/bin/graphicator
shrinking /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/lib/libsimgrid.so.3.20
gzipping man pages under /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/share/man/
strip is /nix/store/gpc2wld1s0c6qzx9326cwn1wcx29xzsj-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/lib  /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20/bin
patching script interpreter paths in /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20
checking for references to /build in /nix/store/6sqkxqi4x0wzac78yz38va1lpckwl8w4-simgrid-3.20...

GrahamcOfBorg · 2018-08-16T01:04:41Z

Failure on x86_64-darwin (full log)

Attempted: simgrid

Partial log (click to expand)


Total Test time (real) =  82.04 sec

The following tests FAILED:
        519 - tesh-smpi-privatization-mmap-thread (Failed)
        520 - tesh-smpi-privatization-mmap-raw (Failed)
        521 - tesh-smpi-privatization-mmap-boost (Failed)
Errors while running CTest
builder for '/nix/store/85zqmg2fxgpi3p9ynmp212h2g2scly34-simgrid-3.20.drv' failed with exit code 8
error: build of '/nix/store/85zqmg2fxgpi3p9ynmp212h2g2scly34-simgrid-3.20.drv' failed

R. RyanTM and others added 2 commits June 27, 2018 15:30

simgrid: 3.19.1 -> 3.20 + add darwin

4dd858d

- Fix dependencies to enable build on darwin - Add ctest flag to enable parallel testing

mickours mentioned this pull request Jun 28, 2018

simgrid: 3.19.1 -> 3.20 #42680

Closed

GrahamcOfBorg added the 6.topic: darwin Running or building packages on Darwin label Jun 28, 2018

LnL7 reviewed Jun 28, 2018

View reviewed changes

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jun 28, 2018

matthewbauer approved these changes Jul 25, 2018

View reviewed changes

matthewbauer requested changes Aug 2, 2018

View reviewed changes

Use simpler parallel testing logic

182c1bc

because it is already done in nixpkgs/pkgs/stdenv/generic/setup.sh

matthewbauer approved these changes Aug 14, 2018

View reviewed changes

matthewbauer merged commit b856696 into NixOS:master Aug 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update simgrid 3.19.1 -> 3.20 + parallel tests + enable darwin #42721

Update simgrid 3.19.1 -> 3.20 + parallel tests + enable darwin #42721

mickours commented Jun 28, 2018

LnL7 commented Jun 28, 2018

LnL7 Jun 28, 2018

mickours Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

mickours commented Jun 28, 2018

matthewbauer left a comment

mickours commented Aug 2, 2018

matthewbauer commented Aug 2, 2018

mickours commented Aug 3, 2018 •

edited

matthewbauer commented Aug 3, 2018 •

edited

mickours commented Aug 6, 2018

mpickering commented Aug 14, 2018

matthewbauer commented Aug 16, 2018

GrahamcOfBorg commented Aug 16, 2018

GrahamcOfBorg commented Aug 16, 2018

GrahamcOfBorg commented Aug 16, 2018

Update simgrid 3.19.1 -> 3.20 + parallel tests + enable darwin #42721

Update simgrid 3.19.1 -> 3.20 + parallel tests + enable darwin #42721

Conversation

mickours commented Jun 28, 2018

Motivation for this change

Things done

LnL7 commented Jun 28, 2018

LnL7 Jun 28, 2018

Choose a reason for hiding this comment

mickours Jun 28, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

mickours commented Jun 28, 2018

matthewbauer left a comment

Choose a reason for hiding this comment

mickours commented Aug 2, 2018

matthewbauer commented Aug 2, 2018

mickours commented Aug 3, 2018 • edited

matthewbauer commented Aug 3, 2018 • edited

mickours commented Aug 6, 2018

mpickering commented Aug 14, 2018

matthewbauer commented Aug 16, 2018

GrahamcOfBorg commented Aug 16, 2018

GrahamcOfBorg commented Aug 16, 2018

GrahamcOfBorg commented Aug 16, 2018

mickours commented Aug 3, 2018 •

edited

matthewbauer commented Aug 3, 2018 •

edited