infiniband-diags: init at 2.0.0 #36152

aij · 2018-03-01T03:37:11Z

Motivation for this change

Tools for InfiniBand

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
N/A Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
N/A Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Executables tested (some quite minimally):

I'm only just learning about these tools though, so testing is minimal at best...

@markuskowa I saw you packaged rdma-core. Do you happen to have IB hardware to test on?

cc @Mic92

markuskowa · 2018-03-01T04:12:24Z

Unfortunately I do not have access to IB hardware at the moment.

Mic92 · 2018-03-01T08:32:55Z

@GrahamcOfBorg build infiniband-diags

GrahamcOfBorg · 2018-03-01T08:33:11Z

Failure on aarch64-linux (full log)

Partial log (click to expand)

Package ‘infiniband-diags-2.0.0’ in /var/lib/gc-of-borg/nix-test-rs-12/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-12/pkgs/tools/networking/infiniband-diags/default.nix:26 is not supported on ‘aarch64-linux’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-03-01T08:33:45Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

Package ‘infiniband-diags-2.0.0’ in /Users/graham/nix-borg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/tools/networking/infiniband-diags/default.nix:26 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-03-01T08:33:52Z

Success on x86_64-linux (full log)

Partial log (click to expand)

patching script interpreter paths in /nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0/bin/dump_lfts.sh: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0/bin/ibrouters: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0/bin/ibnodes: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0/bin/ibstatus: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0/bin/ibhosts: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0/bin/dump_mfts.sh: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0/bin/ibswitches: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
checking for references to /tmp/nix-build-infiniband-diags-2.0.0.drv-0 in /nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0...
/nix/store/d3sk3qsazxhp28bl1qia3m2r71sqbhsn-infiniband-diags-2.0.0

Mic92 · 2018-03-01T08:35:38Z

check_lft_balance.pl wants perl in buildInputs at least.

aij · 2018-03-01T12:02:00Z

After generating an lft-output and ibnetdiscover-cache, check_lft_balance.pl -l ~/lft-output -i ~/ibnetdiscover-cache now gets as far as

iblinkinfo: option requires an argument -- 'S'

apparently from

    $iblinkinfo_output = `iblinkinfo --load-cache $ibnetdiscover_cache -S $switch_guid`;

So, I may need to get an IB switch before I can test that properly... fortunately, they're not too expensive. 😄

markuskowa · 2018-03-01T19:29:48Z

There is ibsim, which can be partially used to simulate an IB fabric (https://github.com/markuskowa/nixpkgs/blob/ofed/pkgs/os-specific/linux/ibsim/default.nix).
That might help here. I started looking into IB integration but did not have time to work on it recently.
Having a NixOS test, which can test the IB utilities and fabric setup in a VM would be awesome (nixos/tests/rxe.nix is a small step in that direction).

markuskowa · 2018-03-01T04:18:30Z

pkgs/tools/networking/infiniband-diags/default.nix

+    export CFLAGS="-I${opensm}/include/infiniband"
+    ./autogen.sh
+  '';
+


I would recommend adding the following code. When I tried to package it the perl based executables failed to start
without setting PERL5LIB.

postInstall = '' rm -r $out/var ''; postFixup = '' for pls in `find $out/bin -name "*.pl"`; do echo "wrapping $pls" wrapProgram $pls --prefix PERL5LIB : "$out/lib/perl5/site_perl" done '';

Thanks. FWIW, the approach in perl-modules seems a little nicer, but it should ideally be factored out into a separate function like wrapProgram or patchShebangs. (Though I'm not sure how often it would be useful outside of actual perl packages...)

markuskowa · 2018-03-01T08:46:49Z

pkgs/tools/networking/infiniband-diags/default.nix

+    sha256 = "06x8yy3ly1vzraznc9r8pfsal9mjavxzhgrla3q2493j5jz0sx76";
+  };
+
+  nativeBuildInputs = [ autoconf automake libtool pkgconfig ];


Should be perl in buildInputs not enough?

Sorry, I missed it. buildInputs is the right place.

aij · 2018-03-02T03:57:25Z

I think everything is working at least to some extent... Some tools provide me useful information, some output nothing but exit(0), and some I only tested far enough to see they print a help message.

10000 packets transmitted, 10000 received, 0% packet loss, time 1217 ms
rtt min/avg/max = 0.040/0.121/0.319 ms

Doesn't seem as low-latency as I was expecting though... 🤷‍♂️

Mic92 · 2018-03-02T09:01:03Z

@GrahamcOfBorg build infiniband-diags

GrahamcOfBorg · 2018-03-02T09:01:30Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

Package ‘infiniband-diags-2.0.0’ in /Users/graham/nix-borg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/tools/networking/infiniband-diags/default.nix:38 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-03-02T09:01:57Z

Success on x86_64-linux (full log)

Partial log (click to expand)

/nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibhosts: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibfindnodesusing.pl: interpreter directive changed from "/usr/bin/perl" to "/nix/store/s029gmjpgvm776wfb56naqny9qja9339-perl-5.24.3/bin/perl"
/nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibrouters: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibidsverify.pl: interpreter directive changed from "/usr/bin/perl" to "/nix/store/s029gmjpgvm776wfb56naqny9qja9339-perl-5.24.3/bin/perl"
/nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibstatus: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
/nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibswitches: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
checking for references to /tmp/nix-build-infiniband-diags-2.0.0.drv-0 in /nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0...
wrapping /nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibfindnodesusing.pl
wrapping /nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0/bin/ibidsverify.pl
/nix/store/9sa5ps2z3x1jjkc2b241k223apgg93xh-infiniband-diags-2.0.0

GrahamcOfBorg · 2018-03-02T09:05:13Z

Failure on aarch64-linux (full log)

Partial log (click to expand)

Package ‘infiniband-diags-2.0.0’ in /var/lib/gc-of-borg/nix-test-rs-31/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-31/pkgs/tools/networking/infiniband-diags/default.nix:38 is not supported on ‘aarch64-linux’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Mar 1, 2018

aij force-pushed the infiniband-diags branch from 1bfe52c to 8fab741 Compare March 1, 2018 11:27

markuskowa reviewed Mar 1, 2018

View reviewed changes

infiniband-diags: init at 2.0.0

6dfe151

aij force-pushed the infiniband-diags branch from 8fab741 to 6dfe151 Compare March 2, 2018 02:24

Mic92 merged commit 609f0e0 into NixOS:master Mar 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infiniband-diags: init at 2.0.0 #36152

infiniband-diags: init at 2.0.0 #36152

aij commented Mar 1, 2018 •

edited

markuskowa commented Mar 1, 2018 •

edited

Mic92 commented Mar 1, 2018

GrahamcOfBorg commented Mar 1, 2018

GrahamcOfBorg commented Mar 1, 2018

GrahamcOfBorg commented Mar 1, 2018

Mic92 commented Mar 1, 2018

aij commented Mar 1, 2018

markuskowa commented Mar 1, 2018

markuskowa Mar 1, 2018

aij Mar 2, 2018

markuskowa Mar 1, 2018

Mic92 Mar 1, 2018 •

edited

markuskowa Mar 1, 2018

aij commented Mar 2, 2018

Mic92 commented Mar 2, 2018

GrahamcOfBorg commented Mar 2, 2018

GrahamcOfBorg commented Mar 2, 2018

GrahamcOfBorg commented Mar 2, 2018

infiniband-diags: init at 2.0.0 #36152

infiniband-diags: init at 2.0.0 #36152

Conversation

aij commented Mar 1, 2018 • edited

Motivation for this change

Things done

markuskowa commented Mar 1, 2018 • edited

Mic92 commented Mar 1, 2018

GrahamcOfBorg commented Mar 1, 2018

GrahamcOfBorg commented Mar 1, 2018

GrahamcOfBorg commented Mar 1, 2018

Mic92 commented Mar 1, 2018

aij commented Mar 1, 2018

markuskowa commented Mar 1, 2018

markuskowa Mar 1, 2018

Choose a reason for hiding this comment

aij Mar 2, 2018

Choose a reason for hiding this comment

markuskowa Mar 1, 2018

Choose a reason for hiding this comment

Mic92 Mar 1, 2018 • edited

Choose a reason for hiding this comment

markuskowa Mar 1, 2018

Choose a reason for hiding this comment

aij commented Mar 2, 2018

Mic92 commented Mar 2, 2018

GrahamcOfBorg commented Mar 2, 2018

GrahamcOfBorg commented Mar 2, 2018

GrahamcOfBorg commented Mar 2, 2018

aij commented Mar 1, 2018 •

edited

markuskowa commented Mar 1, 2018 •

edited

Mic92 Mar 1, 2018 •

edited