udocker: init at 1.1.1 #34493

bzizou · 2018-02-01T12:41:55Z

Motivation for this change

udocker may be useful for HPC environments using NIX

This package works at image-creation-time (docker should be installed) and at runtime with at least P1 and P2 modes.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

dotlambda · 2018-02-01T12:53:50Z

pkgs/applications/virtualization/udocker/default.nix

@@ -0,0 +1,35 @@
+{ stdenv, fetchurl, proot, patchelf, fakechroot, runc, python, pythonPackages, coreutils }:
+
+stdenv.mkDerivation rec {


Please use buildPythonApplication

dotlambda · 2018-02-01T12:54:19Z

pkgs/applications/virtualization/udocker/default.nix

+stdenv.mkDerivation rec {
+
+  version = "1.1.1";
+  name = "udocker-${version}";


Please specify pname = "udocker"; and get rid of name.

dotlambda · 2018-02-01T12:54:57Z

pkgs/applications/virtualization/udocker/default.nix

+  version = "1.1.1";
+  name = "udocker-${version}";
+
+  src = fetchurl {


You could also use fetchFromGitHub.

dotlambda · 2018-02-01T12:55:40Z

pkgs/applications/virtualization/udocker/default.nix

+      substituteInPlace udocker.py --replace /usr/sbin:/sbin:/usr/bin:/bin $PATH
+      substituteInPlace udocker.py --replace /bin/chmod ${coreutils}/bin/chmod
+      substituteInPlace udocker.py --replace /bin/rm ${coreutils}/bin/chmod
+      substituteInPlace udocker.py --replace "autoinstall = True" "autoinstall = False"


This belongs into postPatch.

dotlambda · 2018-02-01T12:56:41Z

pkgs/top-level/all-packages.nix

@@ -17453,6 +17453,8 @@ with pkgs;
  testssl = callPackage ../applications/networking/testssl { };

  umurmur = callPackage ../applications/networking/umurmur { };
+
+  udocker = callPackage ../applications/virtualization/udocker { };


You can use pythonPackages.callPackage here

bzizou · 2018-02-01T15:37:50Z

@dotlambda : thank you for your advices! I think that I fixed everything...

A1ve5 · 2018-02-01T15:51:17Z

Hi @bzizou, thank you for this.
Just a minor remark to your comment above: Docker isn't necessary to run udocker.
Cheers

dotlambda · 2018-02-01T15:45:15Z

pkgs/applications/virtualization/udocker/default.nix

@@ -0,0 +1,34 @@
+{ stdenv, fetchFromGitHub, proot, patchelf, fakechroot, runc, python, pythonPackages, coreutils }:
+
+with pythonPackages;


Better put buildPythonApplication, simplejson, etc. in the argument set.

dotlambda · 2018-02-01T15:48:06Z

pkgs/applications/virtualization/udocker/default.nix

+    homepage = https://www.gitbook.com/book/indigo-dc/udocker;
+    license = stdenv.lib.licenses.asl20;
+    maintainers = [ stdenv.lib.maintainers.bzizou ];
+    platforms = stdenv.lib.platforms.linux;


Drop stdenv.lib. in the 3 lines above

dotlambda · 2018-02-01T15:51:11Z

pkgs/applications/virtualization/udocker/default.nix

+          repo = "udocker" ;
+          rev = "v${version}";
+          sha256 = "134xk7rfj0xki9znryk5qf1nsfa318ahrrsi1k6ia7kipp7i3hb4";
+        };


Indentation

bzizou · 2018-02-01T18:10:40Z

@A1ve5 : even at image creation time?

dotlambda · 2018-02-02T08:55:00Z

pkgs/top-level/all-packages.nix

@@ -17453,6 +17453,8 @@ with pkgs;
  testssl = callPackage ../applications/networking/testssl { };

  umurmur = callPackage ../applications/networking/umurmur { };
+
+  udocker = pythonPackages.callPackage ../applications/virtualization/udocker { };


I think this is better suited for tools/virtualization.

dotlambda · 2018-02-02T08:55:46Z

pkgs/applications/virtualization/udocker/default.nix

+  postPatch = ''
+      substituteInPlace udocker.py --replace /usr/sbin:/sbin:/usr/bin:/bin $PATH
+      substituteInPlace udocker.py --replace /bin/chmod ${coreutils}/bin/chmod
+      substituteInPlace udocker.py --replace /bin/rm ${coreutils}/bin/chmod


You probably don't want to replace rm by chmod.

dotlambda · 2018-02-02T09:01:02Z

@GrahamcOfBorg build udocker

GrahamcOfBorg · 2018-02-02T09:01:14Z

Success on x86_64-linux (full log)

Partial log (click to expand)

writing dependency_links to udocker.egg-info/dependency_links.txt
reading manifest file 'udocker.egg-info/SOURCES.txt'
writing manifest file 'udocker.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/b73vn0zk93bzsnwc8266qc4hb08ga96k-udocker-1.1.1

GrahamcOfBorg · 2018-02-02T09:01:35Z

Success on aarch64-linux (full log)

Partial log (click to expand)

writing dependency_links to udocker.egg-info/dependency_links.txt
reading manifest file 'udocker.egg-info/SOURCES.txt'
writing manifest file 'udocker.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/j7b2f4sbmfsydwryq6ywg1sax2ddrykm-udocker-1.1.1

dotlambda · 2018-02-02T09:07:13Z

Ran 0 tests in 0.000s

Please specify the appropriate checkPhase, e.g. python -m unittest discover --pattern='*_tests.py'.
If the tests don't work (e.g. because they need network access) specify doCheck = false and write a comment with the reason.

bzizou · 2018-02-02T11:20:06Z

@A1ve5 : checks are launched with tox right?

dotlambda · 2018-02-02T11:27:21Z

But according to https://github.com/indigo-dc/udocker/blob/master/tox.ini, tox should simply call python -m unittest discover --pattern='*_tests.py' I believe.
I think it is generally not advised to use tox inside Nixpkgs, but @FRidh can certainly clarify this.

A1ve5 · 2018-02-02T11:43:48Z

@dotlambda our jenkins is running: nosetests -v tests/unit_tests.py --with-xcoverage --cover-package=udocker

A1ve5 · 2018-02-02T11:47:43Z

@bzizou, yes, to build the images we rely on external tools like Docker. However, when one run "udocker create xyz" the image should be already in local repo and Docker isn't necessary. Not sure if that was your question.

bzizou · 2018-02-02T12:58:46Z

Tests are ok with NOSE_EXCLUDE=test_03_create_repo,test_04_is_repo,test_02__get_group_from_host nosetests -v tests/unit_tests.py
So I'm going to push an amended commit with this line as checkPhase.

Note that the tests are passing with just nosetests -v tests/unit_tests.py inside a nix-shell --pure but I guess that the 3 failing tests are trying to write into the store...

FRidh · 2018-02-03T08:03:38Z

@GrahamcOfBorg build udocker

GrahamcOfBorg · 2018-02-03T08:04:11Z

Success on x86_64-linux (full log)

Partial log (click to expand)

Test Unique.filename(). ... ok
Test _check_output(). ... ok
Test check_output(). ... ok
Test get_output(). ... ok
Test get_output(). ... ok

----------------------------------------------------------------------
Ran 308 tests in 1.134s

OK

GrahamcOfBorg · 2018-02-03T08:07:17Z

Success on aarch64-linux (full log)

Partial log (click to expand)

Test _check_output(). ... ok
Test check_output(). ... ok
Test get_output(). ... ok
Test get_output(). ... ok

----------------------------------------------------------------------
Ran 308 tests in 4.100s

OK
/nix/store/imrmhan3xnqvfdyw7sxw8lpaaciahmsm-udocker-1.1.1

GrahamcOfBorg · 2018-02-03T11:15:37Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

Package ‘udocker-1.1.1’ in /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/tools/virtualization/udocker/default.nix:30 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Feb 1, 2018

dotlambda reviewed Feb 1, 2018

View reviewed changes

bzizou force-pushed the udocker branch 2 times, most recently from be6fcf0 to 1e91fef Compare February 1, 2018 15:36

dotlambda reviewed Feb 1, 2018

View reviewed changes

bzizou force-pushed the udocker branch from 1e91fef to 5ee53e3 Compare February 1, 2018 18:07

dotlambda reviewed Feb 2, 2018

View reviewed changes

udocker: init at 1.1.1

b2cf8cf

bzizou force-pushed the udocker branch from 5ee53e3 to b2cf8cf Compare February 2, 2018 13:00

FRidh merged commit 274d4ea into NixOS:master Feb 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

udocker: init at 1.1.1 #34493

udocker: init at 1.1.1 #34493

bzizou commented Feb 1, 2018

dotlambda Feb 1, 2018

dotlambda Feb 1, 2018

dotlambda Feb 1, 2018

dotlambda Feb 1, 2018

dotlambda Feb 1, 2018

bzizou commented Feb 1, 2018

A1ve5 commented Feb 1, 2018

dotlambda Feb 1, 2018

dotlambda Feb 1, 2018

dotlambda Feb 1, 2018

bzizou commented Feb 1, 2018

dotlambda Feb 2, 2018

dotlambda Feb 2, 2018

dotlambda commented Feb 2, 2018

GrahamcOfBorg commented Feb 2, 2018

GrahamcOfBorg commented Feb 2, 2018

dotlambda commented Feb 2, 2018 •

edited

bzizou commented Feb 2, 2018

dotlambda commented Feb 2, 2018

A1ve5 commented Feb 2, 2018

A1ve5 commented Feb 2, 2018

bzizou commented Feb 2, 2018

FRidh commented Feb 3, 2018

GrahamcOfBorg commented Feb 3, 2018

GrahamcOfBorg commented Feb 3, 2018

GrahamcOfBorg commented Feb 3, 2018

		@@ -0,0 +1,35 @@
		{ stdenv, fetchurl, proot, patchelf, fakechroot, runc, python, pythonPackages, coreutils }:

		stdenv.mkDerivation rec {

		@@ -0,0 +1,34 @@
		{ stdenv, fetchFromGitHub, proot, patchelf, fakechroot, runc, python, pythonPackages, coreutils }:

		with pythonPackages;

udocker: init at 1.1.1 #34493

udocker: init at 1.1.1 #34493

Conversation

bzizou commented Feb 1, 2018

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bzizou commented Feb 1, 2018

A1ve5 commented Feb 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bzizou commented Feb 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotlambda commented Feb 2, 2018

GrahamcOfBorg commented Feb 2, 2018

GrahamcOfBorg commented Feb 2, 2018

dotlambda commented Feb 2, 2018 • edited

bzizou commented Feb 2, 2018

dotlambda commented Feb 2, 2018

A1ve5 commented Feb 2, 2018

A1ve5 commented Feb 2, 2018

bzizou commented Feb 2, 2018

FRidh commented Feb 3, 2018

GrahamcOfBorg commented Feb 3, 2018

GrahamcOfBorg commented Feb 3, 2018

GrahamcOfBorg commented Feb 3, 2018

dotlambda commented Feb 2, 2018 •

edited