petsc: 3.8.4 -> 3.10.3 #55358

jamtrott · 2019-02-06T19:26:53Z

These changes upgrade PETSc to the most recent release and add several useful linear solver backends, including HYPRE, MUMPS, scalapack, and SPAI. In addition, the Python wrapper package petsc4py is built.

This has been tested on Mac OS X 10.13.6 (High Sierra).

Also, build libesmumps and libptesmumps to later support the mumps package.

srhb

Evaluation currently fails due to some misses in the meta sections. Can you fix these up? :)

pkgs/development/libraries/science/math/hypre/default.nix

pkgs/development/libraries/science/math/mumps/default.nix

pkgs/development/libraries/science/math/petsc/default.nix

pkgs/development/libraries/science/math/spai/default.nix

pkgs/development/python-modules/petsc4py/default.nix

jamtrott · 2019-02-08T07:38:02Z

Evaluation currently fails due to some misses in the meta sections. Can you fix these up? :)

Hi Sarah. Thank you for reviewing!

I fixed those mistakes and have pushed some new commits. I'm not sure whether it was necessary to create a commit for each separate package, but they can always be squashed before merging anyway.

srhb · 2019-02-08T11:55:06Z

Ideally I'd like this to be squashed down to one commit per package, would you mind doing that? :)

jamtrott · 2019-02-08T12:02:39Z

Ideally I'd like this to be squashed down to one commit per package, would you mind doing that? :)

No problem at all. What would be the correct way of doing so? Do I close this pull request and submit a new one? Or is it possible to overwrite the branch with a forced push?

srhb · 2019-02-08T12:04:30Z

@jamtrott The easiest way is to locally undo your changes with a soft reset, eg git reset master -- this will leave your changed files but uncommitted. You can now piecemeal commit each logical package, and finally force push your branch, which will update this PR. :)

dotlambda

Please add the pythonPackages. prefix to the respective commits.

pkgs/development/python-modules/mpi4py/default.nix

dotlambda · 2019-02-09T11:27:04Z

pkgs/development/python-modules/petsc4py/default.nix

+  meta = {
+    description = "Python bindings for PETSc, the Portable, Extensible Toolkit for Scientific Computation";
+    homepage = https://bitbucket.org/petsc/petsc4py;
+    license = stdenv.lib.licenses.bsd3;


Suggested change

license = stdenv.lib.licenses.bsd3;

license = lib.licenses.bsd3;

dotlambda · 2019-02-09T11:27:18Z

pkgs/development/python-modules/petsc4py/default.nix

@@ -0,0 +1,43 @@
+{ stdenv


Suggested change

{ stdenv

{ lib

dotlambda · 2019-02-09T11:27:45Z

pkgs/development/python-modules/petsc4py/default.nix

+    openmpi
+    openssh
+    mpi4py
+  ];


Some of those should probably be propagated. Why is python there?

Hi @dotlambda, thank you for your comments. I admit that I'm not sure about buildInputs versus propagatedBuildInputs. Do you have any suggestions or guidelines for determining which packages should be propagated?

The python package is not needed, and I have removed it from the list.

See https://nixos.org/nixpkgs/manual/#reference:

If something is exclusively a build-time dependency, use buildInputs; if it is (also) a runtime dependency, use propagatedBuildInputs.

See above for cython.
I don't find mpi4py in their setup.py.
numpy is in install_requires and should thus be propagated.
What do you infer the dependency on openssh from?

Sorry for taking so long before getting back to this. Anyway, I've pushed another couple of commits that address the issues you raise.

Both cython and numpy should be fixed now.

The dependency on openssh is needed to avoid tests from failing when using openmpi. This looks to me like a common pattern, and you will find the same in, for example, pkgs/development/python-modules/mpi4py/default.nix.

For reference, the test failures without openssh look something like this:

running install tests running test -------------------------------------------------------------------------- The value of the MCA parameter "plm_rsh_agent" was set to a path that could not be found: plm_rsh_agent: ssh : rsh Please either unset the parameter, or check that the path is correct -------------------------------------------------------------------------- [srl-login1:13710] [[INVALID],INVALID] FORCE-TERMINATE AT Not found:-13 - error plm_rsh_component.c(327) [srl-login1:13710] *** Process received signal *** [srl-login1:13710] Signal: Segmentation fault (11) [srl-login1:13710] Signal code: Address not mapped (1) [srl-login1:13710] Failing at address: (nil) [srl-login1:13710] [ 0] /nix/store/fivq0nbggp4y8mhy3ixprqd7qyn1hy2j-glibc-2.27/lib/libpthread.so.0(+0x11ed0)[0x1555532aeed0] [srl-login1:13710] *** End of error message *** [srl-login1:13707] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon on the local node in file ess_singleton_module.c at line 532 [srl-login1:13707] [[INVALID],INVALID] ORTE_ERROR_LOG: Unable to start a daemon on the local node in file ess_singleton_module.c at line 166 -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_ess_init failed --> Returned value Unable to start a daemon on the local node (-127) instead of ORTE_SUCCESS -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): ompi_mpi_init: ompi_rte_init failed --> Returned "Unable to start a daemon on the local node" (-127) instead of "Success" (0) -------------------------------------------------------------------------- *** An error occurred in MPI_Init_thread *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [srl-login1:13707] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! builder for '/nix/store/5h917aj1lxgxj4vpq1h5zn3ncpnarr32-python2.7-petsc4py-3.10.1.drv' failed with exit code 1 error: build of '/nix/store/5h917aj1lxgxj4vpq1h5zn3ncpnarr32-python2.7-petsc4py-3.10.1.drv' failed

This prevents issues when using multiple cython-based packages together (for example, mpi4py and petsc4py) due to code that has been generated with incompatible cython versions. Also, update homepage.

jamtrott · 2019-02-18T08:25:36Z

@srhb, @dotlambda, would you agree that the latest revisions address the issues you have raised?

dotlambda · 2019-02-21T09:17:52Z

pkgs/development/python-modules/mpi4py/default.nix


-  buildInputs = [ mpi openssh ];
+  buildInputs = [ cython mpi openssh ];


I guess cython belongs into nativeBuildInputs because it's listed in setup_requires: https://bitbucket.org/mpi4py/mpi4py/src/0d17284a209ebcc7af362e7d110d518308d43299/setup.py#setup.py-464.
See also the new explanation not yet available in the published manual: https://github.com/NixOS/nixpkgs/blob/b4acd97/doc/languages-frameworks/python.section.md#buildpythonpackage-parameters

Thanks, this should be fixed now.

dotlambda · 2019-02-21T09:18:46Z

pkgs/development/python-modules/petsc4py/default.nix

+
+  PETSC_DIR = "${petsc}";
+
+  setupPyBuildFlags = [ "build_src --force" ];


You should add the same comment here.

dotlambda · 2019-02-21T09:22:55Z

pkgs/development/python-modules/petsc4py/default.nix

+    openmpi
+    openssh
+    mpi4py
+  ];


See above for cython.
I don't find mpi4py in their setup.py.
numpy is in install_requires and should thus be propagated.
What do you infer the dependency on openssh from?

dotlambda · 2019-02-21T09:23:49Z

pkgs/development/python-modules/petsc4py/default.nix

+
+  propagatedBuildInputs = [
+    petsc
+    openmpi


I don't find openmpi in ther setup.py.
petsc can probably be dropped from here since you specify PETSC_DIR.

I moved openmpi to nativeBuildInputs and removed petsc. The package doesn't build without openmpi.

Remove mpi4py and make other packages nativeBuildInputs, except numpy, which is propagated.

tkonolige · 2019-07-11T17:13:57Z

Would it be possible to parametrize these packages by which MPI implementation they use instead of hardcoding openmpi?

It would also be nice to have debugging be optional.

Also, the PETSc developers recommend compiling with -O3.

FRidh · 2020-02-09T09:11:40Z

Would be good to see the merge conflicts resolved, because this seems otherwise ready.

stale · 2020-08-07T14:32:48Z

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

stale · 2021-06-04T23:56:05Z

I marked this as stale due to inactivity. → More info

jamtrott added 3 commits February 6, 2019 14:18

scotch: Enable building on darwin

e654992

Also, build libesmumps and libptesmumps to later support the mumps package.

scalapack: Enable building on darwin

f985c03

maintainers: add jamtrott (James D. Trotter)

5d3b88c

jamtrott requested a review from FRidh as a code owner February 6, 2019 19:26

GrahamcOfBorg added the 6.topic: python label Feb 6, 2019

srhb requested changes Feb 7, 2019

View reviewed changes

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Feb 8, 2019

jamtrott added 4 commits February 8, 2019 13:08

hypre: init at 2.14.0

a6a9e03

mumps: init at 5.1.2

54c2f41

spai: init at 3.0

5b095f6

petsc: 3.8.4 -> 3.10.3

c2f003b

dotlambda reviewed Feb 9, 2019

View reviewed changes

jamtrott added 2 commits February 10, 2019 16:18

pythonPackages.mpi4py: Re-compile cython code

15f103c

This prevents issues when using multiple cython-based packages together (for example, mpi4py and petsc4py) due to code that has been generated with incompatible cython versions. Also, update homepage.

pythonPackages.petsc4py: init at 3.10.1

07b91c6

dotlambda requested changes Feb 21, 2019

View reviewed changes

jamtrott added 2 commits June 17, 2019 11:07

pythonPackages.mpi4py: Move cython to nativeBuildInputs

689c107

pythonPackages.petsc4py: Fix build inputs and add missing comment

8b2a673

Remove mpi4py and make other packages nativeBuildInputs, except numpy, which is propagated.

ofborg bot added the 2.status: merge conflict label Jun 17, 2019

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 7, 2020

ryantm removed the 2.status: merge conflict label Oct 3, 2020

ryantm added the 2.status: merge conflict label Oct 3, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 3, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 4, 2021

jamtrott closed this Jun 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

petsc: 3.8.4 -> 3.10.3 #55358

petsc: 3.8.4 -> 3.10.3 #55358

jamtrott commented Feb 6, 2019

srhb left a comment

jamtrott commented Feb 8, 2019

srhb commented Feb 8, 2019

jamtrott commented Feb 8, 2019

srhb commented Feb 8, 2019 •

edited

dotlambda left a comment

dotlambda Feb 9, 2019

dotlambda Feb 9, 2019

dotlambda Feb 9, 2019

jamtrott Feb 10, 2019

dotlambda Feb 10, 2019

dotlambda Feb 21, 2019

jamtrott Jun 17, 2019

jamtrott commented Feb 18, 2019

dotlambda Feb 21, 2019

jamtrott Jun 17, 2019

dotlambda Feb 21, 2019

jamtrott Jun 17, 2019

dotlambda Feb 21, 2019

dotlambda Feb 21, 2019

jamtrott Jun 17, 2019

tkonolige commented Jul 11, 2019 •

edited

FRidh commented Feb 9, 2020

stale bot commented Aug 7, 2020

stale bot commented Jun 4, 2021

	license = stdenv.lib.licenses.bsd3;
	license = lib.licenses.bsd3;


		buildInputs = [ mpi openssh ];
		buildInputs = [ cython mpi openssh ];


		PETSC_DIR = "${petsc}";

		setupPyBuildFlags = [ "build_src --force" ];

petsc: 3.8.4 -> 3.10.3 #55358

petsc: 3.8.4 -> 3.10.3 #55358

Conversation

jamtrott commented Feb 6, 2019

srhb left a comment

Choose a reason for hiding this comment

jamtrott commented Feb 8, 2019

srhb commented Feb 8, 2019

jamtrott commented Feb 8, 2019

srhb commented Feb 8, 2019 • edited

dotlambda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamtrott commented Feb 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkonolige commented Jul 11, 2019 • edited

FRidh commented Feb 9, 2020

stale bot commented Aug 7, 2020

stale bot commented Jun 4, 2021

srhb commented Feb 8, 2019 •

edited

tkonolige commented Jul 11, 2019 •

edited