Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] pythonPackages.numpy-mkl: refactor adding mkl support #48676

Closed
wants to merge 1 commit into from
Closed

[WIP] pythonPackages.numpy-mkl: refactor adding mkl support #48676

wants to merge 1 commit into from

Conversation

costrouc
Copy link
Member

Motivation for this change

Would love to see packaging python applications with mkl support similar to how conda packages mkl. I realize that this pull request is very trivial and probably not the correct approach to packaging mkl into numpy and would love any input. Hopefully this pull request gets enough input that it can be used as a guide on how to integrate mkl with python packages. Packages that I would like to eventually target are NumExpr, SciPy, and Scikit-Learn.

I would like to especially thank @bhipple for the mkl PR! I have been wanting that for a long time but did not know how to do it. If at all possible I would like to hear input from @bhipple and @markuskowa. @smaret you provided great input making sure that it works with darwin and since I do not have a mac I would appreciate any help.

I am aware that this will cause mass rebuilds and will eventually move to staging when it is working. Currently if it is on staging it will require mass rebuilds making it harder to test.

Things done

Adding mkl support for numpy. As of now this PR does not succeed in building. The numpy tests fail.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@costrouc costrouc requested a review from FRidh as a code owner October 18, 2018 19:12
@costrouc costrouc changed the title pythonPackages.numpy: refactor adding mkl support pythonPackages.numpy-mkl: refactor adding mkl support Oct 18, 2018
@FRidh FRidh self-assigned this Oct 20, 2018
@FRidh
Copy link
Member

FRidh commented Oct 20, 2018

Thanks. I've pushed 277b73a to staging.

@FRidh FRidh closed this Oct 20, 2018
@costrouc
Copy link
Member Author

@FRidh this pull request is not complete sorry and it won't build properly. I'll add the WIP headline

@costrouc costrouc changed the title pythonPackages.numpy-mkl: refactor adding mkl support [WIP] pythonPackages.numpy-mkl: refactor adding mkl support Oct 20, 2018
@FRidh
Copy link
Member

FRidh commented Oct 20, 2018

I made some more modifications and now the packages that I tested seem to work with it.

@costrouc
Copy link
Member Author

Oh great and thank you! I look forward to benchmarking the performance differences between the two builds numpy and numpy-mkl.

@bhipple
Copy link
Contributor

bhipple commented Oct 22, 2018

Thanks for this work @costrouc and @FRidh! I have a bit of a side project to package as much of the Anaconda distribution as possible in Nix, so that scientific computing users have more choices on how to install and distribute their packages. It's great to see others are interested in this as well! CC @jluttine

@costrouc
Copy link
Member Author

costrouc commented Oct 22, 2018 •
edited

@bhipple this is a goal of mine as well. I am happy to help in any way to help theses goals.

You may be interested that I have a PR that will add binderhub support for nix. This will mean that any nix repository can be turned into an interactive notebook only requiring a web browser.

Goals for me include:

  • adding mkl support to necessary packages (scipy and scikit-learn)
  • cleaning up the python ecosystem in nix
  • add support for jupyterlab extensions lots of javascript...
  • adding ways to benchmark different package configurations (e.g. openblas vs mkl)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

Assignees

@FRidh FRidh

Labels
6.topic: python 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@costrouc @FRidh @bhipple @GrahamcOfBorg