Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many packages from SciPy sprints important to numpy, scipy, numba, matplotlib #64751

Merged
merged 29 commits into from Jul 17, 2019

Conversation

costrouc
Copy link
Member

@costrouc costrouc commented Jul 14, 2019

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@costrouc costrouc requested a review from FRidh as a code owner July 14, 2019 22:36
@costrouc
Copy link
Member Author

Needed for python downstream testing. Motivation in directory https://github.com/costrouc/python-downstream-testing

@FRidh
Copy link
Member

FRidh commented Jul 15, 2019

I like what you've done in your repo. It's been on my mind to set up a Hydra instance to do the same things for numpy, scipy and pandas, but also core packages such as pip, setuptools and wheel.

@costrouc
Copy link
Member Author

This is a great opportunity for us to get more attention to nix. Currently a lot of work is being out into doing this work with conda (however due to conda structure they are not able to test all downstream packages -- it's a manual process for them). Getting nix to build current master and downstream packages of many repositories is really not that difficult. The only issue I've run into is that some nix derivations were configured to build the pypi tarbal not GitHub. Do you have any advice? What would the resources required be? Ive considered using Hercules CI and hydra. I see the build infrastructure being the harder part. I'm determined to get the POC working and know how to get this noticed once it is working.

@FRidh
Copy link
Member

FRidh commented Jul 15, 2019

Initially, I think I would go for a Hydra instance, with a job per project. Each job could evaluate the same expressions, however, in each job one of the expressions would be overridden to point to master. We could eventually also look into combinations (numpy@master and scipy@master). Hydra could monitor the repo's, fetching e.g. every 5 minutes. I think a single quadcore machine would be more than sufficient.

If projects want to use it and make it part of their CI they could use Travis to perform a Nix build. The project would then need a simple Nix expression that injects the package in the package set and builds a list of packages.

@costrouc
Copy link
Member Author

Thanks for the advice. I have a xeon quad core at home that's not doing much at the moment (and running nixos) so I'll setup hydra. I'm not too familiar with hydra (tired to set it up once). I think that will be the bigger learning curve for me.

Copy link
Member

@FRidh FRidh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't make the PR too big.

pkgs/development/python-modules/drms/default.nix Outdated Show resolved Hide resolved
@ofborg ofborg bot requested a review from FRidh July 15, 2019 17:25
costrouc added a commit to costrouc/python-downstream-testing that referenced this pull request Jul 15, 2019
@ofborg ofborg bot requested review from lsix, lovek323, domenkozar and madjar July 15, 2019 18:21
@costrouc
Copy link
Member Author

@FRidh I am stopping to add packages. Right now working on getting nix-review to pass

@costrouc costrouc force-pushed the python-downstream-init branch 2 times, most recently from c333940 to efa9a7e Compare July 15, 2019 19:34
@costrouc costrouc changed the title [WIP] Many packages from SciPy sprints important to numpy, scipy, numba, matplotlib Many packages from SciPy sprints important to numpy, scipy, numba, matplotlib Jul 16, 2019
@costrouc
Copy link
Member Author

@FRidh this package is ready for review and merge. I know that it causes a significant number of rebuilds. Happy to rebase to staging if needed. I have testing that all packages build.

@FRidh
Copy link
Member

FRidh commented Jul 17, 2019

The decorator bump may cause trouble. Well, let's hope it's OK.

@FRidh FRidh merged commit a0fae1b into NixOS:master Jul 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants