New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Many packages from SciPy sprints important to numpy, scipy, numba, matplotlib #64751
Conversation
Needed for python downstream testing. Motivation in directory https://github.com/costrouc/python-downstream-testing |
I like what you've done in your repo. It's been on my mind to set up a Hydra instance to do the same things for numpy, scipy and pandas, but also core packages such as pip, setuptools and wheel. |
This is a great opportunity for us to get more attention to nix. Currently a lot of work is being out into doing this work with conda (however due to conda structure they are not able to test all downstream packages -- it's a manual process for them). Getting nix to build current master and downstream packages of many repositories is really not that difficult. The only issue I've run into is that some nix derivations were configured to build the pypi tarbal not GitHub. Do you have any advice? What would the resources required be? Ive considered using Hercules CI and hydra. I see the build infrastructure being the harder part. I'm determined to get the POC working and know how to get this noticed once it is working. |
Initially, I think I would go for a Hydra instance, with a job per project. Each job could evaluate the same expressions, however, in each job one of the expressions would be overridden to point to master. We could eventually also look into combinations (numpy@master and scipy@master). Hydra could monitor the repo's, fetching e.g. every 5 minutes. I think a single quadcore machine would be more than sufficient. If projects want to use it and make it part of their CI they could use Travis to perform a Nix build. The project would then need a simple Nix expression that injects the package in the package set and builds a list of packages. |
Thanks for the advice. I have a xeon quad core at home that's not doing much at the moment (and running nixos) so I'll setup hydra. I'm not too familiar with hydra (tired to set it up once). I think that will be the bigger learning curve for me. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please don't make the PR too big.
6034b9f
to
52ee87c
Compare
52ee87c
to
e7b9d34
Compare
@FRidh I am stopping to add packages. Right now working on getting nix-review to pass |
c333940
to
efa9a7e
Compare
efa9a7e
to
fcbc479
Compare
fcbc479
to
05c5a17
Compare
@FRidh this package is ready for review and merge. I know that it causes a significant number of rebuilds. Happy to rebase to staging if needed. I have testing that all packages build. |
The decorator bump may cause trouble. Well, let's hope it's OK. |
Motivation for this change
Things done
sandbox
innix.conf
on non-NixOS)nix-shell -p nix-review --run "nix-review wip"
./result/bin/
)nix path-info -S
before and after)