Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages.poetry: init at 0.12.10 #53599

Merged
merged 9 commits into from Jan 10, 2019
Merged

Conversation

jbaum98
Copy link
Contributor

@jbaum98 jbaum98 commented Jan 7, 2019

Motivation for this change

Add the poetry package and its dependencies.

Closes #53132.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@worldofpeace
Copy link
Contributor

@jbaum98 Why enableParallelBuilding = true; in all the packages you've added?

@jbaum98
Copy link
Contributor Author

jbaum98 commented Jan 8, 2019

Thanks for your thorough feedback and sorry for what appears to have been a pretty sloppy PR.

@jbaum98 Why enableParallelBuilding = true; in all the packages you've added?

This could be entirely misguided, but I always set enableParallelBuilding = true; because I think 99% you want to be using all your cores, in part based on #5489.

@worldofpeace
Copy link
Contributor

Although for pylev getting the GitHub source worked so we could run tests because it uses setup.py, for clikit I'm running into a chicken-egg problem because it wants to use poetry as its build system to build from pyproject.toml. Thoughts?

My instinct is we bootstrap just by not using tests, and then somehow use that that poetry to rebuild its dependencies with tests, but I'm not sure how to do that.

I think we'd have to add a build-python-package-poetry at https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/interpreters/python if we wanted to build these project that use poetry. And obviously we'd have to merge this pr first before that.

However it is confusing that they'd distribute a setup.py in the tarball but not on their github repos.

@jbaum98
Copy link
Contributor Author

jbaum98 commented Jan 8, 2019

However it is confusing that they'd distribute a setup.py in the tarball but not on their github repos.

I'm guessing poetry produces the setup.py automatically when distributing because poetry isn't ubiquitous.

@jbaum98
Copy link
Contributor Author

jbaum98 commented Jan 8, 2019

Okay, I think I've fixed anything that isn't the poetry package test situation. Should I squash these commits one per package or one all together and we can land this PR, and then figure out how to bootstrap poetry?

@worldofpeace
Copy link
Contributor

Okay, I think I've fixed anything that isn't the poetry package test situation. Should I squash these commits one per package or one all together and we can land this PR, and then figure out how to bootstrap poetry?

I'll give your new commits a quick look to see if they're resolved properly. And yes we'd like a commit for each package.

As for the poetry test situation, I agree with that flow.

@worldofpeace
Copy link
Contributor

Remember to add doCheck = false; for the ones that we still can't get tests for.
Also add a comment above with why.

@worldofpeace
Copy link
Contributor

@jbaum98 Great, squash at will. Then I will begin some testing locally.

@worldofpeace
Copy link
Contributor

As for being able to build poetry based python packages, I think we can do something similar to what's done for flit.

Copy link
Member

@Ma27 Ma27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the enableParallelBuilding thing: I might've missed something while skimming through the thread, but as it's only used for make parallelism (which happens to be the "default" build approach when using stdenv.mkDerivation), why is it used here? IIRC buildPythonPackage directly invokes setup.py.

Another concern I have is that none of these packages has maintainers listed: we already have numerous breaking python packages on Hydra (due to too strict dependency constraints mostly), so I'd like to see as much of these packages as possible having maintainers that can help us keeping this package set as stable as possible.

@jbaum98
Copy link
Contributor Author

jbaum98 commented Jan 8, 2019

I'm happy to be the maintainer for these packages if that makes sense, but I'll need to add myself to the maintainers list.

@jbaum98 jbaum98 force-pushed the poetry branch 2 times, most recently from 95275bc to 196b3ee Compare January 8, 2019 19:15
@worldofpeace
Copy link
Contributor

#53599 (comment)
we don't have Python 3.4 on master anymore, so this shouldn't be needed.

My thought was that this is the constraint in the build file and it doesn't hurt to leave it here as documentation/just in case.

I agree with @jbaum98 even if it's not needed. I tend to try and make the derivation respect the things specified by upstream as close as possible.

Unless they want crazy nonsense that is, then maybe patch it all away 😄

@Ma27
Copy link
Member

Ma27 commented Jan 8, 2019

I'm happy to be the maintainer for these packages if that makes sense, but I'll need to add myself to the maintainers list.

That shouldn't be a problem I guess. Just add yourself in the first commit in this branch (this is what most new contributors do).

checkInputs = [ pytestrunner pytest hypothesis ];

# pytestrunner is only needed to run tests
patches = [ ./no-setup-requires-pytestrunner.patch ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you submit that upstream?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't sure, pytestrunner instructs people to do this https://github.com/pytest-dev/pytest-runner#usage

Copy link
Member

@dotlambda dotlambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last few commits should be squashed into the appropriate ones. Otherwise, this is fine.

@worldofpeace worldofpeace merged commit 1b1ea35 into NixOS:master Jan 10, 2019
@worldofpeace
Copy link
Contributor

@jbaum98 Thanks for your contribution 🎆

@jbaum98 jbaum98 deleted the poetry branch January 10, 2019 22:50

let
cleo6 = cleo.overrideAttrs (oldAttrs: rec {
version = "0.6.8";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually can't have a Python package depend on a version that's not the default because this can lead to conflicts. Was there a reason we allowed this back then?
And isn't poetry only ever used as an application, i.e. never imported? In that case, we should move it out of pythonPackages.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually can't have a Python package depend on a version that's not the default because this can lead to conflicts. Was there a reason we allowed this back then?
And isn't poetry only ever used as an application, i.e. never imported?

Yeah it is weird that I forgot about that, especially since we're always mentioning this 😄
I can move this out of python-modules.

Copy link
Member

@dotlambda dotlambda Feb 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're sure it's never imported, feel free to go ahead. Thank you in advance!

@adisbladis
Copy link
Member

adisbladis commented Nov 19, 2019

Fyi I've repackaged poetry with poetry2nix in https://github.com/adisbladis/poetry2nix/tree/master/pkgs/poetry (1.0.0b5).

This fixes a bunch of issues that this package currently has:

  • It doesn't propagate PYTHONPATH
    This ensures that things building with poetry as a PEP518(pyproject.toml) build-system will not get a mix of dependencies from your pyproject.toml and nixpkgs poetry.
    It does this by using a similar "vendoring" approach as the official poetry installer does.
  • build-system is a first class supported thing
    Poetry is injected into the build if it's defined as a build-system
  • It does not require overriding any modules from python-packages.nix
    A unique coherent package set is created for every leaf package, thus avoiding conflicts between dependencies.
  • It's dynamic and builds from pyproject.toml/poetry.lock.
    This makes packaging things much easier, a typical Poetry project packaged looks something like:
poetry2nix.mkPoetryPackage {
  inherit python;
  pyproject = ./pyproject.toml;
  poetryLock = ./poetry.lock;
  src = fetchFromGitHub {
    owner = "sdispater";
    repo = "poetry";
    rev = "1.0.0b5";
    sha256 = "1l7vd233p9g8sss5cxl6rm6qyf55ahbb9sw7zm3sd5xchcsiyday";
  };
}

I intend to get this into nixpkgs as soon as Poetry releases a 1.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[New Package Request] Add Poetry – for Linux et al.
7 participants