Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

babelfish: add meta.platforms #109178

Closed
wants to merge 1 commit into from
Closed

babelfish: add meta.platforms #109178

wants to merge 1 commit into from

Conversation

cole-h
Copy link
Member

@cole-h cole-h commented Jan 12, 2021

Otherwise, Hydra will not build it, and it won't be substitutable from
the cache.

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Otherwise, Hydra will not build it, and it won't be substitutable from
the cache.
@cole-h cole-h mentioned this pull request Jan 12, 2021
10 tasks
Copy link
Member

@hugolgst hugolgst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cole-h
Copy link
Member Author

cole-h commented Jan 12, 2021

As commented in #108942 (comment):

Even though this may be the case, I feel like platforms should be set explicitly in all packages so that, when using plain stdenv.mkDerivation (or any wrapper that doesn't automagically set platforms), people don't forget to set it. The manual states that "Platforms should be set" (admittedly for the purpose of getting cached by Hydra).

If somebody who is new to Nixpkgs bases their package on something that is buildPythonPackage or buildGoModule (but doesn't use a wrapper like those), they might not see platforms getting set and do the same, and (if uncaught) end up wondering why their package isn't being substituted.

Though it is true that my "substitution" reasoning (for this particular package) has been proven false.

@cole-h cole-h reopened this Jan 12, 2021
@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Jan 12, 2021

any wrapper that doesn't automagically set platforms

buildGoModule sets this.

We have 71 packages setting this out of 879. I am pretty sure not any random person will use exactly this package.

rg buildGoModule -C 1000 | grep platforms.unix | wc -l
71

rg buildGoModule | wc -l
879

Also if someone starts a new package they should probably take an example from the contribution guide or that fits their usecase and not convert buildGoModule to mkDerivation.

@infinisil
Copy link
Member

I think the fundamental issue here is that meta fields aren't being required. If we want people to always set a platform, we should enforce that it's set. So then if it's already set by a wrapper, people don't have to set it in their own expressions. Of course this is a bigger undertaking, but I think that's the best way we can bring people to set all the fields we expect in the end. So I don't think these single changes are worth it.

P.S.: @SuperSandro2000 Try to only close PR's once an agreement has been reached with the author, otherwise this feels very cold.

@SuperSandro2000
Copy link
Member

Sorry, closed by accident.

@infinisil
Copy link
Member

Another thing: Ideally it should be allowed to set platforms multiple times, where each definition restricts the supported set further (so merging by intersection).

Copy link
Contributor

@kevingriffin kevingriffin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do like having meta.platforms be set explicitly. It honestly slipped my mind, because I based this off of another go package that didn't have it. As @infinisil said, intersection merge feels like the most correct way of doing things in the future.

I feel good either way given how few packages already have this, but have a weak preference for adding the explicit declaration in preparation for a more interesting merge strategy in the future.

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Jan 13, 2021

I feel good either way given how few packages already have this, but have a weak preference for adding the explicit declaration in preparation for a more interesting merge strategy in the future.

Going by how many packages have something in nixpkgs is not an argument we can use except if we talk about backwards compatibility and how to move things on when doing tree wide changes. There are so many packages which are poorly maintained and only had treewide sed changes like unquote URLs in the last 5 years or so. Many of them use now anti patterns and if we base design decisions on how often something is done like this in nixpkgs on those packages we get really distorted stats.

Also I just thought about Go, Python and Rust packages again and if they are written in their pure languages without any c-bindings than they are most of the time Unix compatible except if they are specially designed for some operating system. If they have c-bindings or are only designed for a few platform then we should set platforms. For all other packages the compatibility depends more on the compiler or interpreter and setting platforms there only hurts non Linux platforms.

For example almost all python packages which only contain pure python code will work on Linux and Darwin and probably on all platforms
that pythonn supports. It would just add a huge maintaince work if nix ever supported a new platform that is not unix and python also supported that. Then someone would need to change platforms for all all python packages they want to use just to discover in the end that all pure python packages just work.

Platforms right now (and broken) is mostly an information for hydra to not built packages which are known to not compile and to also inform users that the package they try to use definitely does not compile on their platform. It does not indicate that a package is properly tested on that platform. Seeing that I am one of the maybe 5 people which actively review packages on another platform than linux for me this just boils down to either a huge maintaince work (which I won't take) or actively putting non Linux platforms in a giant disadvantage.

In the current way of things we can't express that a package is known to be working well on a platform without restricting it on non tested platforms. There are proposed ways above to maybe change that but they are way out of scope of this PR.

@cole-h if this is really important to you I just merge this PR but I am actively thinking about removing platforms.unix from all go, python and rust packages to not have duplicated information in nixpkgs.

@cole-h
Copy link
Member Author

cole-h commented Jan 13, 2021

I didn't mean for this change to be so controversial, nor did I mean to attack / argue / etc. with you. I was just going "by the book" as it were -- the Nixpkgs manual specifies, when reviewing new packages, to ensure the platforms are set (granted, it's reasoning is so that Hydra will build it). If you want to remove meta.platforms, it should be accompanied by updated documentation.

Thanks all for your input.

@cole-h cole-h closed this Jan 13, 2021
@cole-h cole-h deleted the babelfish branch January 13, 2021 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants