New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove meta.available checks #45727
Remove meta.available checks #45727
Conversation
Thanks for the mention! Let me also mention @Ericson2314, @shlevy, @matthewbauer as common users of the attribute in question (via `git log -p -S meta.available`).
1) Note that `meta.available` wasn't designed for this use in the first place, it was designed for NixOS/nix#1771.
In #35906 (comment) I wrote about
Now I'm experimenting with `meta.supported` that only checks the platforms
And `meta.supported` is not in nixpkgs.
2) I agree that the current use of `meta.available` might be problematic for porting to new platforms when you might want to keep packages marked broken while still having everything in the common nixpkgs, but I don't think we should just duplicate meta.platforms of those packages everywhere (i.e. "But encapsulation!" argument).
Since there's a demand now, let's just implement `meta.supported`.
I propose you to copy the thing in the change to pkgs/tools/filesystems/ceph/generic.nix in this PR to `meta` in make-derivation.nix and then do `s/meta.available/meta.supported/` treewide. That should work.
(Also, `x.meta.available or false` must be just `x.meta.available` (same for `supported`) as those attributes should always be available, it's a bug that has to be fixed if it doesn't always work.)
3) So, specifically:
- I completely agree with `allowBroken` and "user or Darwin trying to port to RISC-V" arguments, but this would be fixed with `meta.supported`, not by replicating partially evaluated versions of `meta.supported` everywhere.
- For `allowUnsupportedSystem` I think you should just add new platform to the package in question, but then mark packages `broken` on said platforms if you so desire. So that use case should be fixed with `meta.supported` too.
- `allowUnfree`, too, would be fixed, but I kinda liked the side-effect of the old way: if you didn't have `allowUnfree` set, everything that could be built without unfree deps would be built without them. Arguably, that's not a good feature, though. (And I'm not sure anything actually uses it yet, I think that's unlikely.) Having those things just fail to evaluate is better for debugging.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Strong no. I've tried to articulate where this should and shouldn't be used, I'm not sure you've even understood my points. I'll try agin
allowUnsupportedSystem
should never be used or used to be used.allowBroken
should be used for the purposes of trying to get something to work.
That means if you see something that ought to be build but doesn't, make sure it's marked broken (rather than impossible with meta.platforms
). If you want to try building something, make sure it's marked broken or working first rather than using allowUnsupportedSystem
. This try-state system allows conveying information which is impossible to convey today.
- Packages should only use
enableFoo ? !(foo.meta.available or false)
for purely optional dependencies that are never needed under any situation.
The libseccomp example is a false example because I believe libseccomp is in fact needed on Linux. If so, just that dependency edge should be modified.
Also, I am suspicious that @dezgeg, who frequently though not always is against support more systems / combinations of features, is also against the abstractions that make supporting more combinations tractable.
Also, this proposal adds back more |
I agree on getting rid of meta.available as much as possible. IMO it is much clearer to be more specific when something is 'available' and include the condition directly.
I think that's a bad idea. It doesn't make sense to put these things in meta in the first place. Let's just use |
@@ -16,7 +16,7 @@ | |||
|
|||
, # If enabled, GHC will be built with the GPL-free but slower integer-simple | |||
# library instead of the faster but GPLed integer-gmp library. | |||
enableIntegerSimple ? !(gmp.meta.available or false), gmp | |||
enableIntegerSimple ? false, gmp ? null |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO gmp ? null
is confusing to new users. gmp will never be null even if gmp is broken.
@@ -11,8 +11,8 @@ | |||
, hostPlatform | |||
, buildPackages | |||
, withSelinux ? false, libselinux | |||
, withLibseccomp ? libseccomp.meta.available, libseccomp | |||
, withKexectools ? kexectools.meta.available, kexectools | |||
, withLibseccomp ? !stdenv.hostPlatform.isRiscV, libseccomp |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned in my comment, I think this can be replaced with any (x: x == stdenv.system) libseccomp.meta.platforms
. This avoids the allowBroken
issue while also making it more reusable for other platforms.
@matthewbauer you cannot do |
I'm for the record I'm fine with doing |
I do agree that @Ericson2314 libseccomp isn't required on linux, I have an arm box where I disabled that because because it didn't work. |
No, I do not understand the following point at all:
Why not? In the pull request message, I have listed the use case where this is useful.
How do I know a priori whether a package is "ought to build" on a given platform? Let's have a concrete example:
Yes, trying to build something is the exact thing I want to achieve. But before these
It used to be a required dependency for Nix on Linux, but it was explicitly made optional because RISC-V doesn't support libseccomp: NixOS/nix@690ac7c. And I personally agree with that tradeoff. It's far better to have Nix working at all on RISC-V even if insecure in a multiuser context than not supporting Nix on RISC-V at all. But obviously, once RISC-V supports libseccomp, Nix on RISC-V should be switched to use it once it is tested that Nix actually works in that configuration, and that the code to enable the less secure configuration just to support Nix on RISC-V is removed from the nix expression for Nix. And thus, if the code reads:
I can immediately know that the workaround applies to RISC-V, and to only RISC-V. If I were to replace
I cannot know if that workaround can be removed safely (without potentially breaking some other system) or not without digging into the definition of
What I want is that Nixpkgs works as consistently as possible even in the rarely used configurations or architectures. Disabling optional stuff to temporarily make things to work should be used as the last resort. And on a more general level, code is read more often than it is read. When reading code, code which says With all that said, I got the impression from the people commenting on this (including me) that code of the form |
anything is better than `null`
Oops, I missed those while skimmind through the thing. I wholeheartedly agree that no new `null`s should be introduced into package arguments. Those are cancer.
@matthewbauer
> Since there's a demand now, let's just implement `meta.supported`.
I think that's a bad idea. It doesn't make sense to put these things in meta in the first place. Let's just use ```any (x: x == stdenv.system) pkg.meta.platforms```. ```meta``` should avoid calculated values as much as possible.
It's lazy, though, so I don't think this argument works out. It might even be more efficient to keep it there, since nix does no common expression elimination itself. That needs to be measured, thought, I have no idea if one new thunk in `meta` would cost more than some `n` reevaluations of `any (x: x == stdenv.system) pkg.meta.platforms`.
@dezgeg
With all that said, I got the impression from the people commenting on this (including me) that code of the form `any (x: /* something here */) libseccomp.meta.platforms` (or an equivalent helper function) would be either equivalent or better compared to `libseccomp.meta.available`.
I agree. I think that binding that expression to `meta.supported` would be cleaner, but I'm inclined to yield to a helper function like
```
withLibseccomp ? lib.isSupportedOnTheHostPlatform libseccomp
```
(with a better name that eludes me ATM).
…--
Thinking some more about this, I'm pretty sure that doing the `meta.supported` thing will make memory consumption go up a little, so a helper function is probably a better choice, considering current problems with #38635.
|
It sounds like we have consensus with the compromise. That's a relief. But for posterity, let me respond to the points:
Ah, I think this is the core of the misunderstanding. Read this as is "it is always permitted to build without
This reads as "it is always permitted to build without
This reads as " The differences here are modal, just like https://tools.ietf.org/html/rfc2119. |
Yes, this is matter of taste. The README mentions Ubuntu and Windows, so I'd say |
You can only do that assuming rampant duplication where downstream packages constantly guess when their optional dependencies are available. Even with @matthewbauer's plan, then |
@@ -6,7 +6,7 @@ | |||
, storeDir ? "/nix/store" | |||
, stateDir ? "/nix/var" | |||
, confDir ? "/etc" | |||
, withLibseccomp ? libseccomp.meta.available, libseccomp | |||
, withLibseccomp ? stdenv.hostPlatform.isLinux && !stdenv.hostPlatform.isRiscV, libseccomp # RISC-V support in progress https://github.com/seccomp/libseccomp/pull/50 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could do
{ ...
, withLibseccomp ? stdenv.hostPlatform.isLinux && libseccomp.meta.available
}:
# RISC-V support in progress https://github.com/seccomp/libseccomp/pull/50
assert stdenv.hostPlatform.isLinux && !libseccomp.meta.available
-> stdenv.hostPlatform.isRiscV;
# Useless on other platforms
assert withLibseccomp -> stdenv.hostPlatform.isLinux;
This precisely formalizes:
Libseccomp SHOULD be used on Linux where possible. If we are on linux other than RISC-V libseccomp MUST be available, to ensure we only don't use it intentionally, never automatically. If we are not on Linux we MUST NOT try to depend on libseccomp; even if it could somehow be built for us, we wouldn't know how to use it and we wouldn't want to pretend otherwise.
No. I repeat: the fact that libseccomp was made optional on Linux is only due to libseccomp not having support for RISC-V yet. It is a workaround. Once libseccomp gains support for RISC-V, the code for the workaround should be dropped entirely, since it is no longer needed.
I suppose by "it doesn't work" you mean Nix builds fail at runtime due to some ancient kernel which lacks support for it? But that doesn't affect building Nix, right? All you need is the runtime configuration switch you mentioned? |
In other words, you were forced to make a guess. The reality (when I last checked): in Nixpkgs it builds successfully and works on As far as I am aware, the rules for changing Is that not the case? Since when? Where has this been discussed and documented? I certainly don't see anything in https://github.com/NixOS/rfcs. |
I do not understand at all. If I find with |
@dezgeg It is fine to conceive of |
At the risk of being pedantic, When I said about
I don't think using |
Agreed.
I don't think the facts you describe are that different from the current situation. To take your Cocoa UI x11 example, a package that so far only is expected to work on Linux might well make a bunch of such things unconditional. If this is an end application, rather than library, it's probably fairly likely, I think it's only practical to always consider the possibility that the dependency may not be needed. And this risk of "overly needed dependencies" is always there, whether one uses |
Anyways, the 18.09 fork is soon, so I suppose we better get drafting on the compromise. We can debate the points of things obviated by the compromise after. One of you all want to do it? I have a few too many other last minute things to do. |
Maybe we need to have a separate platform «Linux with userns and seccomp and all the new interfaces like that turned off in the kernel» then? Because it annoyingly still exists in practice. |
dc71ac3
to
f6b29c6
Compare
Success on aarch64-linux (full log) Attempted: nix, systemd The following builds were skipped because they don't evaluate on aarch64-linux: ghc Partial log (click to expand)
|
Success on x86_64-darwin (full log) Attempted: ghc, nix The following builds were skipped because they don't evaluate on x86_64-darwin: systemd Partial log (click to expand)
|
Success on x86_64-linux (full log) Attempted: ghc, nix, systemd Partial log (click to expand)
|
As of f6b29c69beb2cd60aebf19c0d088e53ae939dc4f
pkgs/tools/filesystems/ceph/generic.nix
pkgs/development/libraries/wiredtiger/default.nix
still use `==` instead of `platformMatch`.
|
This sort of code breaks config.{allowBroken, allowUnsupportedSystem} = true by making them do unpredictable things.
This sort of code breaks config.{allowBroken, allowUnsupportedSystem} = true by making them do unpredictable things.
This sort of code breaks config.{allowBroken, allowUnsupportedSystem} = true by making them do unpredictable things.
This reverts commit 79d8353. This sort of code breaks config.{allowBroken, allowUnsupportedSystem} = true by making them do unpredictable things.
This sort of code breaks config.{allowBroken, allowUnsupportedSystem} = true by making them do unpredictable things.
This sort of code breaks config.{allowBroken, allowUnsupportedSystem} = true by making them do unpredictable things.
f6b29c6
to
2666fb0
Compare
Good catch, fixed ceph and wiredtiger (they hadn't matched my sed expression). |
Success on x86_64-darwin (full log) Attempted: ghc, nix The following builds were skipped because they don't evaluate on x86_64-darwin: systemd Partial log (click to expand)
|
Success on aarch64-linux (full log) Attempted: nix, systemd The following builds were skipped because they don't evaluate on aarch64-linux: ghc Partial log (click to expand)
|
Success on x86_64-linux (full log) Attempted: ghc, nix, systemd Partial log (click to expand)
|
I got the impression this solution is satisfactory to all for now, it can be improved later further if necessary. |
Before these
meta.available
checks spread into Nixpkgs,config.{allowBroken, allowUnfree, allowUnsupportedSystem}
only affected the visibility of packages: switching those config options to true just gave access to more packages without affecting the other ones. But with these sort of checks, those options will now affect hashes of packages:With the old way, if a (say) Darwin user would try to build some package and find it (or its dependencies) marked not supported/broken on Darwin, they could try to build the package with
config = { allowUnsupportedSystem = true; allowBroken = true; }
(or with the equivalentNIXPKGS_ALLOW_*
env vars) to see if it would actually build or gauge how much there is to fix. All this without cloning Nixpkgs or editing any files in it.With these
meta.available
checks in, the above can't be done reliably anymore because these checks will now enable some unwanted dependencies. As a practical example, trying the above on any package depending on Nix will always lead to build failure because it will now attempt to build libseccomp for Darwin and fail.The old behaviour was useful. The new behaviour is insane. Hence remove these
meta.available
checks to restore what used to work.