Note that there may be a transition period of making various platform attributes more flexible in the wake of this change.

Maybe add $NIXPKGS_ALLOW_UNSUPPORTED_PLATFORMS?

That seems orthogonal, we already have the switch anyway.

What is the switch to do it on ad-hoc way from command line?

@shlevy not completely orthogonal: before the change, there is an environment variable that allows to experiment with platform support without changing any persistent configuration.

@dezgeg --arg config '{ allowUnsupportedPlatforms = true; }'

@7c6f434c Sure. I think the mac users are the most affected by this issue, I'll let them decide if they want it.

But --arg config {} overrides any other .nixpkgs/config.nix options, no?

Sure. You could import it from your command line... Why do we need an easy switch here exactly? allowUnsupportedPlatforms is not even something I think we should have, I'm just not removing it because it's unnecessary for what I want to do.

@shlevy I think your «should» is about the state where there is an actual tristate (but then we will need a variety of cheap overrides for allowUnknownPlatforms…)

As far as I know the primary use case is for Mac users to test if some package they want to use but depends on packages marked Linux-only happens to work by chance. Sounds pretty useful to me.

Also I personally do stuff like NIXPKGS_ALLOW_BROKEN=1 nix-build -A linux_rpi.src to compute source hashes of RPi-specific stuff on my main machine, but apparently that doesn't need NIXPKGS_ALLOW_BROKEN, I thought it did...

@LnL7 If you use allowUnsupportedPlatforms = true, IMO you're explicitly asking for things like the libseccomp Nix issue, no?

While I kind of agree, the primary use-case for allowBroken on darwin is to bypass meta.platforms = stdenv.lib.platforms.linux for packages that work fine. It's pretty tricky to override platforms without allowing unsupported packages.

@LnL7 Right, but I think the right thing to do there is fix the platforms setting. Implicit in the open world nature of it these days is that we can't actually test on all valid platforms, so we shouldn't use it as an "I've tested this" flag any more.

Per what @shlevy said, hydraPlatforms is sort-of a the known-good list. That can be cleaned up later.

@7c6f434c if we have meta.knownBad and meta.knownGood. "known-broken but probably fixable" would be satisfying meta.knownBad and meta.platforms

@7c6f434c Right now, the build platform is ignored entirely. I guess I figured it would be complex to go 2 dimensional, and hey if the cross stuff gets really good the build platform shouldn't ever matter 😂. In practice, the meta.broken thing can be used to add arbitrary boolean expressions.

Agreed. We could add meta.broken = !stdenv.isLinux to packages like libseccomp for now, then allowing unsupported systems will mostly work as desired.

@LnL7 err wouldn't that be the other way around? The allowing broken shouldn't add extra libseccomp deps on Darwin, right? The idea would be to make a lot more things meta.platforms = lib.platforms.unix or whatever except for things like libseccomp, then builds that fail get the new meta.knownBad or whatever we want to call it.

@Ericson2314 allow broken isn't really used to allow broken packages AFAIK 😄

@LnL7 Exactly. But with this change it would be. We'd sort of get today's allowBroken by default (minus the new libsecomp problems), and then add more blacklist and shrink the whitelist as needed.

@dezgeg @7c6f434c @LnL7 OK with merging this and iterating from here?

I think the error message still refers to enabling allowBroken, this needs fixing first:

error: Package ‘uboot-jetson-tk1_defconfig-2018.03’ in /home/tmtynkky/opt/nixpkgs/pkgs/misc/uboot/default.nix:88 is not supported on ‘x86_64-unknown-linux-gnu’, refusing to evaluate.
 
a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

(use '--show-trace' to show detailed location information)

@dezgeg updated

So which is the current recommendation to work with this configuration:

• allowBroken to true to get linux-pinned packages built on other platforms, e.g. try to fix those and enable unix
• allowUnsupportedPlatform to true for stuff like libseccomp

@shlevy Can we have a hint in the documentation please.

Yeah, I'll update the docs before merging this, just want to make sure the outstanding objections have been addressed.

@dezgeg's other concern in IRC over accidental optional/missing deps was resolved in that this one is even better about that than before. I hope that means we're in the clear but @dezgeg can confirm.

Hmm, does this enableIfAvailable work such that the result changes if the user sets allowUnsupportedPlatform?

@dezgeg Yes. IMO that's the only reasonable behavior, if you ask for unsupported platforms you should get them.

Now I'm utterly confused. I thought the point of the(se) change(s) to have separate options for allowing build of packages marked with meta.broken = true; and allowing build of packages with meta.platforms not containing the current system. That itself makes sense to me.

However, now (not necessarily due to this commit but whatever introduced this enableIfAvailable thing) if you set config.allowUnsupportedPlatform due to this use case @periklis mentioned ("to get linux-pinned packages built on other platforms, e.g. try to fix those and enable unix"), you also get as a a side effect some packages that used to build fine start breaking because now enableIfAvailable affects this package that used to work fine. And there's no way of opting out of that side-effect.

Did I understood this correctly? If so this sounds like a massive unsuspecting footgun.

@dezgeg

@periklis's full quote is

• allowBroken to true to get linux-pinned packages built on other platforms, e.g. try to fix those and enable unix

The idea is that if there's a package that should work on more than linux, first change it from meta.platforms = linux to meta.platforms = unix and allowBroken = !stdenv.hostPlatform.isLinux, and then use config.allowBroken = true;. config.allowUnsupportedPlatform shouldn't really be need to be used, and will indeed mess things up things like enabled optional libseccomp dependencies.

Uhh... but previously, if you were on Darwin and encountered some package that claims to be unsupported on Darwin, and if you had a theory that this was only due to the package (or its dependencies) having too strict platforms bounds (i.e. in practice, meta.platforms = linux;), you could test this theory with absolutely no code changes at all, just by doing NIXPKGS_ALLOW_BROKEN=1 nix-build '<nixpkgs>' -A foo, (i.e. adding a single environment variable or the equivalent --arg config incantation) and seeing how far you get. This exact workflow has been suggested on IRC several times.

After these changes, that workflow is no longer possible without those side-effects that I mentioned. Now you need to clone nixpkgs, start making those s/platforms.linux/platforms.unix/ edits before you even know if the end result is going to work or not. If it didn't all the effort of manually editing those files went to waste. To me this is a step backwards.

@dezgeg Yes that's true. That wasn't free though, the cost was tons of manual boilerplate for optional deps. The thing that makes this less bad is the meta.platforms = unix and allowBroken = !stdenv.hostPlatform.isLinux change can be done ahead of anyone trying to debug the specific package. We do enough of that one-time speculative work, and the workflow is NIXPKGS_ALLOW_BROKEN=1 nix-build '<nixpkgs>' -A foo just like before.

That wasn't free though, the cost was tons of manual boilerplate for optional deps.

How is changing say,

buildInputs = [ ... numactl ];

to:

buildInputs [ ... ] ++ lib.enableIfAvailable numactl;

"tons of less boilerplate" than:

buildInputs [ ... ] ++ lib.optional (stdenv.isLinux && !stdenv.isArm) numactl;

?

In both cases you had to modify the expression anyway, since there is no a priori way of knowing whether this specific package allows numactl to be optional or not (or whether it only allows it to be optional with some additional --without-numactl etc. flag). Sure, you save the one-time effort of writing the (stdenv.isLinux && !stdenv.isArm) part but now the tradeoff is everyone else who is reading the code won't know whether numactl will be included or not without looking at its expression and breaking this use-case in discussion.

The thing that makes this less bad is the meta.platforms = unix and allowBroken = !stdenv.hostPlatform.isLinux change can be done ahead of anyone trying to debug the specific package.

I don't understand. Who is supposed to be the one doing that change? The person who wanted to test their theory of the package's dependency closure having too strict platforms.linux? I somehow doubt most people want to be doing pull requests where the effective change is changing one error message to another.

I have to admit some serious fatigue with this issue. I think this change moves us in the right direction to reasonable semantics for the meta fields and lets us separate the question of what platforms make sense for a package (which belongs with that package) and which packages needn't depend on that package (which belongs downstream). But mostly I just want to be done with this back and forth. So if someone wants to take this on, great, if someone wants to revert enableIfAvailable, fine, just let me know what you decide at the end of the day.

@dezgeg

buildInputs = [ ... numactl ];

OK we all agree this with the null trick is bad, right? Because the null trick means there is no acknowledgement of the optionalness of the dep which is awful.

Your two things are both valid. Maybe that will help:

buildInputs [ ... ] ++ lib.enableIfAvailable numactl;

This is for "purely optional bonus deps". You use this if you really don't care whether numactl is available, and just want to use it if possible. This is why it also respects the other parts of meta like allowUnfree.

If B enableIfAvailable depends on C, and A depends on B-with-C, it's A's responsibility to make sure B uses C, not B's. [One would probably do a withC ? c.meta.available or false argument to B.]

buildInputs [ ... ] ++ lib.optional (stdenv.isLinux && !stdenv.isArm) numactl;

This is for a deps that are mandatory under certain conditions. You use this if you sometimes need the dependency, not merely could use it.

So yeah, I again think this PR should be fine given your concerns, along with the "should work" vs "known good/bad" multi-state. Whether optional deps are far more often of the form 2 than 1 is a separate matter.

I don't understand. Who is supposed to be the one doing that change?

Think of it as a TODO tracking mechanism. Maintainers for a platform (as oppose to the package) might think some packages ought to work on their platform, so they can do this change.

Where would the documentation need to be changed @periklis?

@Ericson2314 I think we need an extra section for allowUnsupportedPlatform in chapter 6

@dezgeg indicated that this PR itself is fine, or at least more fine than continued use of lib.enableIfAvailable. So let's write the docs (maybe I'll get to that), merge this, and continue deciding the fate of lib.enableIfAvailable somewhere else. I think this is is a good change on its own, whether or not lib.enableIfAvailable stays.

@Ericson2314 Thanks for picking this up!

OK documentation is added, clarifying division of labor between meta.platforms and meta.broken, and specifically not mentioning lib.enableIfAvailable in order to not blow by @dezgeg's lingering doubts. Merging momentarily.