👍 from me.

This almost perfectly matches my own thoughts. I’m all for it. 👍

Given that the only hardware platform that >90% have with root access is x86_64, and that we don't have a free software Darwin image that can be booted in a VM, doing portability fixes on x86_64-linux does sound appealing.

And with the current amount of effort towards cross-compilation, don't we have any cross-targets in use by Nixpkgs users where musl is the preferred libc?

IMHO it's best to keep musl support in a branch or overlay so that the effort of maintaining musl-specific patches is not inflicted on package maintainers who don't care about musl.

(The same applies to RISC-V support BTW, which really should also be the subject of a RFC.)

I am in favor of a design where everything is in one repo where a broken musl doesn't impact glibc package users. I am not in favor of a branch based approach or overlay. glibc should only have a temporary special status in this regard, such that at some point we might even switch to musl as the default library.

Overlays work best when the package set is different. In the case of musl it seems like it would be hard to reach into all the packages with overrides. Nix is quite composable but some times it's not possible to pry open a derivation enough to replace the inner parts (like when a function is hiding the scope).

How about accepting the musl patches but not making it a CI failure? It's not official but it's already the case that we have supported platforms and unsupported ones in nixpkgs.

I mean as part of an unsupported platform we would set the expectations: it's okay to break the musl patches as part of the maintenance cycle. The musl team can have their own build and test infrastructure to sweep after breaking changes.

I'm super for it. And I'd like some cross platform to in fact be required by CI too. I think it's good that different platforms make packaging harder, it keeps us and upstream honest. Lastly, Musl, with it's minimal set of features, is a great test because if a package is built with musl, it can probably be built with anything.

Just to reiterate, the key to writing portable software is parametricity: don't add endless #ifdefs or conditionals in Nix, but abstract over platform differences as far upstream as possible so downstream need not care. With this principle, more platforms is not linearly more work.

@edolstra are you also against the cross-compilation support in the main repository?

Making a call on macOS failures is already needed, and with musl I can at least reproduce the problem locally and see if it is easy to fix; so it is just the question about the level of expected stability.

@Ericson2314 it's like every platform. First it should go through a trial phase where compilation success is not required. Once it has been demonstrated as a viable target it can be promoted to the supported platforms.

The question is, who is responsible for deciding which platforms are supported or not :)

@zimbatm Agreed, I think a release.nix + release-cross.nix merge should be done by 18.03 with no new required-to-pass derivations, as exactly that second step.

Alright let's do this. Any objections @edolstra ?

I should add that @dtzWill was a bit more to the point in the original RFC than me pontificating on parametricity above. I think prior portability efforts were a bit more "conditional-explosion-y" than the current ones: the patches we're capable of landing today really do improve the software in code quality too, and I'd love to upstream them. Given the necessary ugliness of earlier efforts, I understand and sympathize @edolstra's reservations. Given the elegance we can get away with now, I don't think those concerns still apply.

I've noticed the RFC as-is doesn't go much in to how maintenance will work:

• how we test and verify patches work
• how well will musl be supported in the end
• do most packages work out of the box with musl?
• what impact does this have on Hydra?
• what impact does this have on ofborg?
• how do we detect if a patch is breaking musl support?

We should also take care to remember this adds a bit more complexity to many derivations.

Does musl work on macOS? What happens if someone tries to use musl on MacOS in Nixpkgs?

@graham I think the idea is that initially, nothing is supported, but merely allowed to exist.

how we test and verify patches work

The only way to is to build, but the package maintainer isn't obligated to do this.

how well will musl be supported in the end

Depends how the experiment goes

do most packages work out of the box with musl?

Yes. Packages that don't will probably be eventually broken on glibc as it becomes more standards-enforcing. (I didn't know this until I read the RFC. This is great news as too much of GNU doesn't require opt-in to extra GNU-specific features.)

What impact does this have on Hydra / of borg?

Nothing. Well almost nothing; there are a few odd jobs in release-cross.nix.

how do we detect if a patch is breaking musl support?

We don't need to for an unsupported experiment.

the idea is that initially, nothing is supported, but merely allowed to exist.
Depends how the experiment goes

I don't love the idea of having unsupported, unexercised code sitting in Nixpkgs. What is the point of having it in nixpkgs, if it is allowed to be broken by people who don't even know it is being broken? At that level of "unsupported", there is very little reason to have it in the repo. Nixpkgs receives almost 500 commits a week, and a lot of rot can set in over that time.

I think if we're going to add major new support, part of the experiment should be at least diagnostics-level checking of the code per PR.

If you disagree with this, please pretend we're a few months down the line and musl is a resounding success and I re-asked all those questions.

P.S. I feel positive about this project.

A new libc or port is simply not something that can realistically live out of tree, overlays etc. aren't expressive enough. Saying that we need a full support story before we can start work proving viability, attracting maintainers, and building userbase is effectively saying "go fork nixpkgs, this is never going to happen upstream."

Or if you disagree, how exactly do you propose we go about this kind of thing such that we have a realistic chance of ever getting it merged, given the above constraints?

The point of having it in nixpkgs is so that people can use it, and iterate on it. People undertaking these efforts understand there is burden on them to do whatever form of CI they need on their own resources until there is enough community support for more. If things bitrot, that's what git rm is for.

I hesitate to post this because I think it's a worse solution than allowing it upstream, but: if the ultimate answer here is no (and, setting this particular RFC aside, we don't seem to have any way to get to an ultimate answer in these...), then I'll (on my own fork) start nixpkgs-next à la linux-next, where all sorts of experiments are regularly merged on top of master and it's expected that the individual experiment maintainer makes sure things don't break for everyone else (enforced by being removed from the repo list until fixes are in) and users who want to take advantage can use a common place that's kept up-to-date with master.

To reiterate, this is a worse solution than finding a way to let this kind of work happen upstream. We're not the kernel community, and the last thing we need is further impaired discoverability, further divergence in possible user configurations, and more third-party infrastructure. But I'm tired of ideas being shot down because of potential maintenance burden without any viable alternative existing, so there's a viable alternative.

You can also count on IOHK helping maintain musl (as a consumer of the work).

About the integration: I'd start with one codebase (nixpkgs master) and decide otherwise if it's too much conditionalization once the prototype is ready for review.

OK, I think this is a go then. @dtzWill any modifications before we merge?

I'd say double check with @grahamc that his questions are addressed. That should be quick back and forth.

1. "musl team" is reponsible, to be formed please form that as part of this RFC. It sounds like you have the team here, it just needs to be written down. We could create a proper GitHub team handle that can be pinged, too.
2. These changes usually can be safely applied on all platforms (although sometimes they are not for rebuild reasons) I'd be inclined to just do it anyway, especially if these changes are largely fixes improving compliance and correctness resulting in higher-quality programs" holds true. Reducing the confusion and variability and chances some other patch only breaks on musl. Essentially, reducing the cyclomatic complexity of nixpkgs.
3. Please open an issue on https://github.com/nixos/ofborg with a details description (or even, a PR!) of how to add it to the evaluation checker. Here is probably the best place to look on how the eval checks are done cross-platform: https://github.com/NixOS/ofborg/blob/released/ofborg/src/outpaths.nix

Following these, as far as I'm concerned,

I would say that if a musl patch is changing just the assumption «Linux = Android xor glibc», it could be OK to delay its application on Darwin to reduce the number of rebuilds, but eventually applying it could indeed be a good idea.

As a non-contributor, I don't know if my opinion carries any weight at all, but here it is anyway:

• I don't have a strong opinion on process or responsibilities, but if it can be made to work then as NixWRT wrangler I would find it absolutely marvellous: switching my (admittedly small) package set from glibc to musl was accomplished in a single evening and the only change needed in nixpkgs was actually a cross-compilation fix not a musl fix. It opens up so many opportunities for using the Nix package collection in low-resource/embedded scenarios.

• Despite that, I would not be in favour of this if I thought that it was going to be a gateway to "combinatorial explosion hell" - adding ugly conditional fixes to packages to get them to build with a feature-impoverished or buggy C library. But by all accounts it's not that and could even be looked on as a form of "linter" - improving the compatibility and standards-compliance of the code in the package collection.

tl;dr I can't speak to whether this approach to supporting musl-in-nixpkgs is the right one, but I can say I'd definitely be a user of musl-in-nixpkgs and so the need addressed by this RFC is (for me, anyway) real.

Alright, I suppose it's ready? 😇

@grahamc I liked the team idea so I put it in the RFC, hope that's alright 👍.

I'm interested in being the part of the musl team, too.

I would not be in favour of this if I thought that it was going to be a gateway to "combinatorial explosion hell" - adding ugly conditional fixes to packages to get them to build with a feature-impoverished or buggy C library. But by all accounts it's not that and could even be looked on as a form of "linter" - improving the compatibility and standards-compliance of the code in the package collection.

Hear, hear!

(... I just switched my NixOS profile to use musl-built Nix 2 😁 ...)

Note that this currently requires a small patch to Nix to avoid stack-allocating large buffers: dtzWill/nix@b538660

Just going to say that I'm really hoping that if we'll accept this RFC, we'll do so by 18.03.

@Ericson2314: do you mean that you want the musl-support patches to keep getting cherry-picked to the stable 18.03 branch already? EDIT: 18.03 is very close to (belated) branching ATM.

@vcunat I guess I didn't think that far; correct me if I'm wrong @dtzWill, but I thought stream of musl fixes and slowed to a trickle :).

All I was thinking is if we're going to go through with this, it would be a cool thing to publicize (somewhat, is it is technically an experiment). If we end up going ahead but only decide after after 18.03, we miss a good window to give the experiment more visibility.

Until we have a "nixpkgs team" to make a decision on this, I propose to delegate this decision to the release team: @vcunat and @fpletz

Merged this because it reflects the current situation.

It seems the 1.1.19 -> 1.1.20 upgrade broke GCC for me: NixOS/nixpkgs#46064 (comment)