@shlevy, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @grahamc and @domenkozar to be potential reviewers.

@TikhonJelvis FYI

This looks great. I'd be happy to help with the nixpkgs-darwin-unstable channel.

Why do we need a separate channel for Darwin?

nixpkgs-unstable doesn't really test things that matter to Darwin users and this paves the way towards more formal darwin-nixpkgs releases (like we have on NixOS). Is there a better way to do that? Ideally we'd have some sort of cross-platform task force for nixpkgs releases, but I think most Linux nixpkgs users are already on NixOS...

It would be better to add this to the nixpkgs-unstable channel, unless we expect this job to be broken for extended periods of time. That way, people who need both Linux and OS X builds (e.g. in Hydra jobsets) can depend on a single Nixpkgs tree.

The OS X builds should not be included in the NixOS jobsets, however.

Are you assuming that Linux is generally going to be less broken than Darwin, so it's unlikely for Linux stuff to hold up Darwin channel releases?

Edit: I'm not saying that's an unreasonable assumption 😄

I'm curious though: if we decided we wanted to have a release of nixpkgs that we assert works well for Darwin users (and are willing to maintain), what would that look like?

I agree with @edolstra, let's make these part of nixpkgs-unstable and have us macOS users watchdog those packages.

My experience with using the current nixpkgs-unstable on OS X is that it breaks more often than the Linux/NixOS packages and that OS X issues are fixed on a different time scale than Linux/NixOS issues. (This includes both OS X-specific patches on our end and upstream issues.)

I think there are enough platform-specific issues that separating the two channels will let each one fix platform-specific issues quickly, without being blocked on the problems specific to the other platform.

In particular, packages I depended on were broken for extended periods of time after being included in nixpkgs-unstable. I would be happy if nixpkgs-unstable only included packages that at least built correctly on OS X, but that might slow the channel down as a whole.

@domenkozar @edolstra I think it's more about disentangling the two concerns, as @TikhonJelvis suggests:

1. We could add several more jobs to nixpkgs-unstable, but then if something on the Darwin side breaks, it holds up Linux channel release
2. We fundamentally (and for the foreseeable future) have a much smaller work force on Darwin than we do on Linux, so fixing those things is almost guaranteed to be slower
3. So if we stick more jobs into nixpkgs-unstable, we'll likely have more Linux users breathing down our necks about broken builds, which is good for awareness but if we have a fundamental limit on person-hours to contribute, the channel will still be slowed down by macOS's jobs
4. macOS users still want a reasonably well tested channel where they can expect stuff to work. I doubt they care about it being super recently up to date (so e.g., if the last time all jobs passed was a month ago, that isn't the end of the world), but stuff should work.

To put it another way, Linux users always have NixOS releases they can pin to if they want stuff that works. macOS users have no such thing, so either we start making periodic nixpkgs releases for macOS, or we have a rolling release with lots of jobs, specifically geared at macOS. Does that make sense? I don't know how else to give Mac users a good experience here, but I'd love suggestions.

@copumpkin @edolstra I think we have to be very careful fragmenting our community. I am obviously biased here, someone who uses macOS and Linux for many different environments nowadays.

I'm very much for having nixpkgs (macOS/Linux) channel with lots of tests that stop the update on failures.

Since we already have NixOS stable channels that get security fixes, I would add macOS support for them. I think we have enough infrastructure to get that going, including enough macOS users.

TL;DR I'm for not splitting up channels.

It's not about fragmenting the community though, right? The two "fragments" would be working on the same code in the same branch, but optimizing for different concerns. You personally might be in favor of blocking nixpkgs-unstable for two months due to macOS failures, but I can guarantee that a fair number of Linux users here who track that channel will be annoyed at macOS holding them up, and understandably so, because there's no fundamental reason that a Linux user shouldn't get a newer version of a package just because that new version introduced a slight incompatibility with Darwin that we haven't had time to patch yet.

All that having two separate channels means is that the notion of "in a good state" is different across different subcommunities within the project, all working on the same codebase. It's the same reason we have nixos-unstable vs. nixpkgs-unstable: nixos-unstable adds additional VM tests and such that don't really mean much to non-NixOS users but that the NixOS community cares about. Contributors to both still work on the same codebase with a shared goal of improving nixpkgs.

All this Darwin channel would do is add another independent pointer into the branch history of our master branch. All code/fixes for both channels would still go into the usual staging/master branches, so there's no meaningful fracturing of code effort, just a meaningful separation of user experience.

@copumpkin that's very helpful.

I see where the motivation comes from now, but I wonder if separating channels is still the right cure.

If a commit breaks Linux OR macOS channel update, it should just be reverted.

Having two channels will encourage breaking other platform, since it won't be blocking your platform.

I wonder if following is not good enough:

a) add more tests/packages required for channel to pass
b) using CI, pindown and revert commits that break a channel update

Yeah, in principle I think that makes sense, and I'm definitely not sure that a channel is the best solution.

My fear is just the practice of it, given the way we work on the project. Often someone just wants to quickly bump a version of some tool/library, tests it locally on their own platform, and pushes it up. If I had a reasonable proposal to let a Linux user test on Darwin, I'd feel better about being "strict" about this sort of thing, but here's the scenario I fear:

1. Linux user A pushes random update to package P
2. Mac user B uses P on a daily basis and points out that the new version of P is now broken on Darwin
3. A tries some speculative fixes, trying to coax Hydra to build a new version properly on Darwin, but with feedback loops of several hours, it's pretty frustrating and doesn't get very far.
4. B does not have the skill to fix P (on average this is true; we have comparatively few actual maintainers of nixpkgs on Darwin), but is now sad that the package no longer works.

Now have two options:

1. We revert the update A made, which is sad for the community as a whole (and A will probably be frustrated after wasting time on it), but fixes B for now. Perhaps the revert could include a comment warning the next person to come along against updating without testing on Darwin. But even then, it's fairly rare for a Linux user to have access to a Darwin box to test on, so the comment might ask you to test on both systems but it's hard to.
2. We leave the update A made in place but tell B to pin nixpkgs against an older version that still works. A and B will now be on different versions but will both have working software.

TBC, I don't really like either solution, but #2 feels less disruptive to users, and gives them an out. There's nothing stopping people who want or need to ensure that they have consistent versions on both Linux and Darwin from using the same channel on both, but now there's a "I'm just a semi-casual nixpkgs user on Darwin and really just want my packages to work reliably" option for Darwin folks as well. It's not quite a nixpkgs-YEAR.MONTH, but it's a step in that direction, I think.

Edit: I do think there's a nonzero (although not terribly high) chance that we could someday provide a somewhat functional lightweight Darwin build testing environment for Linux users, but until that day I don't think I can expect Linux folks to test on Darwin. The other way around is usually slightly easier, but still a pain and we haven't automated any of that stuff yet. I do think that's far more likely to become reality than the other direction.

2. being "less disruptive to users" is very subjective given that:

a) it encourages for user A not to care about user B, resulting into the culture of "it works for me on Linux"
b) moreover, it pushes macOS users into full-on catch-up game. Given that we have lots of Linux users, it's a lot of work
c) you can pin nixpkgs to a specific commit now already and I encourage anyone to do it
d) you would essentially adding more and more packages to block the darwin channel, but you will never be able to please everyone except for requiring all darwin packages to pass (which in turn would really need a better CI for pull request testing and extremely good team of playing the catch-up game)

I think your arguments are good @copumpkin, I'm just trying to see what issues will come up going this path. Sometimes a solution to a problem brings up more problems on the long run.

I guess the question we should ask ourselves is:

• can we define a minimal set of packages that are required to work
• can we make sure if a change breaks, we track who/what broke it can act in timely matter

a) it encourages for user A not to care about user B, resulting into the culture of "it works for me on Linux"

Except it's questionable whether user A should care about user B, or even if it's possible to. "it works for me on Linux" is inevitable given our current technologies, and I don't even think it would be right for us to encourage our developers in general to think otherwise right now

c) you can pin nixpkgs to a specific commit now already and I encourage anyone to do it

Yes, but you can't get automatic catch-up and testing unless you run your own hydra

d) you would essentially adding more and more packages to block the darwin channel, but you will never be able to please everyone except for requiring all darwin packages to pass

I don't think this is true. After all, we don't have all linux packages blocking the unstable channel. And darwin users are by and large very reasonable and understanding that darwin support is by necessity a second-class citizen.

can we define a minimal set of packages that are required to work

I think so. The number of darwin users is small enough and reasonable enough that we can mostly just follow user requests. We don't have to solve the problem once and for all up front.

can we make sure if a change breaks, we track who/what broke it can act in timely matter

Realistically this is up to the ##nix-darwin community to manage, at least until it's easy for linux devs to test things out on darwin.

Except it's questionable whether user A should care about user B, or even if it's possible to. "it works for me on Linux" is inevitable given our current technologies, and I don't even think it would be right for us to encourage our developers in general to think otherwise right now

Right now they can really just merge code to master and wait for Hydra to report successful or unsuccessful Darwin build.

In future we will hopefully have a CI that can test PRs for all platforms. Then comes the decision if something breaks macOS if it should be merged.

I don't think this is true. After all, we don't have all linux packages blocking the unstable channel. And darwin users are by and large very reasonable and understanding that darwin support is by necessity a second-class citizen.

Exactly, but a separate channel would only depend on jobs passing or not. Otherwise it's the same as nixpkgs-unstable.

Ideally we would have a better way to communicate when channel is blocked, given the amount of macOS users these days, I don't think it should be an issue.

Even better would be if we could parse Hydra tested job, to see how many times darwin was the blocker and for how long (and vice-versa).

Right now they can really just merge code to master and wait for Hydra to report successful or unsuccessful Darwin build.

In future we will hopefully have a CI that can test PRs for all platforms. Then comes the decision if something breaks macOS if it should be merged.

This is definitely not enough. It's way too long of a feedback loop and too hard to test changes (you can't e.g. just run a single command and see what it does)

Exactly, but a separate channel would only depend on jobs passing or not. Otherwise it's the same as nixpkgs-unstable.

Right, but my point is that the darwin users will be reasonable about what they put in there.

Anyway, if we're actually OK with putting these jobs into nixpkgs unstable and maintaining that promise to darwin users then the immediate need is solved, but I suspect it will just cause frustration and delays for linux users at little benefit above a separtae darwin channel.

Some questions:

• Should Nix default to the nixpkgs-darwin-unstable channel on OS X?
• Should nixpkgs-unstable include Darwin builds?
• Should nixpkgs-darwin-unstable include Linux builds?
• If I have a Hydra jobset that I want to test on both Linux and OS X (like Nix or patchelf), should I now include both nixpkgs and nixpkgs-darwin as jobset inputs?

Should Nix default to the nixpkgs-darwin-unstable channel on OS X?

I think so, yeah.

Should nixpkgs-unstable include Darwin builds?
Should nixpkgs-darwin-unstable include Linux builds?

I can see these going either way. Probably a good start would be to say no to both (except the tarball in nixpkgs-darwin-unstable but if nixpkgs-darwin-unstable (which will almost certainly lag behind nixpkgs-unstable) turns out to regularly be super broken on linux we may want to change this (so e.g. groups can peg the same nixpkgs rev on darwin and linux based on the channel)

If I have a Hydra jobset that I want to test on both Linux and OS X (like Nix or patchelf), should I now include both nixpkgs and nixpkgs-darwin as jobset inputs?

I think the answer to this is the same as whether you test against both unstable and 17.03. But I'm not completely sure.

@copumpkin thoughts?