[RFC 0085] NixOS Release Stabilization: ZHF on master, new timeline #85
Conversation
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/20-09-release-retrospective/9851/31 |
|
Added defintions section, and improved timeline clarity. Updated new render |
|
|
||
| Normal staging development is disrupted for two weeks. (1 normal iteration) | ||
| Changes to "critical packages" may have to wait for a month to be merged into staging. | ||
| Master development will be uninterrupted, assuming all changes don't introduce new regressions. |
There was a problem hiding this comment.
I think this assumption is too optimistic. However it's IMHO not such a big deal if regressions don't stem from critical packages (such as the ones you've listed above).
There was a problem hiding this comment.
This is mostly from my observation that the week prior to the 20.09 scheduled release date saw ZHF PRs slow down significantly. The "easy to fix" packages, usually have quick turn around, and the more involved package fixes were specific to the 20.09 branch; so non of the more active contributors really wanted to "pick up" the work.
There was a problem hiding this comment.
Also, in practice, the branches diverge quite a bit if a particular package (or package set) had a series of updates from branch-off. It was sometimes not obvious what series of changes allowed for unstable to work, but the beta branch to still be broken.
|
|
||
| Actual release schedule may adjust dates slightly due to unforeseen events. | ||
|
|
||
| Release team will retain the right to refine what constitutes "critical packages", as this may |
There was a problem hiding this comment.
What does "Release team" mean? I know that a few people (including you IIRC) plan to "define" such a team rather than having two RMs with most of the workload, but I'm not sure if it's clear who's part of this at the moment.
There was a problem hiding this comment.
Currently it means the release managers.
There was a problem hiding this comment.
I would like to eventually create a "release team", as many people have expressed interest in being a part of a release; but it's much bigger of a ask to be "the release manager".
| - "Critical packages" | ||
| - Important packages which have many dimensions of build or runtime behavior. | ||
| - Generally these will be packages which are a part of `stdenv`, or bootstrapping a system (e.g. systemd). | ||
| - Initially these will be: `stdenv.cc`,`binutils`, and `systemd` for this RFC. |
There was a problem hiding this comment.
(nit) I guess at least glibc should be listed here as well.
There was a problem hiding this comment.
- Generally these will be packages which are a part of
stdenv, or bootstrapping a system (e.g. systemd).
Separate building/bootstrapping stdenv from that of a (NixOS) system? I'm of the opinion all stdenv deps, including stdenv itself should freeze at this point.
There was a problem hiding this comment.
It was implied that this would include everything in stdenv.cc, but I could make it more explicit.
nix-repl> stdenv.cc.libc.name
"glibc-2.32-25"
nix-repl> stdenv.cc.bintools.name
"binutils-wrapper-2.35.1"
EDIT: wording
| [definitions]: #definition | ||
|
|
||
| - "Freezing" a branch or package. | ||
| - Disallow breaking changes to branch or package. In SemVer terminology, disallow major version bumps. |
There was a problem hiding this comment.
I'm wondering if we should define an exception here: breaking changes can be allowed if those are necessary to fix security issues or fix a critical bug (i.e. something that corrupts your data or crashes your system).
Note: while backporting single patches is the preferred way in that situation, those can be "breaking" as well.
There was a problem hiding this comment.
Yea, and that's the exception I make for the release channel. But only if patching isn't a reasonable alternative.
|
|
||
| - "Freezing" a branch or package. | ||
| - Disallow breaking changes to branch or package. In SemVer terminology, disallow major version bumps. | ||
| - A release branch can be thought of being permanently "frozen" in this regard. Stable may be a better term, but it's already overloaded with a previous release branch or channel. |
There was a problem hiding this comment.
I'm not sure if we should define such a detail here: from my observation this isn't the status quo and IMHO also a fairly debatable, but out-of-scope topic.
There was a problem hiding this comment.
This was more meant to help convey that freezing only applied to breaking changes. No one expects the stable channel to receive 0 updates, but people do expect the stable channel to rarely cause upgrade issues.
|
I have historically been skeptical of slowing down |
I agree with this in general, but there's also issues like what we encounter with plasma which will be really hard to construct. This was a big factor for me to push DM packaging so far from the release date, we really do need the time on unstable for those scenarios to be stabilized. |
|
I nominate myself to shepherd. |
|
I nominate myself to shepherd as well. |
|
Just a heads-up on the timeline, in a few weeks gnome should be releasing the gnome-40 rc, which would be the beginning of this schedule. The only "risk" would be systemd 248 or gcc 11 (+binutils) being released during the stabilization period. However, those PRs usually take several weeks to months to complete; so unless either package gets a new release in the next week or two, the stabilization needed for those updates to just target master would likely go past the current branch-off date. The biggest change would be that they couldn't be crammed in right before branch-off. The real ask is delay the critical updates for 4 weeks (likely not to be a concern for this 21.05 release), and staging-targeted breaking changes for 2 weeks. Either way, with all the issues that we have had with previous releases, I think we should attempt this new more "aggressive" schedule. In the worst case scenario, we may have to delay the release a week to backport a fix. In the best case scenario, most people won't notice a change to unstable workflow. |
|
I nominate myself to shepherd as well. |
|
@ryantm do you want to be shepherd leader? |
|
I was worried when I saw this RFC, because as an unstable user I don't
like the idea of unstable having to slow down to accomodate stable. But
having read it, it seems like the accomodations being asked for are very
small and reasonable, so I don't really see a problem with it, as long
as the set of frozen packages stays very small and focused. The set
described here in the RFC is good, and some change at the RMs'
discretion is entirely reasonable.
I do have a couple of questions though:
First (and maybe this is very naive of me), why can't we just revert
problematic updates after branch-off, like the problematic systemd bump
from last time? This seems to me like it would allow master to continue
at full steam ahead, while allowing the release managers to go ahead
without any recently introduced problematic changes.
Updates to `staging` which will land before branch-off will be
restricted to non-breaking updates
It's not clear to me what this means. Is a git bump (for example) a
breaking update or a non-breaking update? Maybe this could be added to
the terminology section.
|
As someone who also does not use the releases, I would assume that reverting on-branch means that some ZHF fixes have to be different on the release branch compared to the mainline, and so you cannot run a unified ZHF sprint without a mess of some things happenning only on one branch or something like that. |
It can be very hard to capture all the changes that are related to a given bump.
"Disallow breaking changes to branch or package. In SemVer terminology, disallow major version bumps." |
Sure, I can be leader. |
|
> It's not clear to me what this means. Is a git bump (for example) a
breaking update or a non-breaking update? Maybe this could be added to
the terminology section.
"Disallow breaking changes to branch or package. In SemVer terminology, disallow major version bumps."
If the change is `git: 2.30.0 -> 2.30.1` or `git: 2.31.0`, then it's fine. Just worried about
`git: 2.30 -> 3.0.0`, in which there's likely to be some breaking changes which
would need a longer stabilization time to full fix everything.
Maybe "breaking change" should be a top-level "definitions" entry rather
than defined under "frozen". :)
One more thing: the RFC should probably define where the canonical
home for the list of frozen packages will be, since it won't be changed
by RFC. I'd vote for the manual.
|
|
I am not entirely sold on the idea yet. If it solves the stable problems I am all for it. If it works in practice remains to be tested. The one change that should probably be made to this RFC is remove any listing of packages outside of the Another worry I have is: Communication |
The best term would have been stable, but as I also mentioned, that term is already overloaded in the nixpkgs as it generally refers to the current release branch, not the upcoming release. For ZHF, I will probably use the phrase If someone can think of a better term, I will gladly change it.
Agreed, lets test it.
I disagree. Sure, not every "critical" package may be applicable to a release. But I would also like to avoid the situation of, "some people got in a room together and came up with this list without significant community involvement". I think a more appropriate time to revisit the list would be the release retrospective. This at least allows the community to participate, and major stakeholders can express their opinions. I tried to do one for 20.09, which this RFC was one of the action items. Hopefully the next release will have more participation, and I still want to thank those who showed up. I could also argue that creating an RFC to facilitate changes to the release process is only to involve the community more. However, just saying, "I think this would be better" without any input isn't a healthy way to do process changes. And this particular change should have community involvement since I would like to alter some merge criteria on unstable leading up to the release.
I would like to avoid RFCs as much as possible. The process is slow (for good reason), and takes a fair amount of time and energy for the RFC committee to process. I think a retrospective would be a better format as the pain of the release would still likely be fresh, and certain packages would be easier to identify as being problematic. The list of critical packages can just be maintained in the release wiki.
I mention in future work that I will be creating a discourse thread. And will likely ping someone to pin it for a while. I was probably also going to do a reddit post and discord announcement just referring to the discourse thread as well, but since reddit or discord isn't an official platform, I didn't mention them. But NixOS communication problems aren't unique to this RFC, and slightly off-topic. For enforce-ability, I will be quite active (and hopefully the other RM, still TBD) and able to enforce it for most PRs. Most staging-targeted PRs take a while to review anyway, so I'm not too concerned that something will be merged while I was sleeping or away. I mentioned here that it's still somewhat of a desire to have a "release team", which would help distribute these responsibilities and own certain release issues. |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/21-05-call-for-release-manager/10204/16 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/21-05-call-for-release-manager-and-release-members/10204/17 |
| staging. In addition, I also propose to stabilize the release on master | ||
| instead of the release branch, thus avoiding the need to backport every pull | ||
| request, and take advantage of the stabilization that unstable normally | ||
| receives. |
There was a problem hiding this comment.
This should also describe what this means for new packages/modules.
There was a problem hiding this comment.
New packages are welcome. Changes to modules are welcome, assuming it doesn't cause regressions, and the module changes will probably need to have documentation so it's not additional work trying to curate the release notes.
I will modify the definitions section to say that "freezing" will still allow new package and module additions.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
| | Weeks from Release | Branches Affected | Events | | ||
| | --- | --- | --- | | ||
| | -8 Weeks | | Gnome and Plasma(YY.11) packaging begins | | ||
| | -6 Weeks | `staging-next`, `staging` | Restrict breaking changes to Release Critical Packages | |
There was a problem hiding this comment.
"Restrict breaking changes to Release Critical Packages" is phrased ambiguously can be parsed in to opposing ways:
- "restrict X to Y" in the sense of "limit application of X, so that it's only applied to Y", i.e., exclude everything from X except Y
⇒ Only Release Critical Packages may receive breaking changes. (Other packages must not.) - "changes to Y" in the sense of "changes that are made to Y"
⇒ Release Critical Packages must not receive breaking changes. (Other packages may.)
The phrasing should be changed so that it's clear which one is intended (probably the latter). Note that the definition of adjective/participle "restricted" above only halfway helps here, because the current phrasing uses the verb "(to) restrict" instead.
There was a problem hiding this comment.
Since we defined Restricted above, I think we can just say:
| | -6 Weeks | `staging-next`, `staging` | Restrict breaking changes to Release Critical Packages | | |
| | -6 Weeks | `staging-next`, `staging` | Release Critical Packages are Restricted | |
For clarification, this has no implication on master. The changes to master should be able to be captured for each PR, and we can effectively move "green to green". These restrictions are more in regard to Also breakages introduced through staging PRs will likely need a staging PR to fix them, which makes the iteration for correcting such breakages much longer than PRs which target |
|
The FCP comment period has ended. There has been some discussion and changes but I believe they are immaterial to the original intent/spirit of the RFC, so I do not think it is necessary for @garbas and @Mic92 to vote again on accepting the RFC. We offer this RFC to @NixOS/rfc-steering-committee for acceptance and merging! |
|
Awesome, I'll do some announcements :) |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
|
Rendered link is 404 |
Updated now that it has been merged |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/21-05-has-been-released/13407/1 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/what-should-stable-nixos-prioritize/9646/81 |
|
The nixpkgs manual should be updated in light of this. https://nixos.org/manual/nixpkgs/stable/#submitting-changes-staging-branch and https://nixos.org/manual/nixpkgs/stable/#submitting-changes-staging-next-branch do not indicate any freezes relating to the release cycle. |
…ixOS#85) Co-authored-by: Eelco Dolstra <edolstra@gmail.com> Co-authored-by: Ryan Mulligan <ryan@ryantm.com>
In similar spirit to #80, this RFC tries to change the nature in which we conduct the stabilization of a NixOS release. Without Desktop managers being a major risk (due to #80), the release should be free to focus on stabilizing package builds. This should enable us to perform ZHF mostly on master with minor adjustments to staging and a change to policies regarding certain "critical" or "high-risk" packages.
Overall, the goal of this RFC to reduce the pain associated with ZHF (release stabilization) at a slight expense to the staging branch workflow during the stabilization window.
I eluded to this RFC in #80 (comment)
"Freeze" may be a little strong, as non-breaking changes are allowed. But I didn't want to use a term like "half-frozen" or "semifrozen""Restricted" has been chosen as the term to describe disallowing breaking change updates.Related discussions:
- https://discourse.nixos.org/t/20-09-release-retrospective/9851
- https://discourse.nixos.org/t/what-should-stable-nixos-prioritize/9646
cc @worldofpeace @FRidh @ryantm
rendered: https://github.com/NixOS/rfcs/blob/master/rfcs/0085-nixos-release-stablization.md