@GrahamcOfBorg test installer.simple installer.simpleSpecialized installer.simpleUefiGrub installer.simpleUefiGrubSpecialisation installer.separateBoot installer.separateBootFat installer.zfsroot installer.lvm installer.luksroot installer.encryptedFSWithKeyfile installer.swraid installer.simpleLabels installer.simpleProvided installer.btrfsSimple

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite. So for that reason I'm inclined that we close this PR and have an actual solution, because we can't continue to revert it on stable releases. That is actually why it only happened on the release branch last time.

If somebody has a fix for it, or knows how to get to one, that would be better of course. But this is a hardware problem that's hard to reproduce and causes really bad consequences for the ones who encounter them. If the boot loader fails to run, those people are essentially locked out of their system, no way to even roll back, since that depends on the boot loader being functional.

@infinisil I meant the issue from last time is a revert was done, but during the period of the revert no proactive action was taken. Part of that is, as you said, it's hardware specific.

Someone GRUB knowledgeable needs to somehow forward a bug to them https://www.gnu.org/software/grub/manual/grub/html_node/Reporting-bugs.html https://savannah.gnu.org/bugs/?group=grub. Which is hard when it seems to be so specific.

Is there other alternatives than we can do aside from a full revert #61718 (comment)

Though I think stateVersion is less reliable because people mess with it.

@infinisil I intentionally reverted it only on 19.09 so that the update would remain on master and could be fixed later on.

Idea: this could be a little sillier than we thought. I've often had issues with my ESP being corrupted simply by rebooting too quickly (or too brutally :/ ) after a nixos-rebuild thanks to FAT's lack of journalling. Maybe it's actually a case of that?

Idea: this could be a little sillier than we thought. I've often had issues with my ESP being corrupted simply by rebooting too quickly (or too brutally :/ ) after a nixos-rebuild thanks to FAT's lack of journalling. Maybe it's actually a case of that?

tbh, sometimes I have brutal issues that involve nixos-rebuild as well 😁

@lheckemann I don't think so, I didn't reboot until ~20 minutes after the upgrade...

It's not just NixOS either:

• https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1848797
• https://forum.manjaro.org/t/error-symbol-grub-file-filters-not-found-problem/103148
• https://unix.stackexchange.com/questions/531659/symbol-grub-file-filters-not-found-on-kali-linux-vm
• etc.

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite.

@worldofpeace why can't we stick to 2.02 indefinitely? (or until the bug is resolved upstream)

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite.

@worldofpeace why can't we stick to 2.02 indefinitely? (or until the bug is resolved upstream)

Because that's just a terrible practice to do in any infrastructure. Just not use something indefinitely because it has a bug. I hope that doesn't come off too negative 😹 but I see this to be true.

as a side note: grub 2.02 doesn't support reading from f2fs. Anyone who installed nixos with f2fs as / would no longer be able to boot.

I believe @disassembler and I addressed in the go/no-go meeting that we can't do a revert because it will be indefinite.

@worldofpeace why can't we stick to 2.02 indefinitely? (or until the bug is resolved upstream)

Because that's just a terrible practice to do in any infrastructure. Just not use something indefinitely because it has a bug. I hope that doesn't come off too negative 😹 but I see this to be true.

Well I would rephrase it as "not use something until it has a bug".

The issue seems to be with upstream, so as you correctly point out, it should be resolved there. But until that's done, should we use the buggy version?

All that aside, thanks so much for your work on 20.03! ❤️

as a side note: grub 2.02 doesn't support reading from f2fs. Anyone who installed nixos with f2fs as / would no longer be able to boot.

Yeah, that's problematic. Being that it's Grub, I don't see a downgrade being straightforward. We could patch it in.

My last 2 cents: AFAIK grub 2.04 is also needed to boot from a btrfs / that uses zstd compression.

My last 2 cents: AFAIK grub 2.04 is also needed to boot from a btrfs / that uses zstd compression.

Reviewing the News and changes https://git.savannah.gnu.org/cgit/grub.git/tree/NEWS#n1, we can quickly see that backwards compatibility isn't going to be a thing here. Lost features.

So reverting will probably break someone's setup, and not reverting may have someone jump into #61718. I'm not sure there's a happy medium here.

Chiming in to add that this issue is affecting one of my machines as well.

I understand why reverting the update on master is out of the question at this point, but would it be possible to do so just for 20.03, as was done for 19.09? I don't imagine it would break many people's setups then.

This comment links to a Debian bug report explaining the issue in detail.

It seems the underlying problem is not in upstream, but rather a misconfiguration on the user's machine.

Given that, I think the best thing to do is add an entry to the release notes pointing to instructions on how to fix this manually, rather than reverting.

I understand why reverting the update on master is out of the question at this point, but would it be possible to do so just for 20.03, as was done for 19.09? I don't imagine it would break many people's setups then.

As mentioned, 2.04 and 2.02 have different features, and people probably already depend on them since it's committed. So this would be a breaking change to commit to an already released version of NixOS.

This comment links to a Debian bug report explaining the issue in detail.

It seems the underlying problem is not in upstream, but rather a misconfiguration on the user's machine.

Given that, I think the best thing to do is add an entry to the release notes pointing to instructions on how to fix this manually, rather than reverting.

Woah yeah, that looks about right. In the go/no-go it was discussed to do some sort of errata, it's just the information wasn't in the issue to make it.

If this is a configuration issue maybe we can add some asserts to grub module to prevent it?

@misuzu what asserts would you add?

At least some of the misconfigurations involve settings in the firmware, which AFAIK the OS has no access to.

@misuzu what asserts would you add?

First step would be to gather info about systems which affected by this issue: boot configuration, partition layouts of all disks, mount points, etc. Maybe certain pattern is there and can be easily prevented. If not, at least there would be clear reproduction steps which can be documented.

It would definitely have been nice if the module had informed me that compress=zstd isn't supported on grub 2.02 - would have saved me some hours of stress!