Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/lib/utils: Add fileSystems.<name>.depends option and generalise fsBefore (fixes #86955) #86967

Merged
merged 2 commits into from Jun 14, 2021

Conversation

jakobrs
Copy link
Contributor

@jakobrs jakobrs commented May 5, 2020

Motivation for this change

See #86955 for an example configuration where this matters.

What's new

The PR adds a fileSystems.<name>.depends option that can be used to specify what other mount points need to be mounted before this one, and makes fsBefore recognise more forms of dependencies "out of the box", to fix #86955.

Note: The following three comments of mine were created before a force push and no longer apply.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@jakobrs jakobrs changed the title nixos/lib/utils: Generalise fsBefore nixos/lib/utils: Generalise fsBefore (in response to #86955) May 5, 2020
@jakobrs

This comment has been minimized.

@jakobrs jakobrs changed the title nixos/lib/utils: Generalise fsBefore (in response to #86955) nixos/lib/utils: Generalise fsBefore (fixes #86955) May 10, 2020
@jakobrs

This comment has been minimized.

@jakobrs

This comment has been minimized.

@jakobrs jakobrs force-pushed the more-general-fsbefore branch 2 times, most recently from 6cbb64e to 8ccc574 Compare May 23, 2020 10:25
@jakobrs jakobrs changed the title nixos/lib/utils: Generalise fsBefore (fixes #86955) nixos/lib/utils: Add fileSystems.<name>.depends option and generalise fsBefore (fixes #86955) Jun 13, 2020
@stale
Copy link

stale bot commented Jan 19, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md and removed 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md labels Jan 19, 2021
Copy link
Contributor

@nrdxp nrdxp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am successfully using this to work around an issue when combining impermanence and agenix

It's working beautifully so I believe it is ready for merging

Copy link
Member

@dotlambda dotlambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we normalise at all?
The mountPoint already requires there to be no trailing slash and I see no reason why depends shouldn't do the same.
Btw, mountPoint and depends could also use types.path.

@jakobrs
Copy link
Contributor Author

jakobrs commented May 27, 2021

Well when I originally made this #95406 wasn't even created yet. I'm not entirely convinced the depends option should require there be no trailing slash, but I've implemented this and force-pushed (I also rebased against master). However, I still think we should normalise the paths, mainly because it's more obviously correct (and less likely to break). We still need to add a slash at the end of all the paths, with normalisePath we just get the guarantee that we won't add one too many. And the device option still needs to be normalised (it can't require no slash at the end since it's not guaranteed to be a path).

Comment on lines +27 to +28
nonEmptyWithoutTrailingSlash = addCheckDesc "non-empty without trailing slash" types.str
(s: isNonEmpty s && (builtins.match ".+/" s) == null);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for these not to start with a /?

Suggested change
nonEmptyWithoutTrailingSlash = addCheckDesc "non-empty without trailing slash" types.str
(s: isNonEmpty s && (builtins.match ".+/" s) == null);
nonEmptyPathWithoutTrailingSlash = addCheckDesc "non-empty without trailing slash" types.path
(s: isNonEmpty s && (builtins.match ".+/" s) == null);

Note that this doesn't make much sense:

  • any path is non-empty
  • the path / has a trailing slash but is still allowed (even before this change)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the check used by #95406

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you did. I guess we can leave improving this to a different PR.

nixos/modules/tasks/filesystems.nix Outdated Show resolved Hide resolved
nixos/lib/utils.nix Outdated Show resolved Hide resolved
@jakobrs jakobrs force-pushed the more-general-fsbefore branch 2 times, most recently from bdf293e to 76b6a3c Compare May 29, 2021 12:08
Copy link
Member

@dotlambda dotlambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jakobrs Sorry for my confusing comments along the way!

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/384

@dotlambda dotlambda requested a review from infinisil June 1, 2021 20:25
Comment on lines 38 to 40
in hasPrefix a'.mountPoint b'.device
|| hasPrefix a'.mountPoint b'.mountPoint
|| any (dependency: hasPrefix a'.mountPoint dependency) b'.depends;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this! I just have a small stylistic suggestion, but other than that, I think this looks ready to be merged.

Suggested change
in hasPrefix a'.mountPoint b'.device
|| hasPrefix a'.mountPoint b'.mountPoint
|| any (dependency: hasPrefix a'.mountPoint dependency) b'.depends;
in
hasPrefix a'.mountPoint b'.device
|| hasPrefix a'.mountPoint b'.mountPoint
|| any (hasPrefix a'.mountPoint) b'.depends;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jakobrs It ended up in the wrong commit and is still missing the formatting. While the indentation means the hasPrefix lines don't line up as neatly, it shows more clearly on a quick glance what belongs to the expression, separating it from the let .. in.

@dotlambda dotlambda requested a review from talyz June 13, 2021 21:28
@talyz talyz merged commit 2670683 into NixOS:master Jun 14, 2021
@jakobrs jakobrs deleted the more-general-fsbefore branch June 14, 2021 09:25
@emmanuelrosa
Copy link
Contributor

The commit which modifies fsBefore is giving me a build error:

❯ nixos-rebuild build --show-trace
building Nix...
building the system configuration...
error: while evaluating the attribute 'config.system.build.toplevel' at /home/emmanuel/projects/nixpkgs/nixos/modules/system/activation/top-level.nix:293:5:
while evaluating 'foldr' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:52:20, called from /home/emmanuel/projects/nixpkgs/nixos/modules/system/activation/top-level.nix:128:12:
while evaluating 'fold'' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:55:15, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:59:8:
while evaluating the attribute 'assertions' at undefined position:
while evaluating 'g' at /home/emmanuel/projects/nixpkgs/lib/attrsets.nix:298:19, called from undefined position:
while evaluating anonymous function at /home/emmanuel/projects/nixpkgs/lib/modules.nix:140:72, called from /home/emmanuel/projects/nixpkgs/lib/attrsets.nix:301:20:
while evaluating the attribute 'value' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:525:9:
while evaluating the option `assertions':
while evaluating the attribute 'mergedValue' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:557:5:
while evaluating the attribute 'values' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:551:9:
while evaluating the attribute 'values' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:650:7:
while evaluating anonymous function at /home/emmanuel/projects/nixpkgs/lib/modules.nix:537:28, called from /home/emmanuel/projects/nixpkgs/lib/modules.nix:537:17:
while evaluating definitions from `/home/emmanuel/projects/nixpkgs/nixos/modules/tasks/filesystems/zfs.nix':
while evaluating 'dischargeProperties' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:609:25, called from /home/emmanuel/projects/nixpkgs/lib/modules.nix:538:137:
while evaluating the attribute 'condition' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:693:14:
while evaluating the attribute 'condition' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:693:14:
while evaluating the attribute 'enabled' at undefined position:
while evaluating 'g' at /home/emmanuel/projects/nixpkgs/lib/attrsets.nix:298:19, called from undefined position:
while evaluating anonymous function at /home/emmanuel/projects/nixpkgs/lib/modules.nix:140:72, called from /home/emmanuel/projects/nixpkgs/lib/attrsets.nix:301:20:
while evaluating the attribute 'value' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:525:9:
while evaluating the option `boot.zfs.enabled':
while evaluating the attribute 'mergedValue' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:557:5:
while evaluating the attribute 'values' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:551:9:
while evaluating anonymous function at /home/emmanuel/projects/nixpkgs/lib/modules.nix:547:19, called from /home/emmanuel/projects/nixpkgs/lib/modules.nix:547:14:
while evaluating the attribute 'value._type' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:648:73:
while evaluating the attribute 'value.content' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:708:14:
while evaluating the attribute 'default' at /home/emmanuel/projects/nixpkgs/nixos/modules/tasks/filesystems/zfs.nix:113:9:
while evaluating the attribute 'boot.initrd.supportedFilesystems' at undefined position:
while evaluating 'g' at /home/emmanuel/projects/nixpkgs/lib/attrsets.nix:298:19, called from undefined position:
while evaluating anonymous function at /home/emmanuel/projects/nixpkgs/lib/modules.nix:140:72, called from /home/emmanuel/projects/nixpkgs/lib/attrsets.nix:301:20:
while evaluating the attribute 'value' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:525:9:
while evaluating the option `boot.initrd.supportedFilesystems':
while evaluating the attribute 'mergedValue' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:557:5:
while evaluating the attribute 'values' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:551:9:
while evaluating the attribute 'values' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:650:7:
while evaluating anonymous function at /home/emmanuel/projects/nixpkgs/lib/modules.nix:537:28, called from /home/emmanuel/projects/nixpkgs/lib/modules.nix:537:17:
while evaluating definitions from `/home/emmanuel/projects/nixpkgs/nixos/modules/system/boot/stage-1.nix':
while evaluating 'dischargeProperties' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:609:25, called from /home/emmanuel/projects/nixpkgs/lib/modules.nix:538:137:
while evaluating 'dischargeProperties' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:609:25, called from /home/emmanuel/projects/nixpkgs/lib/modules.nix:615:11:
while evaluating the attribute 'content' at /home/emmanuel/projects/nixpkgs/lib/modules.nix:693:14:
while evaluating the attribute 'system.build.fileSystems' at /home/emmanuel/projects/nixpkgs/nixos/modules/tasks/filesystems.nix:241:5:
while evaluating 'toposort' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:454:22, called from /home/emmanuel/projects/nixpkgs/nixos/modules/tasks/filesystems.nix:14:18:
while evaluating 'toposort' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:454:22, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:457:18:
while evaluating 'toposort' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:454:22, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:457:18:
while evaluating 'toposort' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:454:22, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:457:18:
while evaluating 'listDfs' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:415:35, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:456:17:
while evaluating 'dfs'' at /home/emmanuel/projects/nixpkgs/lib/lists.nix:417:27, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:430:8:
while evaluating anonymous function at /home/emmanuel/projects/nixpkgs/lib/lists.nix:420:26, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:420:15:
while evaluating 'fsBefore' at /home/emmanuel/projects/nixpkgs/nixos/lib/utils.nix:17:17, called from /home/emmanuel/projects/nixpkgs/lib/lists.nix:420:29:
while evaluating 'hasPrefix' at /home/emmanuel/projects/nixpkgs/lib/strings.nix:218:5, called from /home/emmanuel/projects/nixpkgs/nixos/lib/utils.nix:38:8:
while evaluating the attribute 'device' at /home/emmanuel/projects/nixpkgs/nixos/lib/utils.nix:30:37:
while evaluating 'normalisePath' at /home/emmanuel/projects/nixpkgs/nixos/lib/utils.nix:29:23, called from /home/emmanuel/projects/nixpkgs/nixos/lib/utils.nix:30:46:
cannot coerce null to a string, at /home/emmanuel/projects/nixpkgs/nixos/lib/utils.nix:29:30

I suspect the problem is that some of my mounts have a null device. Here's an example:

 fileSystems."/home" =
    { label = "NIXOS";
      fsType = "btrfs";
      options = [ "compress=zstd" "noatime" "subvol=/nixos/home" ];
    };

The root filesystem is mounted using a device, but the filesystem (BTRFS) has the label "NIXOS". This label is then used to mount additional subvolumes.

@jakobrs
Copy link
Contributor Author

jakobrs commented Jun 18, 2021

@emmanuelrosa I guess we could just add toString to path to convert null to ""? or we could do this:

    let
      getDevice fs =
        if fs.device != null then
          fs.device
        else if fs.label != null then
          "/dev/disk/by-label/${fs.label}"
        else
          ""; # This will be caught when building the fstab, and we'd rather throw there

      normalisePath = path: "${path}${optionalString (!(hasSuffix "/" path)) "/"}";
      normalise = mount: mount // { device = normalisePath (getDevice mount);
                                    mountPoint = normalisePath mount.mountPoint;
                                    depends = map normalisePath mount.depends;
                                  };

This would have the advantage of having "more correct" semantics, although it's a little more complex.

@jakobrs
Copy link
Contributor Author

jakobrs commented Jun 18, 2021

Here is a PR: #127309

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants