Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to the NixOS Hardened Profile #73763

Merged
merged 2 commits into from Apr 3, 2020

Conversation

kmcopper
Copy link
Contributor

Motivation for this change

Use scudo allocator for the hardened profile by default. Includes changes to compiler-rt to make it more compatible by default with things like man see #65000 for some details. Turning these two options off by default will prevent issues in programs from crashing the malloc, which is critical if we are replacing it as the system malloc. These options are all able to be overridden by simple environment variables even when compiled in so this should be a very safe change. I enabled the allocator zeroing by default in the hardened profile only. The buildSandbox option is currently incompatible with the hardened profile so it should be defaulted to off unless you manually re-enable both it and user namespaces.

I tested in a VM using the master branch. I used the hardened profile for the guest. I used both standard llvmPackages.compiler_rt and llvmPackages_9.compiler_rt for testing and both work. I mostly tested these options affects on programs by using man but I've also noticed needing these options as an environment variable for qemu to function well.

I feel like this should be backported to 19.09 so that the hardened profile can take advantage of it as was intended in #65000 for 19.09 (albeit with the graphene_malloc allocator). I feel the clang allocator has a much more stable history and should be the mkDefault here. I was unsure if and how to make the environment variable depend on the scudo allocator being the default but I do feel like it's safe to set in to profile anyway since it can be overridden easily.

I wasn't sure if maintainers wanted me to apply the compatibility all the way back to 4 which does still have scudo. I included 8 because I'm not sure what is going to land as the default for 20.03

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • [] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @joachifm @lovek323 @dtzWill @7c6f434c

Disables the build sandbox by default to avoid incompatibility with
defaulting user namespaces to false. Ideally there would be some kind of
linux kernel feature that allows us to trust nix-daemon builders to
allow both nix sandbox builds and disabling untrusted naemspaces at the
same time.
@joachifm
Copy link
Contributor

Hi, this LGTM

I think it's inappropriate to backport using scudo by default to 19.09. Non-essential changes that have some chance of producing observable side-effects violate the implied stable "contract", as I understand it.

If the other changes are deemed safe, we can backport those, however.

The compiler-rt changes should go via staging, due to the darwin mass rebuild.

@kmcopper
Copy link
Contributor Author

Depends on #73766 now as per comments.

@kmcopper
Copy link
Contributor Author

kmcopper commented Apr 1, 2020

I was hoping to get this merged and in 20.03 is it too late now to get this into that?

@joachifm
Copy link
Contributor

joachifm commented Apr 1, 2020

Sorry I completely forgot about this

@Mic92
Copy link
Member

Mic92 commented Apr 2, 2020

If @joachifm is too busy, maybe @emilazy or @yegortimoshenko can have a look.

@kmcopper kmcopper changed the title Improvements to the NixOS Hardened Profile [20.03] Improvements to the NixOS Hardened Profile Apr 2, 2020
@kmcopper kmcopper changed the title [20.03] Improvements to the NixOS Hardened Profile Improvements to the NixOS Hardened Profile Apr 2, 2020
@joachifm joachifm merged commit 18b89e7 into NixOS:master Apr 3, 2020
@joachifm
Copy link
Contributor

joachifm commented Apr 3, 2020

Thank you. Please open PRs against the release branch for backports.

@flokli
Copy link
Contributor

flokli commented Apr 3, 2020

Urgh - disabling the nix sandbox by default, just because Nix doesn't play well with disabled user namespaces sounds like a very scary hack to me. Not really a fan of 00ac71a TBH.

@joachifm
Copy link
Contributor

joachifm commented Apr 3, 2020

Please elaborate.

@flokli
Copy link
Contributor

flokli commented Apr 3, 2020

I consider disabling the Nix sandbox by default dangerous and a huge impurity. Also, see NixOS/nix#3006 (review) why disabling the sandbox alltogether isn't really nice. With this PR merged, shouldn't Nix degrade gracefully?

@flokli
Copy link
Contributor

flokli commented Apr 3, 2020

I need to admit, I don't exactly know how and when Nix can enable (some) sandboxing these times, but disabling it alltogether sounds like a bad idea.

@emilazy
Copy link
Member

emilazy commented Apr 3, 2020

I agree, it would be far preferable to merge the kernel patch to allow disabling unprivileged userns creation than it is for the hardened profile to reduce security by silently disabling build sandboxing entirely.

@dasJ
Copy link
Member

dasJ commented Apr 4, 2020

Just found this, I think it would have been better to allow user namespaces rather than disabling sandboxing.
Ever since we switched to the hardened profile, we use boot.kernel.sysctl."user.max_user_namespaces" = 512; which sounds a lot less frightening than disabling a sandbox.

@joachifm
Copy link
Contributor

joachifm commented Apr 4, 2020

Alright, it seems prevailing consensus is to revert that commit.

IMO, allowing arbitrary code execution on a machine is anathema to "hardening" to begin with. I consider build sandboxing to be mainly a development aide. I'm unconvinced that disabling the sandbox makes you materially worse off than you already are by handing out access to a C toolchain and all the tools you'd ever need for exploiting bugs in the host kernel. If you're serious about the security of your deployment, do not allow it to run build jobs.

@emilazy
Copy link
Member

emilazy commented Apr 4, 2020

I run the hardened profile on all my NixOS machines and do development on them; I'm sure I'm not alone. The defence in depth hardening features are more useful when running untrusted code, not less. Meanwhile, there's no good reason to use unsandboxed builds at all unless you're in a platform that doesn't support them.

flokli added a commit that referenced this pull request Apr 5, 2020
As discussed in #73763, prevailing
consensus is to revert that commit. People use the hardened profile on
machines and run nix builds, and there's no good reason to use
unsandboxed builds at all unless you're in a platform that doesn't
support them.

This reverts commit 00ac71a.
@flokli
Copy link
Contributor

flokli commented Apr 5, 2020

Reverted that commit in a8989b3.

@emilazy
Copy link
Member

emilazy commented Apr 5, 2020

For what it's worth, though I support using a hardened allocator, this currently makes Firefox crash on startup. This doesn't seem to be scudo specific – I get the same results with jemalloc too – but it's unfortunately challenging to work around (seemingly requires a new mount namespace).

It's well-documented that Firefox uses its own jemalloc allocator which can't be overridden without additional effort (see e.g. the Whonix wiki), but a hard SIGSEGV at startup (with the firefox package; the binary packages start the crash reporter) seems somewhat unfortunate. Will continue poking, but for now, anyone wanting to use Firefox with the hardened profile can revert to:

environment.memoryAllocator.provider = "libc";

@kmcopper kmcopper deleted the hardening-profile branch April 7, 2020 19:39
@Flakebi
Copy link
Member

Flakebi commented Apr 16, 2020

Thanks for working on hardening NixOS @kmcopper and @emilazy.
As I was struck by segfaults in the last few days and it took a while to find this issue, I want to add this here:
The freshly enabled allocator leads to segfaults in firefox (as you mentioned), thunderbird, rustc (yeah, all the mozilla related tools) and some background services that get started by evolution.

The easiest way to get a segfault that I found is running rustc --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg /dev/null (in case it is useful for someone).

@kmcopper
Copy link
Contributor Author

kmcopper commented Apr 19, 2020

@Flakebi Does environment.variables.SCUDO_OPTIONS = "" or environment.variables.SCUDO_OPTIONS = "ZeroContents=0"

Change the behavior or does it remain? I'm beginning to think made an additional way to scope the preload especially if we end up switching to graphene in the future as it's much less compatible with various applications than scudo is and I already needed to apply some compatibility fixes to the clang builds in nixpkgs.

For the record I am more pro graphene malloc then I am pro scudo malloc. It's more that I found the performance and hardening trade off to me more reasonable than the graphene malloc on it's own so I decided to go with that one since the small compatibility issues with man and some other stuff were easily resolved when I changed the LLVM stuff to default to non-aggressive faulting on certain kinds of program issues.

@Flakebi
Copy link
Member

Flakebi commented Apr 19, 2020

Setting these variables does not change anything.

env SCUDO_OPTIONS= rustc --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=s
ysroot --print=cfg /dev/null

-> Segfault

env SCUDO_OPTIONS=ZeroContents=0 rustc --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=s
ysroot --print=cfg /dev/null

-> Segfault

@kmcopper
Copy link
Contributor Author

Have you tested at all with environment.memoryAllocator.provider = "graphene-hardened"; on rust as well?

@Flakebi
Copy link
Member

Flakebi commented Apr 19, 2020

With environment.memoryAllocator.provider = "graphene-hardened";

  • rustc works,
  • thunderbird works,
  • firefox does not work,
  • nix stopped working:
$ nixos-rebuild test
building Nix...
building the system configuration...
error: out of memory

@emilazy
Copy link
Member

emilazy commented Apr 19, 2020

IIRC the Firefox segfault is because it passes something to scudo free that was allocated with jemalloc or vice versa; I don't think there's going to be an easier fix than implementing the replace-malloc API in a separate library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants