New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vmTools.debClosureGenerator: Fix non-determinism in dependency graph #107958
Conversation
By default, Perl versions since 5.8.1 use randomization to make hashes resistant to complexity attacks. That randomization makes building VM images such as ubuntu1804x86_64 non-deterministic because the (imported) derivations built by deb/deb-closure.pl are not stable. This can easily be observed by repeating the following sequence of commands and noting the path of the image's .drv: nix-instantiate -E '(import <nixpkgs> {}).vmTools.diskImageFuns.ubuntu1804x86_64 {}' nix-store --delete /nix/store/*ubuntu-18.04-bionic-amd64.nix One source of non-determinism is the handling of Provides/Replaces, which depends on the order of iteration over %packages. Here is a diff showing the corresponding change in output: >>> awk -virtual awk: using original-awk - original-awk: libc6 (>= 2.14) +virtual awk: using mawk + mawk: libc6 (>= 2.14) - mawk: libc6 (>= 2.14) ->>> libc6 This patch sorts packages by name for Provides/Replaces processing, which seems to result in stable output. (If the above turns out not to be sufficient, one could also set the PERL_HASH_SEED and PERL_PERTURB_KEYS environment variables, documented in 'perlrun', to disable Perl's built-in randomization. Complexity attacks are not an issue as we control and trust all inputs.)
I've run those commands 3 times and the output is deterministic for me. Am I missing something? |
@SuperSandro2000: Well, yes; you're missing my nice demo :) More seriously: I have tried in multiple (Linux-based) environments, and have never seen the same I just ran it 10 times; here is what I am seeing (NixOS, sandboxed):
I also see the same kind of thing on a RHEL7 box (single-user installation). |
After running it like 10 times I finally got a hash mismatch. |
Interesting. No idea how to explain the relative stability in your environment. |
Hi @SuperSandro2000, What do you think would be the next steps for this PR? Would you have specific reviewers to suggest? Thanks, -D |
Great; thanks! |
Motivation for this change
By default, Perl versions since 5.8.1 use randomization to make hashes resistant to complexity attacks.
That randomization makes building VM images such as
ubuntu1804x86_64
non-deterministic because the (imported) derivations built bydeb/deb-closure.pl
are not stable.This can easily be observed by repeating the following sequence of commands and noting the path of the image's
.drv
file:One source of non-determinism is the handling of
Provides
/Replaces
, which depends on the order of iteration over%packages
. Here is a diff showing the corresponding change in output:This patch sorts packages by name for
Provides
/Replaces
processing, which seems to result in stable output.(If the above turns out not to be sufficient, one could also set the
PERL_HASH_SEED
andPERL_PERTURB_KEYS
environment variables, documented inperlrun
, to disable Perl's built-in randomization. Complexity attacks are not an issue as we control and trust all inputs.)Things done
sandbox
innix.conf
on non-NixOS linux)nixos/tests/os-prober.nix
nixos/tests/virtualbox.nix
nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)nix path-info -S
before and after)