Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducible setuptools #104009

Merged
merged 5 commits into from Nov 18, 2020
Merged

Conversation

basile-henry
Copy link
Contributor

Motivation for this change

Part of the efforts at https://r13y.com/ to get a reproducible iso_minimal

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Copy link
Member

@FRidh FRidh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Have you checked bootstrapped-pip is also reproducible?

pkgs/development/python-modules/setuptools/default.nix Outdated Show resolved Hide resolved
@basile-henry
Copy link
Contributor Author

basile-henry commented Nov 17, 2020

@FRidh nix-build -A python38Packages.bootstrapped-pip --check seems to work just fine 👍 Are there more checks you would like me to do?

Someone on IRC mentioned that only gnutar exposes the --mtime flag (and not bsdtar). Do I need to do something special on darwin?

@FRidh
Copy link
Member

FRidh commented Nov 17, 2020

@GrahamcOfBorg build python3.pkgs.setuptools

@r-burns
Copy link
Contributor

r-burns commented Nov 17, 2020

I think the tar command is fine, bootstrapped-pip and setuptools are building and passing --check for me on darwin :)

@basile-henry
Copy link
Contributor Author

@r-burns I think you might need to run nix-build -A pythonPackages.setuptools.src --check. Otherwise it might get that from the cache (I think). It was the derivation that makes up setuptools's src that was previously not reproducible, not setuptools itself (I should have specified that better in the PR description 😅 ).

@FRidh
Copy link
Member

FRidh commented Nov 17, 2020

@basile-henry by changing the expression the derivation is changed, so there is no issue with cache using here. Anyway, I will wait for ofborg on darwin for confirmation.

@basile-henry
Copy link
Contributor Author

so there is no issue with cache using here

You are right for the first build. What I am unsure about is how deep --check goes. I was under the impression it only checks 1 derivation deep, but maybe that's not the case? If so, how does it know how deep to check?

@FRidh
Copy link
Member

FRidh commented Nov 17, 2020

The src attribute is not a fixed-output derivation. --check does not recurse.

@basile-henry
Copy link
Contributor Author

Building it once on darwin is indeed sufficient to check if the tar command works. Thanks for checking that 👍

I just wanted to check for myself if --check recursed or not and as you said it does not:

➜ cat test.nix
let pkgs = import <nixpkgs> {};

    not-deterministic = pkgs.stdenv.mkDerivation {
      name = "date";
      buildPhase = ''
        mkdir $out
        date > $out/date
      '';
      phases = [ "buildPhase" ];
    };
in
pkgs.stdenv.mkDerivation {
  name = "test";
  src = not-deterministic;
  buildPhase = ''
    mkdir $out
    cp date $out/date
  '';
  phases = [ "unpackPhase" "buildPhase" ];
}

➜ nix-build test.nix
/nix/store/sf7p5x1wr070vkg668dlndlfjhl83mbk-test

➜ nix-build test.nix --check
checking outputs of '/nix/store/53ryaaxhd4xd8bs4jh3rm3i0dbxf9fdf-test.drv'...
unpacking sources
unpacking source archive /nix/store/q3dnl180w8b7pc8lm4j8k978jr872n8p-date
source root is date
building
/nix/store/sf7p5x1wr070vkg668dlndlfjhl83mbk-test

➜ cat $(nix-build test.nix)/date
Tue Nov 17 10:17:21 UTC 2020


# Here we untar the sdist and retar it in order to control the timestamps
# of all the files included
tar -xzf dist/${pname}-${version}.post0.tar.gz -C dist/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the filename always contain post0 or can that change with an update?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the name of the release tag post0 is deterministic now (it used to contain the date). The patch of setup.cfg ensures that.

It is possible that an update of the source changes that (if the build system upstream is updated), but I would assume that the tag-date.patch would also need to be updated then. If/when that happens this buildPhase should fail loudly, so I don't think hardcoding it is an issue.

If you have any suggestion to make this better, do tell! 😄

Copy link
Member

@prusnak prusnak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can replace the hardcoded value of '1970-01-01' with "@$SOURCE_DATE_EPOCH"

pkgs/development/python-modules/setuptools/44.0.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/setuptools/default.nix Outdated Show resolved Hide resolved
Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
@FRidh FRidh merged commit 548db36 into NixOS:staging Nov 18, 2020
@basile-henry basile-henry deleted the basile-henry/r13y-setuptools branch November 18, 2020 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants