Preview: deterministic build #2281

alexanderkjeldaas · 2014-04-15T14:02:10Z

This is a set of changes that makes the system_tarball_pc derivation deterministic.

Stdenv bootstrap:

The stdenvs created during bootstrap are numbered to make it easier to debug/understand
BUG: for the bootstrap stdenvs, binutils was after the bootstrap binaries in path.
Deterministic archives are enabled early during bootstrapping of stdenvs.

Stdenv builder:

The generic builder depends on libfaketime. libfaketime needs a home. libfaketime is currently distributed through github, that that does not work during bootstrap when SSL is not available. It should be moved to tarballs.nixos.org.
Feature: Setting useFakeTime fixes the time during builds. This can break builds, but is easy to manage compared to patching. Additional environment variables are available for configuring libfaketime.
Feature: A fake "date" utility is prepended to the path
Check: The build directory (/tmp/nix-build-foo-x.y.z) is not allowed to appear in artifacts ($out) to avoid non-deterministic output.

Gcc:

Gcc by default now defines DATE and TIME to be (time_t)0.
TODO: -frandom-seed is not set. This is relevant for c++ code.
PGO is turned off for gcc. Note: there is no consensus on doing this as it reduces compilation performance.

Various changes:

The xsltproc utility creates random identifiers. A post-processing stage using perl was added to the nixos manual. This could be generalized.
All uses of 'gzip -9' has been replaced with 'gzip -9n'.
Added options to cpio for default uid/gid, and default mtime for tar.
File lists are sorted before being added to cpio/tar.

alexanderkjeldaas · 2014-04-15T14:06:40Z

Known issues:

The inode changes done in perl seems to be buggy (boot problems), but they might not be needed.
Some patches are not required because of the gcc-wrapper and libfaketime features.
I'm not sure the overridden 'date' binary works correctly.

shlevy · 2014-04-16T13:45:43Z

I think nix uses the equivalent of (time_t) 1 for its file a/m/ctime changes

domenkozar · 2014-04-19T11:29:44Z

Wonderful. @alexanderkjeldaas that means most of other PRs can be closed?

thoughtpolice · 2014-06-06T14:27:42Z

@vcunat @alexanderkjeldaas Can we get some of these things merged on the pending stdenv branch? I know that we haven't solved the GCC question re: PGO, so we'll have to leave that commit out, but a lot of these changes are not very intrusive, and merging them would reduce burdens later and get us much closer to a deterministic build.

vcunat · 2014-06-09T17:40:51Z

Ah, I completely forgot this series of work. The currently staged stdenv has been quite tested by me, so I would merge it about the current state (after Hydra verifies gcc on Darwin). I'll look at this afterwards, as I wanted to do another iteration of stdenv stuff (there were some others I missed this time).

alexanderkjeldaas · 2014-06-09T19:12:27Z

@vcunat I'll just leave it as-is then I guess.

On Mon, Jun 9, 2014 at 7:40 PM, Vladimír Čunát notifications@github.com
wrote:

Ah, I completely forgot this series of work. The currently staged stdenv
has been quite tested by me, so I would merge it about the current state
(after Hydra verifies gcc on Darwin). I'll look at this afterwards, as I
wanted to do another iteration of stdenv stuff (there were some others I
missed this time).

—
Reply to this email directly or view it on GitHub
#2281 (comment).

thoughtpolice · 2014-06-29T01:47:11Z

@alexanderkjeldaas I'm going to begin merging some of this work into HEAD soon. I'm probably not going to merge everything in one go, so feel free to rebase this when you get a chance. I'll update with what I've pushed upstream.

alexanderkjeldaas · 2014-06-30T20:47:29Z

I've rebased

On Sun, Jun 29, 2014 at 3:47 AM, Austin Seipp notifications@github.com
wrote:

@alexanderkjeldaas https://github.com/alexanderkjeldaas I'm going to
begin merging some of this work into HEAD soon. I'm probably not going to
merge everything in one go, so feel free to rebase this when you get a
chance. I'll update with what I've pushed upstream.

—
Reply to this email directly or view it on GitHub
#2281 (comment).

alexanderkjeldaas · 2014-07-01T21:29:56Z

I've added a minor fix for python 2.7.7 that I forgot to cherry-pick from my internal branch.

7c6f434c · 2014-08-30T10:13:56Z

So, what is the status of cherry-picking? Github doesn't easily show this, unfortunately…

Obviously, this will never get directly in master (only in staging) and it gets chery-picked in small pieces.

I actually support reproducibility, although some people seem to like PGO too much…

vcunat · 2014-08-30T10:46:07Z

PGO is probably the only questionable thing here, IIRC. I'm planning to really review and test this within the next 10 days.

Remove datetime from nscd.

This includes two changes: 1) Fix a bug where the bootstrap-tools is always used instead of binutils 2) Enable strip --enable-deterministic-archives as soon as a new binutils is available.

1) Make the core python libraries deterministic. 2) Make the python libraries created by glib deterministic.

joachifm · 2016-12-19T22:25:48Z

Note that 3157dbe is probably no longer required due to tytso/e2fsprogs@a2143b5

joachifm · 2016-12-19T22:29:42Z

I've experimented with patching gzip -9 -> gzip -9n, but it doesn't seem to matter (to nix-build --check, anyway). Could this be because we now fix timestamps after unpacking sources? Or are there still reasons for doing it, even with SOURCE_DATE_EPOCH?

joachifm · 2016-12-20T12:32:16Z

Hm, I guess gzip -n still makes sense if the thing being compressed was created as part of the build.

joachifm · 2016-12-20T15:06:12Z

I've been going through most of these. A brief summary so far:

I think the following are obsolete

smartmontools passes nix-build --check & the string that is patched out
in fd1101a
no longer occurs in the source.
glibc currently contains a patch that removes datetime from nscd
the openldap patch is no longer relevant, I think, the build no longer skips
elf patching & the build passes --check
groff and opensp pass --check on my end
the improvement to the linux stdenv are at least partially covered (strip is called with --deterministic-archives)

The following need more work

syslinux fails --check, but the patch in this PR is insufficient
as reported previously, the manual is still indeterministic

I've not looked at libgpg-error, busybox, python, perl, or the gcc stuff, nor the general libfaketime support.

joachifm · 2016-12-20T15:18:27Z

Regarding the manual, https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain#libxslt indicates that the issue of random ids is fixed/worked on upstream, so perhaps we want to just skip that for now.

cleverca22 · 2016-12-20T19:03:00Z

something that can help to stress-test determinism, http://manpages.ubuntu.com/manpages/xenial/man1/disorderfs.1.html

this is a fuse filesystem that randomizes the order of files in a directory, so you cant accidentally rely on the fs doing that most of the time

joachifm · 2016-12-21T02:24:25Z

Some of the perl stuff was lost when perl16 was removed, but otherwise it seems like @vcunat (or whomever) picked up most of the specific package fixes. I think we're left with gcc, faketime in stdenv, stdenv numbering, and the fake date command thing, all of which seem like they could be profitably dealt with on their own.

vcunat · 2016-12-21T10:46:39Z

IIRC I had tried hard to pick whatever I could verify/make clearly advantegous. For some issues (like PGO) I didn't succeed in a reasonable amount of time, so I left those behind.

cbarrett · 2018-04-12T03:51:27Z

FYI this PR is being linked to from https://reproducible-builds.org/who/. I volunteer to ping whoever's necessary to get an update posted (don't have the knowledge for more, unfortunately).

Ekleog · 2018-09-26T17:52:31Z

(triage) My reading of the comments in this thread is that most of the changes either have been merged or are no longer needed. The remaining changes that would require being split out to separate PRs would be:

GCC stuff (should be done, see Make __DATE__/__TIME__ deterministic when NIX_ENFORCE_PURITY=1 #51510)
Faking the time in stdenv
Numbering the names of stdenv
Fake date command in stdenv

Does that sound correct to those actually involved?

rht · 2018-12-04T17:24:24Z

Note: the GCC stuff (force-setting __DATE__ and __TIME__) in #2281 (comment) has been superseded by SOURCE_DATE_EPOCH, which is already set in stdenv, and also that the feature for checking for SOURCE_DATE_POCH has been patched to the early gcc versions as well (4.9.x, 5.x, 6.x).

rht · 2018-12-04T19:44:29Z

The fake date command is no longer necessary, where faketime by itself is sufficient, so I suppose the remaining points are 1. numbering the names of stdenv, and 2. using faketime in stdenv.

mmahut · 2019-08-12T11:38:02Z

Are there any updates on this pull request, please?

worldofpeace · 2019-09-20T19:44:14Z

This is not actionable in its current state. Please open issues with references to the changes within this PR if needed.

davidak · 2019-10-14T17:28:38Z

There is a GitHub project for this topic: https://github.com/NixOS/nixpkgs/projects/10

@worldofpeace can you add this and every related issue (even closed) to it?

Then https://reproducible-builds.org/who/ can link to that.

The current status is visible on https://r13y.com/ (at least for nixos-unstable's iso_minimal).

vcunat · 2019-10-14T18:22:26Z

Perhaps it might make more sense to have one ticket for each (important) irreproducible package? (or "unreproducible"? whatever) At least for those that someone started to investigate and didn't immediately manage to fix it completely.

worldofpeace · 2019-10-14T19:03:43Z

Sure, it makes sense to have issues in the tracker for this purpose. Though I'm not sure if I'll have the time to distill them from this thread. @vcunat that is an important point for those who did take time to investigate that

vcunat added the enhancement label Jun 9, 2014

vcunat added the stdenv label Jun 9, 2014

alexanderkjeldaas added 8 commits September 2, 2014 12:19

Make glibc compilation more pure.

1472989

Remove datetime from nscd.

Make perl 5.16 binary deterministic.

2410fa1

Make the linux bootstrap environments more deterministic.

61c55ad

This includes two changes: 1) Fix a bug where the bootstrap-tools is always used instead of binutils 2) Enable strip --enable-deterministic-archives as soon as a new binutils is available.

Improve python library determinism.

0d6467c

1) Make the core python libraries deterministic. 2) Make the python libraries created by glib deterministic.

Improve determinism for libgcrypt, libgpg-error, and busybox.

84a9791

Make openssl deterministic.

7f014c7

Make groff and opensp deterministic.

30a92c4

Make __DATE__/__TIME__ deterministic when NIX_ENFORCE_PURITY=1

3dba999

This was referenced Dec 18, 2016

Deterministic initramfs #21273

Closed

stdenv setup: disallow references to the build directory #21287

Closed

joachifm mentioned this pull request Dec 20, 2016

libfaketime: make the build reproducible #21302

Closed

joachifm mentioned this pull request Dec 20, 2016

Remove uses of gzip that might capture build time #21306

Merged

rht mentioned this pull request Jan 15, 2017

fetchBower: fold arg in all-packages into the nix file #21906

Closed

7 tasks

FRidh mentioned this pull request Feb 8, 2017

Python: determinism of the interpreters and bytecode (pyc) files #22570

Closed

jgeerds requested a review from edolstra May 18, 2017 22:12

veprbl added the 2.status: merge conflict label Nov 28, 2018

rht mentioned this pull request Dec 4, 2018

Make __DATE__/__TIME__ deterministic when NIX_ENFORCE_PURITY=1 #51510

Closed

10 tasks

rht mentioned this pull request Dec 4, 2018

stdenv: Add a fake date utility together with setup #51519

Closed

10 tasks

worldofpeace closed this Sep 20, 2019

vcunat added this to others in Deterministic builds Oct 14, 2019

FRidh mentioned this pull request Jan 3, 2020

perlPackages.TimeDate: fix tests that fail as of 2020 #76810

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preview: deterministic build #2281

Preview: deterministic build #2281

alexanderkjeldaas commented Apr 15, 2014

alexanderkjeldaas commented Apr 15, 2014

shlevy commented Apr 16, 2014

domenkozar commented Apr 19, 2014

thoughtpolice commented Jun 6, 2014

vcunat commented Jun 9, 2014

alexanderkjeldaas commented Jun 9, 2014

thoughtpolice commented Jun 29, 2014

alexanderkjeldaas commented Jun 30, 2014

alexanderkjeldaas commented Jul 1, 2014

7c6f434c commented Aug 30, 2014

vcunat commented Aug 30, 2014

joachifm commented Dec 19, 2016

joachifm commented Dec 19, 2016

joachifm commented Dec 20, 2016

joachifm commented Dec 20, 2016

joachifm commented Dec 20, 2016

cleverca22 commented Dec 20, 2016 •

edited

joachifm commented Dec 21, 2016

vcunat commented Dec 21, 2016 •

edited

cbarrett commented Apr 12, 2018

Ekleog commented Sep 26, 2018 •

edited

rht commented Dec 4, 2018

rht commented Dec 4, 2018

mmahut commented Aug 12, 2019

worldofpeace commented Sep 20, 2019

davidak commented Oct 14, 2019

vcunat commented Oct 14, 2019

worldofpeace commented Oct 14, 2019

Preview: deterministic build #2281

Preview: deterministic build #2281

Conversation

alexanderkjeldaas commented Apr 15, 2014

alexanderkjeldaas commented Apr 15, 2014

shlevy commented Apr 16, 2014

domenkozar commented Apr 19, 2014

thoughtpolice commented Jun 6, 2014

vcunat commented Jun 9, 2014

alexanderkjeldaas commented Jun 9, 2014

thoughtpolice commented Jun 29, 2014

alexanderkjeldaas commented Jun 30, 2014

alexanderkjeldaas commented Jul 1, 2014

7c6f434c commented Aug 30, 2014

vcunat commented Aug 30, 2014

joachifm commented Dec 19, 2016

joachifm commented Dec 19, 2016

joachifm commented Dec 20, 2016

joachifm commented Dec 20, 2016

joachifm commented Dec 20, 2016

cleverca22 commented Dec 20, 2016 • edited

joachifm commented Dec 21, 2016

vcunat commented Dec 21, 2016 • edited

cbarrett commented Apr 12, 2018

Ekleog commented Sep 26, 2018 • edited

rht commented Dec 4, 2018

rht commented Dec 4, 2018

mmahut commented Aug 12, 2019

worldofpeace commented Sep 20, 2019

davidak commented Oct 14, 2019

vcunat commented Oct 14, 2019

worldofpeace commented Oct 14, 2019

cleverca22 commented Dec 20, 2016 •

edited

vcunat commented Dec 21, 2016 •

edited

Ekleog commented Sep 26, 2018 •

edited