I figure those are the .dist jobs, right?

Assuming one doesn't trust your tarballs, they could use those instead, right?

The next step might be to somewhat automatize this step of taking the latest bootstrap tarballs? Any issues with the idea? I'm thinking a maintainer script that could be run manually to produce the update and a list of tarballs to download from hydra, with appropriate names.

I downloaded them from the individual jobs (bootstrapFiles.busybox and bootstrapFiles.bootstrapTools) because I didn't know about the dist job. I will update the URLs to download directly from Hydra.

I updated the URLs to point to Hydra. The busybox sha256 does not match if I use the value obtained from Hydra or manual sha256sum. I fixed the armv7l one by TOFU but the others are still incorrect.

gcc in the bootstrap tarball is nonfunctional because it is missing libssp.

This seems to get a little further, but fails on xz:

libtool: link: gcc -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wlogical-op -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -g -O2 -o .libs/xzdec xzdec-xzdec.o xzdec-tuklib_progname.o xzdec-tuklib_exit.o  ../../src/liblzma/.libs/liblzma.so -Wl,-rpath -Wl,/nix/store/njjhdzw71igfdy9zibyij8nv8nqgapac-xz-5.2.4/lib
/nix/store/xhkjbgb64krykh5mij8zryvswaayr9j8-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_sigmask'
/nix/store/xhkjbgb64krykh5mij8zryvswaayr9j8-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_join'
/nix/store/xhkjbgb64krykh5mij8zryvswaayr9j8-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_create'

The next step might be to somewhat automatize this step [...]

In any case I believe updates of bootstrap tarball (or busybox) should be rather rare events, hopefully years apart. Right now the various failures and fixups seem to need much more work than the automatizable parts.

For me it failed building GCC. I ran into the the same error as on i686 in #66528 (comment). I am currently testing to see if disabling LTO works around the problem.

I believe updates of bootstrap tarball (or busybox) should be rather rare events

Yes, I agree, hence "somewhat" automatize. My main thought here is that each time it happens it's years apart, people come and go, people forget what needs to happen to do this properly.

For that it's enough to write some docs/howto :-)

I can't reproduce the xz failure. I was able to build armv7l stdenv except for a failing allowedRequisites check because expand-response-params and zlib are getting libssp.so from bootstrap-tools.

I'm testing this again, but there's a bit being left unsaid in this thread so I'm not sure I'm testing the right thing. Check my working and see where I'm going wrong.

First off, this branch doesn't evaluate as is due to infinite recursion.

$ git fetch origin pull/79793/head && git checkout FETCH_HEAD
[...]
$ nix-build pkgs/top-level/release-cross.nix -A bootstrapTools.armv7l.dist      
error: infinite recursion encountered, at undefined position
(use '--show-trace' to show detailed location information)

So I'm testing against the merge, which as of writing is:

$ git fetch origin pull/79793/merge && git checkout FETCH_HEAD
[...]
HEAD is now at 16c7ca5dd10 Merge 1f1b31c05afe88b45519d93a46b0bc9eafb5e490 into 816e214c1838d7e7265cdf1ffa361bfac852b38d

Rebuild the tools:

$ nix-build pkgs/top-level/release-cross.nix -A bootstrapTools.armv7l.dist
/nix/store/n3l34vzl1hdi5a7vs4rii4qgfh98zi8x-stdenv-bootstrap-tools-armv7l-unknown-linux-gnueabihf

I then plug the results into pkgs/stdenv/linux/bootstrap-files/armv7l.nix and nix-build --argstr system armv7l-linux -A hello. This builds for a while, then stops after

/nix/store/lmss9iwplnwcngv4xvhp4nsl9lkwh70n-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_sigmask'
/nix/store/lmss9iwplnwcngv4xvhp4nsl9lkwh70n-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_join'
/nix/store/lmss9iwplnwcngv4xvhp4nsl9lkwh70n-bootstrap-tools/bin/ld: ../../src/liblzma/.libs/liblzma.so: undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:501: xzdec] Error 1
make[3]: Leaving directory '/build/xz-5.2.4/src/xzdec'
make[2]: *** [Makefile:420: all-recursive] Error 1
make[2]: Leaving directory '/build/xz-5.2.4/src'
make[1]: *** [Makefile:613: all-recursive] Error 1
make[1]: Leaving directory '/build/xz-5.2.4'
make: *** [Makefile:480: all] Error 2
builder for '/nix/store/93qgcgmsyk5lidq8xx37488677ics6sj-xz-5.2.4.drv' failed with exit code 2

To reduce error I made a quick script to glue the two halves together.

My testing was done by applying 7e7bbcb on top of ac241fb (the latest commit that Hydra had finished building bootstrap tools for at the time I started testing) in order to build the bootstrap tools.

I was planning to split 7e7bbcbc29bc689c58a8430da3529ad5c70fa0d0 into its own PR and then let Hydra build a new version of the bootstrap tools. Based on your error, it looks like now some other change has broken the bootstrap tools again.

I've successfully built hello using these branches. The changes seem sufficient. 👍

Now that the bootstrap tools have glibc 2.30, LTO works again but will probably break with the next glibc update. I'm not sure if I should keep it disabled, or wait and see if it becomes a problem next time glibc is updated.

Also, I have tested building armv6l stdenv on an armv7l machine (using lopsided98@001be94, which is how I normally build for armv6l).

I have not tested armv5tel, but this PR can't make it any more broken than it currently is.

PR #79391 for glibc 2.31 is there already, if you want to try now.

stdenv builds successfully with glibc 2.31 and LTO enabled, so I removed the commit that disables it on ARM.

@FRidh This should not have been merged.

The problem part is the bootstrap URLs, right? If so, we can follow-up with an update.

Yes, this shouldn't break anything any more than it already is.

Just so everything is clear, these are the required next steps:

1. Merge make-bootstrap-tools: add libssp to allow stack protector to work #80497
2. Wait for Hydra to build new bootstrap tarballs
3. Upload new bootstrap tarballs to tarball.nixos.org
4. Create a new PR that updates nixpkgs to use the new tarballs