coreutils: use openssl for faster *sum tools #44937
Conversation
sha256sum, md5sum, etc.
Suggested by matthewbauer review.
In Darwin there is a complain: output '/nix/store/2806zdbv3zdxpy8aslijdml84vrpqf39-stdenv-darwin' is not allowed to refer to the following paths: /nix/store/4q1c84f1ynfvsd85a932375sg0jc89v8-bootstrap-tools /nix/store/6r6xsb18p7cp32b2yhf5v46n0d50ng2s-openssl-1.0.2o
I updated stdenv derivations of linux and darwin to allow it.
It worked for me in parallel, but I didn't mean to change it in this branch. Worth trying to enable it if upstream fixed anything.
GrahamcOfBorg
added
6.topic: stdenv
|
Success on x86_64-linux (full log) Attempted: coreutils Partial log (click to expand)
|
|
Success on aarch64-linux (full log) Attempted: coreutils Partial log (click to expand)
|
|
Success on x86_64-linux (full log) Attempted: coreutils Partial log (click to expand)
|
|
Success on aarch64-linux (full log) Attempted: coreutils Partial log (click to expand)
|
|
I assume someone then needs to build a new bootstrap tarball? |
|
New bootstrap tarballs are built by Hydra, see https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.stdenvBootstrapTools.x86_64-linux.dist and https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.stdenvBootstrapTools.x86_64-linux.test. You could give those a try (e.g. see if you can rebuild stdenv using them). If they work we can upload them to tarballs.nixos.org. |
|
Why is a new bootstrap tarball required? I don't think it is required. |
|
Another option may be to use a static openssl to build coreutils - I guess that will keep the stdenv closure size smaller. |
|
I'm sorry to say this, but I don't think it's strategic to make openssl a stdenv-rebuilding package. It's relatively common that it contains a security bug that we want to fix very fast across all channels, and having to rebuild completely everything might be a significant hindrance. Therefore I'd give a thought to options where the default coreutils in stdenv does not have this support, though there are complications with this as well – packages will tend to keep reference to that one (unless we put the Impact of that slight bloat could be significantly reduced by making |
|
So what about my proposal about making coreutils use a *static* openssl?
It's only for the hashing algorithms.
…On Mon, Aug 27, 2018 at 10:34:49PM -0700, Vladimír Čunát wrote:
I'm sorry to say this, but I don't think it's strategic to make openssl a stdenv-rebuilding package. It's relatively common that it contains a security bug that we want to fix very fast across all channels, and having to rebuild completely everything would be a significant hindrance.
Therefore I'd give a thought to options where the default coreutils in stdenv does not have this support, though there are complications with this as well – packages will tend to keep reference to that one (unless we put the `coreutils-with-openssl` into its build inputs), so it will be hard to avoid having both versions in system closure. Impact of that could be worked around by making `coreutils-with-openssl` mostly from symlinks to the version from stdenv (most of size is locale, for example), though that makes the expression a little more complex.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#44937 (comment)
--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key 7CBD1DA5 - https://emailselfdefense.fsf.org/
|
|
Change in linking doesn't prevent rebuilds (by itself) – it would work if we split away |
|
Ah you mean the rebuilds.
Right, an openssl-for-coreutils would do.
The speed increase in the hash programs is worth it, I think.
…On Tue, Aug 28, 2018 at 12:02:12AM -0700, Vladimír Čunát wrote:
Change in linking doesn't prevent rebuilds – it would work if we split away `openssl-for-coreutils` and didn't update that version too often.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#44937 (comment)
--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key 7CBD1DA5 - https://emailselfdefense.fsf.org/
|
|
OK, that also sounds a nice way. I didn't mean to suggest to give up on the speed increase. The checksumming tools seem rather unlikely to be used much during build, so that's why I first considered that approach. For multiple tools it seems that the optimal configuration for builds and "other" usage differs ("other" = typically system envs, interactive usage, etc.) |
|
BTW, I believe we don't need to immediately propagate this change to the currently used bootstrap tools. (Though we should make sure they still build and test OK, and the Hydra job checks that.) |
|
I agree on the bootstrap tools.
What if we give up on changing coreutils, and instead we add a
"coreutils-with-openssl" for use in NixOS? NixOS have that instead of the
plain coreutils.
…On Tue, Aug 28, 2018 at 12:49:46AM -0700, Vladimír Čunát wrote:
BTW, I believe we don't need to immediately propagate this change to the currently used bootstrap tools. (Though we should make sure they still build and test OK, and the Hydra job checks that.)
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#44937 (comment)
--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key 7CBD1DA5 - https://emailselfdefense.fsf.org/
|
|
I thought we might keep the one in stdenv without openssl but explicit |
|
It sounds great. Do you volunteer for that? :)
…On Tue, Aug 28, 2018 at 01:05:39AM -0700, Vladimír Čunát wrote:
I thought we might keep the one in stdenv without openssl but explicit `coreutils` reference with openssl. I expect that would mitigate almost all the rebuilds. (And perhaps create alias `stdenv.coreutils` for explicit references wanting to avoid that.)
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#44937 (comment)
--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key 7CBD1DA5 - https://emailselfdefense.fsf.org/
|
|
I have computing power for testing. It's worse with time, but I might be able to catch it if there are no complications with it. I'll certainly write here when/before I put a noticeable amount of time into it; feel free (anyone) to do that first, as there are many other nix* things I want to do, especially with release not far off. |
|
Well, the planned part was easy (a few minutes)... but it turned out that keeping stdenv without openssl isn't enough to keep the rebuild amount due to openssl sane – probably due to some spurious |
|
Security updates and to reduce the difference of packages in the darwin stdenv vs linux is exactly why I spent a bunch of time on getting rid of openssl and libarchive in the darwin stdenv. While adding it to the stdenv build is not a major problem, I prefer the |
|
I wanted stdenv with minimal coreutils and default |
|
I propose this: #45720 |
sha256sum, md5sum,. run much faster. Factor 2x to 4x.
You can evaluate that by comparing the cpu time of your 'time sha256sum bigfile' and 'time openssl sha256 bigfile'.
I tried this PR in #43983 not noticing that it altered stdenv.
I modified stdenv allowedRequisites so it is allowed in linux & darwin stdenv; @edolstra said that openssl for coreutils does not rise the stdenv size too much.
The other solution is to add a new coreutils-with-openssl that is referenced in NixOS but not in stdenv, so the user always calls the fast *sum versions.