Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libressl: add 3.1.3, default to it, remove 2.9 #88195

Merged
merged 6 commits into from Jun 19, 2020

Conversation

ruuda
Copy link
Contributor

@ruuda ruuda commented May 19, 2020

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

I also tested compiling Nginx against this LibreSSL, which succeeded.

@doronbehar
Copy link
Contributor

Let's see which of the builds will fail:

@GrahamcOfBorg build acme-client
@GrahamcOfBorg build foundationdb
@GrahamcOfBorg build foundationdb51
@GrahamcOfBorg build foundationdb52
@GrahamcOfBorg build foundationdb60
@GrahamcOfBorg build foundationdb61
@GrahamcOfBorg build libressl
@GrahamcOfBorg build libressl_3_1
@GrahamcOfBorg build linux-steam-integration
@GrahamcOfBorg build mydumper
@GrahamcOfBorg build netcat
@GrahamcOfBorg build openntpd
@GrahamcOfBorg build openntpd_nixos
@GrahamcOfBorg build opensmtpd
@GrahamcOfBorg build ponyc
@GrahamcOfBorg build pony-stable
@GrahamcOfBorg build pounce
@GrahamcOfBorg build python27Packages.foundationdb51
@GrahamcOfBorg build python27Packages.foundationdb52
@GrahamcOfBorg build python27Packages.foundationdb60
@GrahamcOfBorg build python27Packages.foundationdb61
@GrahamcOfBorg build python37Packages.foundationdb51
@GrahamcOfBorg build python37Packages.foundationdb52
@GrahamcOfBorg build python37Packages.foundationdb60
@GrahamcOfBorg build python37Packages.foundationdb61
@GrahamcOfBorg build python38Packages.foundationdb51
@GrahamcOfBorg build python38Packages.foundationdb52
@GrahamcOfBorg build python38Packages.foundationdb60
@GrahamcOfBorg build python38Packages.foundationdb61
@GrahamcOfBorg build s6-networking
@GrahamcOfBorg build skawarePackages.s6-networking
@GrahamcOfBorg build tests.nixos-functions.nixos-test
@GrahamcOfBorg build tests.nixos-functions.nixosTest-test
@GrahamcOfBorg build twa
@GrahamcOfBorg build wasm-pack

@doronbehar
Copy link
Contributor

The foundationdb packages seem broken already (e.g https://hydra.nixos.org/build/119086724/nixlog/1).

But, wasm-pack's failure is more actionable: https://logs.nix.ci/?key=nixos/nixpkgs.88195&attempt_id=88b84284-44e8-4d93-8105-08c64ddc3430 . I think the proper fix for it will be making it switch to openssl. Could you please test this locally before pushing?


libressl = libressl_3_0;
libressl = libressl_3_1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I'd like to see a comment here, indicating future contributors, that libressl should always point to the latest version. See https://discourse.nixos.org/t/nixpkgs-policy-regarding-libraries-available-in-multiple-versions/7026 .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment, with a link to that topic.

@ruuda
Copy link
Contributor Author

ruuda commented May 25, 2020

Thanks for taking a look!

But, wasm-pack's failure is more actionable: https://logs.nix.ci/?key=nixos/nixpkgs.88195&attempt_id=88b84284-44e8-4d93-8105-08c64ddc3430 . I think the proper fix for it will be making it switch to openssl. Could you please test this locally before pushing?

This is the result of a conservative compatibility check in rust-openssl. Compatibility is actually fine, but rust-openssl refuses to compile with anything that is not explicitly listed in its source. This was fixed upstream for LibreSSL 3.1 in sfackler/rust-openssl#1271, which is part of the openssl-sys-v0.9.57 crate. I could switch to OpenSSL, but we will likely run into the same problem at the next OpenSSL version bump.

Is there a way to override a single crate for a Rust project? I suppose we can patch Cargo.lock ... how is this handled in other situations?

Probably switching to OpenSSL is the quick fix, what do you think @doronbehar?

Edit: Yet another alternative would be to make wasm-pack depend on libressl_3_0 explicitly, rather than on libressl. It would need to be updated when libressl_3_0 is no longer supported then.

@ruuda ruuda changed the title libressl: add 3.1.1, default to it, remove 2.9 libressl: add 3.1.2, default to it, remove 2.9 May 25, 2020
@ruuda
Copy link
Contributor Author

ruuda commented May 25, 2020

In the meantime, LibreSSL 3.1.2 was released. I’ve updated this pull request accordingly.

@doronbehar
Copy link
Contributor

Is there a way to override a single crate for a Rust project? I suppose we can patch Cargo.lock ... how is this handled in other situations?

Patching Cargo.lock is a bit annoying... It wouldn't have been that hard if we had only started right from the start to keep derivations for all crates we use to build Rust programs as derivations in Nixpkgs. Rather, we use fixed output derivations which have even more issues than this, see NixOS/nix#2270 & my comment & discussion at #84826 (comment) .

I could switch to OpenSSL, but we will likely run into the same problem at the next OpenSSL version bump.

I think it'll be most proper to let future us handle that :). But seriously, I think this is the cleanest way to handle our issue. More over, I think that we should avoid having some projects depend on Libressl and others on Openssl if they are almost compatible. I mean, unless a project requires specifically Libressl, what advantage does one have on the other? I'm aware of the licensing issue, but is it a big deal for wasm-pack? IDK, it'll need some research...

Perhaps it'll be best to change wasm-pack to use openssl in a different PR, so then the maintainer of wasm will stay focused.

@doronbehar
Copy link
Contributor

Oh and BTW please squash 082e12f .

@doronbehar
Copy link
Contributor

Perhaps it'll be best to change wasm-pack to use openssl in a different PR, so then the maintainer of wasm will stay focused.

There's an upstream issue for wasm-pack with openssl, as indicated by the comment in wasm's default.nix: rustwasm/wasm-pack#650

I'm taking my words back. An even cleaner solution, would be this:

diff --git i/pkgs/top-level/all-packages.nix w/pkgs/top-level/all-packages.nix
index 548b09a6849..8ff32d7fc92 100644
--- i/pkgs/top-level/all-packages.nix
+++ w/pkgs/top-level/all-packages.nix
@@ -26174,6 +26174,7 @@ in
   wasmer = callPackage ../development/interpreters/wasmer { };
 
   wasm-pack = callPackage ../development/tools/wasm-pack {
+    libressl = libressl_3_0;
     inherit (darwin.apple_sdk.frameworks) Security;
   };

Which will even reduce by 1 the amount of packages ofborg will detect as affected by this change.

ruuda added 5 commits May 26, 2020 19:22
Stable LibreSSL releases are supported one year after their OpenBSD
release. OpenBSD 6.5 with the 2.9 branch was released on 2019-05-01.
It is the latest now, let's default to it. 3.0 will still be supported
until October, when OpenBSD 6.6 turns one year old.

Also add reminder to use the latest version, as suggested by doronbehar.
The application is incompatible with LibreSSL 3.1 because rust-openssl
has a compile-time check for supported LibreSSL versions, and the
version of rust-openssl that wasm-pack depends on does not yet support
LibreSSL 3.1.
@ruuda
Copy link
Contributor Author

ruuda commented May 26, 2020

An even cleaner solution, would be this

Yes, that seems like the cleanest solution to me too. I added that, and I squashed 082e12f. Please take another look.

I can also confirm that wasm-pack is no longer affected, because when I tried to build it, I got it from cache.nixos.org.

@doronbehar
Copy link
Contributor

I can also confirm that wasm-pack is no longer affected, because when I tried to build it, I got it from cache.nixos.org.

Exactly 👍.

Let's perform another builds test:

@GrahamcOfBorg build acme-client
@GrahamcOfBorg build foundationdb
@GrahamcOfBorg build foundationdb51
@GrahamcOfBorg build foundationdb52
@GrahamcOfBorg build foundationdb60
@GrahamcOfBorg build foundationdb61
@GrahamcOfBorg build libressl
@GrahamcOfBorg build libressl_3_1
@GrahamcOfBorg build linux-steam-integration
@GrahamcOfBorg build mydumper
@GrahamcOfBorg build netcat
@GrahamcOfBorg build openntpd
@GrahamcOfBorg build openntpd_nixos
@GrahamcOfBorg build opensmtpd
@GrahamcOfBorg build ponyc
@GrahamcOfBorg build pony-stable
@GrahamcOfBorg build pounce
@GrahamcOfBorg build python27Packages.foundationdb51
@GrahamcOfBorg build python27Packages.foundationdb52
@GrahamcOfBorg build python27Packages.foundationdb60
@GrahamcOfBorg build python27Packages.foundationdb61
@GrahamcOfBorg build python37Packages.foundationdb51
@GrahamcOfBorg build python37Packages.foundationdb52
@GrahamcOfBorg build python37Packages.foundationdb60
@GrahamcOfBorg build python37Packages.foundationdb61
@GrahamcOfBorg build python38Packages.foundationdb51
@GrahamcOfBorg build python38Packages.foundationdb52
@GrahamcOfBorg build python38Packages.foundationdb60
@GrahamcOfBorg build python38Packages.foundationdb61
@GrahamcOfBorg build s6-networking
@GrahamcOfBorg build skawarePackages.s6-networking
@GrahamcOfBorg build tests.nixos-functions.nixos-test
@GrahamcOfBorg build tests.nixos-functions.nixosTest-test
@GrahamcOfBorg build twa

@doronbehar
Copy link
Contributor

Among those that failed to build by ofborg at this point, are:

  • acme-client (Darwin only)
  • foundationdb51
  • foundationdb52
  • foundationdb60
  • python27Packages.foundationdb51
  • python27Packages.foundationdb52
  • python27Packages.foundationdb60
  • python37Packages.foundationdb51
  • python37Packages.foundationdb52
  • python37Packages.foundationdb60
  • python38Packages.foundationdb51
  • python38Packages.foundationdb52
  • python38Packages.foundationdb60

All of the foundationdb related packages seem to fail already to build on hydra, according to:

✖ (Failed) foundationdb-5.1.7 from 2020-05-12 - https://hydra.nixos.org/build/119082079
✖ (Failed) foundationdb-5.2.8 from 2020-05-12 - https://hydra.nixos.org/build/119102761
✖ (Failed) foundationdb-6.0.18 from 2020-05-12 - https://hydra.nixos.org/build/119084292
✖ (Dependency failed) python2.7-foundationdb-5.1.7 from 2020-05-12 - https://hydra.nixos.org/build/119106730
✖ (Dependency failed) python2.7-foundationdb-5.2.8 from 2020-05-12 - https://hydra.nixos.org/build/119100305
✖ (Dependency failed) python2.7-foundationdb-6.0.18 from 2020-05-12 - https://hydra.nixos.org/build/119086724
✖ (Dependency failed) python3.7-foundationdb-5.1.7 from 2020-05-12 - https://hydra.nixos.org/build/119085667
✖ (Dependency failed) python3.7-foundationdb-5.2.8 from 2020-05-12 - https://hydra.nixos.org/build/119110244
✖ (Dependency failed) python3.7-foundationdb-6.0.18 from 2020-05-12 - https://hydra.nixos.org/build/119080156
✖ (Dependency failed) python3.8-foundationdb-5.1.7 from 2020-05-12 - https://hydra.nixos.org/build/119108276
✖ (Dependency failed) python3.8-foundationdb-5.2.8 from 2020-05-12 - https://hydra.nixos.org/build/119091045
✖ (Dependency failed) python3.8-foundationdb-6.0.18 from 2020-05-12 - https://hydra.nixos.org/build/119084236

(Thanks hydra-check).

As for acme-client - it's indicated that it fails on Darwin only now, but it seems it failed also before this PR: https://hydra.nixos.org/job/nixpkgs/nixpkgs-20.03-darwin/acme-client.x86_64-darwin

So to summarize, this PR Should be good to go.

@ruuda
Copy link
Contributor Author

ruuda commented May 29, 2020

@thoughtpolice, cc @fpletz, is there anything else I can do to help move this forward?

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-policy-regarding-libraries-available-in-multiple-versions/7026/4

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/167

@ruuda
Copy link
Contributor Author

ruuda commented Jun 16, 2020

In the meantime, LibreSSL 3.1.3 has been released. I updated this PR accordingly.

@ruuda ruuda changed the title libressl: add 3.1.2, default to it, remove 2.9 libressl: add 3.1.3, default to it, remove 2.9 Jun 16, 2020
@Lassulus
Copy link
Member

got this output from nix-review, not sure if there is any regression. Somebody will have to check.

$ nix build --no-link --keep-going --option build-use-sandbox relaxed -f /home/lass/.cache/nixpkgs-review/pr-88195-1/build.nix
warning: ignoring the user-specified setting 'sandbox', because it is a restricted setting and you are not a trusted user
builder for '/nix/store/b6crzcwl1jwi2ivpan06343f069drdb6-foundationdb-5.1.7.drv' failed with exit code 2; last 10 log lines:
    FILE* f;
                   ^     
  In file included from /nix/store/al9wk2c1l2j6fdpclj4k9cnxyc4i0k3b-glibc-2.30-dev/include/unistd.h:1170:0,
                   from ./flow/Platform.h:49,
                   from ./flow/flow.h:40,
                   from flow/Profiler.actor.cpp:39:
  /nix/store/al9wk2c1l2j6fdpclj4k9cnxyc4i0k3b-glibc-2.30-dev/include/bits/unistd_ext.h:34:16: note: old declaration '__pid_t gettid()'
   extern __pid_t gettid (void) __THROW;
                  ^~~~~~
  make: *** [flow/generated.mk:71: .objs/flow/Profiler.actor.g.cpp.o] Error 1
cannot build derivation '/nix/store/kfvhlfhi6qrmiabayhznskld8pcbvkgr-python2.7-foundationdb-5.1.7.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/s6g4xrp1627f8xzrqhkp5bji2kdr1scj-python3.7-foundationdb-5.1.7.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/isy84sg2vdygvjkwfcsh3dvkn8h8cg29-python3.8-foundationdb-5.1.7.drv': 1 dependencies couldn't be built
builder for '/nix/store/yjlipkd1kaxwfrlk7i4yczlnwajpci4z-foundationdb-5.2.8.drv' failed with exit code 2; last 10 log lines:
   static uint64_t gettid() { return syscall(__NR_gettid); }
                   ^~~~~~
  In file included from /nix/store/al9wk2c1l2j6fdpclj4k9cnxyc4i0k3b-glibc-2.30-dev/include/unistd.h:1170:0,
                   from ./flow/Platform.h:49,
                   from ./flow/flow.h:40,
                   from flow/Profiler.actor.cpp:21:
  /nix/store/al9wk2c1l2j6fdpclj4k9cnxyc4i0k3b-glibc-2.30-dev/include/bits/unistd_ext.h:34:16: note: old declaration '__pid_t gettid()'
   extern __pid_t gettid (void) __THROW;
                  ^~~~~~
  make: *** [flow/generated.mk:71: .objs/flow/Profiler.actor.g.cpp.o] Error 1
cannot build derivation '/nix/store/y6bq7sismipmqzc43yqmgn27za6a4030-python2.7-foundationdb-5.2.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/7ldl4nmdlbacsfaaz4x9kbhjlrh64w4z-python3.7-foundationdb-5.2.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/7cpzyi9z9kprvs3l2024yd2am6knwmms-python3.8-foundationdb-5.2.8.drv': 1 dependencies couldn't be built
builder for '/nix/store/m8widbiqwd5pfp9d4zr9nb0b7rc6pbsn-foundationdb-6.0.18.drv' failed with exit code 2; last 10 log lines:
      l = std::min(FLOW_KNOBS->MAX_METRIC_LEVEL-1, (int64_t)(::log(1.0/x) / FLOW_KNOBS->METRIC_LEVEL_DIVISOR));
                                                             ^~
  make: *** [flow/generated.mk:78: .objs/flow/Platform.cpp.o] Error 1
  flow/TDMetric.actor.h: In instantiation of 'void ContinuousMetric<T>::change() [with T = long int]':
  flow/TDMetric.actor.h:1232:9:   required from 'void ContinuousMetric<T>::onEnable() [with T = long int]'
  flow/SystemMonitor.cpp:262:1:   required from here
  flow/TDMetric.actor.h:1277:10: error: 'log' was not declared in this scope
         log((toggleTime - tv.time) / x) /
         ~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
  make: *** [flow/generated.mk:78: .objs/flow/SystemMonitor.cpp.o] Error 1
cannot build derivation '/nix/store/n60rvrpiaajggiyrqx4r7krkpcmm8qmx-python2.7-foundationdb-6.0.18.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/m5kmlfslz84fsimdn0sximj08p5cvwvr-python3.7-foundationdb-6.0.18.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/qvmmxpnfndl6qwp360gnhr6vwra7lcwx-python3.8-foundationdb-6.0.18.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/f425jywjabaij7pg3fr9mzz6m2j9h1fn-env.drv': 12 dependencies couldn't be built
[0 built (3 failed), 0.0 MiB DL]
error: build of '/nix/store/f425jywjabaij7pg3fr9mzz6m2j9h1fn-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/88195
2 packages blacklisted:
tests.nixos-functions.nixos-test tests.nixos-functions.nixosTest-test

12 packages failed to build:
foundationdb51 foundationdb52 foundationdb60 python27Packages.foundationdb51 python27Packages.foundationdb52 python27Packages.foundationdb60 python37Packages.foundationdb51 python37Packages.foundationdb52 python37Packages.foundationdb60 python38Packages.foundationdb51 python38Packages.foundationdb52 python38Packages.foundationdb60

15 packages built:
acme-client foundationdb libressl mydumper netcat openntpd opensmtpd pony-stable ponyc pounce python27Packages.foundationdb61 python37Packages.foundationdb61 python38Packages.foundationdb61 s6-networking twa

@doronbehar
Copy link
Contributor

@Lassulus We already checked 3.1.2 at #88195 (comment) and it seems that these packages were already broken. Let's run once more for 3.1.3, the builds that shouldn't fail:

@GrahamcOfBorg build acme-client
@GrahamcOfBorg build libressl
@GrahamcOfBorg build libressl_3_1
@GrahamcOfBorg build linux-steam-integration
@GrahamcOfBorg build mydumper
@GrahamcOfBorg build netcat
@GrahamcOfBorg build openntpd
@GrahamcOfBorg build openntpd_nixos
@GrahamcOfBorg build opensmtpd
@GrahamcOfBorg build ponyc
@GrahamcOfBorg build pony-stable
@GrahamcOfBorg build pounce
@GrahamcOfBorg build s6-networking
@GrahamcOfBorg build skawarePackages.s6-networking
@GrahamcOfBorg build tests.nixos-functions.nixos-test
@GrahamcOfBorg build tests.nixos-functions.nixosTest-test
@GrahamcOfBorg build twa

@ruuda
Copy link
Contributor Author

ruuda commented Jun 19, 2020

Looks like all the builds that are expected to pass still do, please take another look @Lassulus.

@Lassulus Lassulus merged commit 6d2c0b8 into NixOS:master Jun 19, 2020
@ruuda ruuda deleted the libressl-3.1 branch June 19, 2020 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants