firefox: 82.0.3 -> 83.0, firefox-esr: 78.4.1esr -> 78.5.0esr, nss: 3.56 -> 3.58, cacert refactor #104176

andir · 2020-11-18T15:23:38Z

Motivation for this change

This updates the firefox source packages to the latest upstream version.

This is the continued work of #104051 & #100765.

In #100765 @vcunat pointed out that we could decouple cacert from the NSS package to make it more rebuild friendly. Just rebuilding packages that depend on NSS seems to be about ~100. Rebuilding all the packages that depend on cacert is >9k as of this writing.This makes it much more feasible to upgrade high-profile packages that are (rightfully) pedantic on their NSS version like firefox and thunderbird.

Things done

firefox NixOS test executed
firefox-esr NixOS test executed
thunderbird pinned to older NSS version (FIXME!)
verified that pipewire still works

Paging @vcunat and @ajs124 for the NSS & cacert changes, @worldofpeace & @mweinelt for the pipewire functionality and finally @stigtsp and @taku0 for the actual firefox change. :)

andir · 2020-11-18T15:24:41Z

pkgs/applications/networking/browsers/firefox/common.nix

+  ] ++
+
+  # there are two flavors of pipewire support
+  # The patches for the ESR release and the patches for the current stable


this paragraph looks weird. I should reflow/refromat/reword it.

andir · 2020-11-18T15:25:50Z

pkgs/applications/networking/browsers/firefox/common.nix

  ];

  MACH_USE_SYSTEM_PYTHON = "1";

  postPatch = ''
    rm -rf obj-x86_64-pc-linux-gnu
-  '' + lib.optionalString pipewireSupport ''
+  '' + lib.optionalString (pipewireSupport && lib.versionOlder ffversion "83") ''


@worldofpeace & @mweinelt can you also test pipewire on the esr build? If it didn't and does not work we can remove this section (and the entire old patch).

Building everything as we speak.

Pipewire works on 78.5.0esr, had linker problems on the firefox 83.0 build, retrying that right now.

Make sure your /tmp has enough inodes as otherwise the build fails frequently.

Doesn't work with Firefox 83.0 unfortunately. When trying to share my screen I only see "Use operating system settings". I see no reaction in the pipewire unit.

shrug shall we disable it then? I have zero clues how to debug this.

I am tempted to keep them enabled as they are now.. Things will not work entirely but maybe it is also an environmental thing and works for someone else.

Fine with me.

ajs124 · 2020-11-18T15:29:10Z

Shouldn't we go with NSS 3.53 instead of 3.57 for ESR? 3.53 is an NSS ESR release, whereas I don't think 3.57 is.

andir · 2020-11-18T15:36:25Z

Shouldn't we go with NSS 3.53 instead of 3.57 for ESR? 3.53 is an NSS ESR release, whereas I don't think 3.57 is.

Sounds reasonable. I'll update the PR.

vcunat · 2020-11-18T17:20:14Z

NOTE: Whenever you update this version also update the cacert package
with a PR to staging. cacert is decoupled from this to reduce the
amount of unnecessary rebuilds while both a built from the same
source project.

Maybe we could use a script to compare whether this makes a difference and thus avoid generating meaningless rebuilds even on staging. For example, I compared the current difference (3.57 vs 3.59) and the cert bundles are binary equal.

EDIT: it could be simple... like cacert.overrideAttrs (a_:{ inherit (nss) src version; }). (after switching to pname to avoid confusion)

andir · 2020-11-18T17:29:59Z

NOTE: Whenever you update this version also update the cacert package
with a PR to staging. cacert is decoupled from this to reduce the
amount of unnecessary rebuilds while both a built from the same
source project.

Maybe we could use a script to compare whether this makes a difference and thus avoid generating meaningless rebuilds even on staging. For example, I compared the current difference (3.57 vs 3.59) and the cert bundles are binary equal.

EDIT: it could be simple... like cacert.overrideAttrs (a_:{ inherit (nss) src version; }). (after switching to pname to avoid confusion)

Yeah that might work. I've been looking at the script that we run on the source tree and did wonder if we couldn't have a fetcher that extracts just the data that we care about but that would duplicate a lot of the logic that is in the script :/

vcunat · 2020-11-18T17:35:06Z

We could make it a fixed-output derivation, but that wouldn't really remove the necessity to check if something's changed. (and the switch wouldn't be trivial, due to us wanting two outputs)

andir · 2020-11-18T18:15:28Z

We could make it a fixed-output derivation, but that wouldn't really remove the necessity to check if something's changed. (and it wouldn't be trivial, due to us wanting two outputs)

Lets scratch the fixed output idea for now.

Where would you put that check script?

This would probably be sufficient to check if we have to update the sources of cacerts:

#!/usr/bin/env nix-shell
#!nix-shell -i bash -p nix

# Build both the cacert package and an overriden version where we use the source attribute of NSS.
# Cacert and NSS are both from the same upstream sources. They are decoupled as
# the cacert output only cares about a few infrequently changing files in the
# sources while the NSS source code changes frequently.
#
# By having cacert on a older source revision that produces the same
# certificate output as a newer version we can avoid large amounts of
# unnecessary rebuilds.
#
# As of this writing there are a few magnitudes more packages depending on
# cacert than on nss.


set -ex

BASEDIR="$(dirname "$0")/../../../.."


CURRENT_PATH=$(nix-build --no-out-link -A cacert.out)
PATCHED_PATH=$(nix-build --no-out-link -E 'with import ./. {}; (cacert.overrideAttrs (_: { inherit (nss) src version; })).out')

# Check the hash of the etc subfolder
# We can't check the entire output as that contains the nix-support folder
# which contains the output path itself.
CURRENT_HASH=$(nix-hash "$CURRENT_PATH/etc")
PATCHED_HASH=$(nix-hash "$PATCHED_PATH/etc")

if [[ "$CURRENT_HASH" !=  "$PATCHED_HASH" ]]; then
    echo "mismatch, consider updating"
    exit 1
fi

vcunat · 2020-11-18T18:43:09Z

Maybe into cacert's passthru.updateScript? Though I don't really know any of the auto-updating styles so far.

@vcunat

In [NixOS#100765] @vcunat pointed out that we could decouple cacert from the NSS package to make it more rebuild friendly. Just rebuilding packages that depend on NSS seems to be about ~100. Rebuilding all the packages that depend on cacert is >9k as of this writing. This makes it much more feasible to upgrade high-profile packages that are (rightfully) pedantic on their NSS version like firefox and thunderbird. [NixOS#100765]: NixOS#100765

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_3.58_release_notes

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_3.59_release_notes

andir · 2020-11-18T19:15:30Z

Maybe into cacert's passthru.updateScript? Though I don't really know any of the auto-updating styles so far.

Done. I also didn't know much about those before today.. It wasn't that hard to implement. I skipped the pname migration of cacert as that would trigger a mass rebuild.

This fixes up the changes done in #104176 where I forgot to include this now required change.

This fixes up the changes done in NixOS#104176 where I forgot to include this now required change. (cherry picked from commit a322b32)

andir requested review from stigtsp, mweinelt, ajs124, vcunat and worldofpeace November 18, 2020 15:23

andir commented Nov 18, 2020

View reviewed changes

andir force-pushed the firefox-83 branch 3 times, most recently from 46872f5 to f975aff Compare November 18, 2020 16:02

ofborg bot added the 8.has: package (new) label Nov 18, 2020

ofborg bot requested a review from edolstra November 18, 2020 16:17

ofborg bot added 11.by: package-maintainer 10.rebuild-darwin: 11-100 10.rebuild-linux: 501+ 10.rebuild-linux: 501-1000 labels Nov 18, 2020

andir added 7 commits November 18, 2020 20:13

firefox: prepare pipewire patches for firefox 83

398cdab

nss_3_53: init

cbd0f89

firefox: fallback to nss_3_53 when version is below 83

c1cfe8b

nss: 3.57 -> 3.58

6c33216

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_3.58_release_notes

firefox: 82.0.3 -> 83.0

fddfa39

firefox-esr-78: 78.4.1esr -> 78.5.0esr

ccb42da

nss: 3.58 -> 3.59

fce1a3e

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/NSS_3.59_release_notes

andir force-pushed the firefox-83 branch from f975aff to fce1a3e Compare November 18, 2020 19:15

andir merged commit e5f945b into NixOS:master Nov 18, 2020

andir added a commit that referenced this pull request Nov 19, 2020

thunderbird: use nss_3_53 instead of nss

a322b32

This fixes up the changes done in #104176 where I forgot to include this now required change.

andir mentioned this pull request Nov 19, 2020

[20.09] firefox: 82.03 -> 83.0, firefox-esr: 78.4.1esr -> 78.5.0esr #104260

Merged

5 tasks

andir added a commit to andir/nixpkgs that referenced this pull request Nov 19, 2020

thunderbird: use nss_3_53 instead of nss

67df165

This fixes up the changes done in NixOS#104176 where I forgot to include this now required change. (cherry picked from commit a322b32)

This was referenced Dec 1, 2020

Vulnerability roundup 94: firefox-78.3.0esr: 1 advisory [8.8] #100297

Closed

Vulnerability roundup 93: firefox-78.3.0esr: 17 advisories [8.8] #99757

Closed

Vulnerability roundup 95: firefox-78.3.1esr: 23 advisories [8.8] #101152

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

firefox: 82.0.3 -> 83.0, firefox-esr: 78.4.1esr -> 78.5.0esr, nss: 3.56 -> 3.58, cacert refactor #104176

firefox: 82.0.3 -> 83.0, firefox-esr: 78.4.1esr -> 78.5.0esr, nss: 3.56 -> 3.58, cacert refactor #104176

andir commented Nov 18, 2020

andir Nov 18, 2020

andir Nov 18, 2020

mweinelt Nov 18, 2020

mweinelt Nov 18, 2020

andir Nov 18, 2020

mweinelt Nov 18, 2020

andir Nov 18, 2020

mweinelt Nov 18, 2020

andir Nov 18, 2020

mweinelt Nov 18, 2020

ajs124 commented Nov 18, 2020

andir commented Nov 18, 2020

vcunat commented Nov 18, 2020 •

edited

andir commented Nov 18, 2020

vcunat commented Nov 18, 2020 •

edited

andir commented Nov 18, 2020 •

edited

vcunat commented Nov 18, 2020

andir commented Nov 18, 2020 •

edited

firefox: 82.0.3 -> 83.0, firefox-esr: 78.4.1esr -> 78.5.0esr, nss: 3.56 -> 3.58, cacert refactor #104176

firefox: 82.0.3 -> 83.0, firefox-esr: 78.4.1esr -> 78.5.0esr, nss: 3.56 -> 3.58, cacert refactor #104176

Conversation

andir commented Nov 18, 2020

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajs124 commented Nov 18, 2020

andir commented Nov 18, 2020

vcunat commented Nov 18, 2020 • edited

andir commented Nov 18, 2020

vcunat commented Nov 18, 2020 • edited

andir commented Nov 18, 2020 • edited

vcunat commented Nov 18, 2020

andir commented Nov 18, 2020 • edited

vcunat commented Nov 18, 2020 •

edited

vcunat commented Nov 18, 2020 •

edited

andir commented Nov 18, 2020 •

edited

andir commented Nov 18, 2020 •

edited