Shipping Security Updates #10851

nbp · 2015-11-06T22:10:44Z

Motivations

The way of handling security updates today involves a lot of humans efforts, going from reading the mailing list, to adding lines to the configuration files.

More over, this implies having control over the top-level derivation, which is not always possible with nix-env, and nix-shell. This has to be patched inside the current pipeline, as what got made for NixOS.

Also, as the current method for applying security updates is complex, we are only restricting them to a few packages, such as glibc, bash and openssl which are used a lot, but we do not such mechanism for all other packages which have minor security / usability updates such as firefox.

As a work-around, today, we offer the option of having a small channel, to get updates from, but this is not a good solution for end-users, as the small channel basically involves recompiling any large project.

More over the current mechanism cannot safely be composed as explained in #4336.

Goals

This work aims at making it transparent, such as security updates are cannot be distinguished from any other updates by the end user.

This work aims at making it easy, such that packagers only have to cherry-pick a commit made on master into a security/master branch to let the buildfarm compile and distribute the package against the latest stable branch.

This work aims at making it fast, such that the buildfarm in charge of producing pre-compiled versions of the packages will only recompile the packages which are being fixed, against the latest stable branch. When a user updates, Nix will download from the latest stable channel, with the addition of the recompiled packages, and Nix will patch all the packages depending on the fixed packages.

This new approach is sane, as it constructs the patches which have to be apply, without any knowledge of the runtime dependency. Thus, all the packages patches can be build up-front without having to realize any of the derivations. This is a big plus to guarantee reproducible builds of the patching mechanism.

Approach

Nixpkgs as a pure function

This approach aims at constructing patches based on the Nix expressions of the packages. Thus we have to change the way we evaluate Nixpkgs to reflect the patching mechanism.

This approach is based on the recent modification of #14000 (#9400), which make Nixpkgs behave as a single function with no external inputs. As the Nix language is pure, the following property holds for Nixpkgs.

    fix f == f (fix f)

This property means that we can call f one extra time on the result of the fix-point without changing the result of the fix-point. As a consequence, all the packages will keep the same hashes.

Recompile only fixed packages

This patching mechanism relies on one hypothesis. This hypothesis is that all packages derivations are expressed in the last evaluation of Nixpkgs, and that dependencies are taken from the previous evaluation of Nixpkgs.

This hypothesis does not hold, as of today, but with the help of a static analysis, we can identify how many times we jump through the Nixpkgs argument before we settle on a derivation, and before we settle on its inputs.

Once this hypothesis is guaranteed, we can change the above expression to

    g (fix f)

Where g is a slightly different version of f. With the above hypothesis, this means that all derivations of packages are taken from g, but the dependencies of g are taken from f.

If f is the stable version of Nixpkgs, and g is a stable version of Nixpkgs with extra patches. This implies we will recompile any patched packages of g, while keeping the same hashes as f for non-patched packages.

At this stage, we have what we expect to let Hydra compile. Still, this stage does not apply any patches to any of the packages.

Building patches

To be able to apply patches to the dependencies, we need the equivalent of a fix-point, to evaluate the dependencies, and the dependencies of the dependencies, and so on … In addition to a fix-point, we want to iterate from the maybe-recompiled packages of g (fix f), and iterate over g to settle on the patched version of the packages. We would name this patched version of packages h.

To construct patches, we need to know the dependencies of the packages, and which one got updated.

To list the dependencies of a package, we have to rely on some form of introspection, encoded as part of the derivations. As a first approach I relied on callPackage, which would give us a super set of the dependencies, but this has a lot of issues, and left-over packages. As a second approach, I now rely on the buildInputs which are exposed by the derivation, but this has the issue of being a sub-set of the dependencies. Still, I am confident that we can instrument the default builder to ensure that buildInputs are a super set of the runtime dependencies.

To check for updates, we can compare the outPath of 2 derivations. If the derivations have a different outPath, this means that the derivations has a different inputs. (Note: hash collisions are unlikely, but if this happens we can easily add a dummy environment variable to get a different hash)

Patches are composed of the hashes of the old versions, and the hashes of the new versions. Making a patch for one derivation, is being able to list the old hashes and associate them with the new hashes. The hashes are the hashes of the dependencies.

Based on the last hypothesis, looking for the dependencies of g (fix f), will gives us hashes of derivations from fix f. These are our old hashes.

Looking for the new hashes, implies that we have to build packages based on patched versions of the packages, i-e h. Thus if we look for the dependencies of g h, we would have our new hashes.

Therefore h definition should look like:

  h = patch (g (fix f)) (g h);

Where patch is a function which on a derivation, will take the value of g (fix f), and apply a patch for any of its dependencies which got updated, by substituting the dependencies from fix f by dependencies from h.

In practice, we also need to feed fix f to this patch function, as we have to make sure that patches can be applied safely, by ensuring that the outPath of the derivation still have the same length.

Status

Implement this approach on top of Nixpkgs
Add test case to ensure that, when the one-application hypothesis holds, we can only recompile fixed packages, and apply patches on the rest of the packages.
Make a static analysis which reports all violations of the one-application hypothesis.
nix-env -f ./. -qaP --drv-path works when quickfixPackages is equal to defaultPackages, and does not cause recompilation of existing packages. (Note: nox-review fails to see that we have no recompilation because of violation of the one-application hypothesis)
Add a test case to verify that nix-env -f ./. -qaP --drv-path always works, when quickfixPackages is equal to defaultPackages.
Hook the test cases as release blockers of Nixpkgs.

This mechanism is disabled by default, as no quickfixPackages are provided yet. Also, when disabled, this work has no impact.

To avoid to bloat and delay these changes, I would recommend that we merge them as soon as possible, even if the one-application hypothesis does not hold yet, and address any issues reported by the static analysis as follow-up issues.

References

NixCon slides: http://nbp.github.io/slides/NixCon/2015.ShippingSecurityUpdates/
NixCon video: https://www.youtube.com/watch?v=RhcKXS00zEE

PyroLagus · 2015-11-07T16:05:29Z

Thank you! This is much needed. I'm excited to see how this will turn out :)

nbp · 2015-11-08T12:10:59Z

@Mathnerd314 :

There's some existing work in https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/replace-dependency.nix if you haven't seen it.

I saw it, and this is exactly why I am not using it, and why I am making this work. I will explain more next Sunday in my talk.

@Mathnerd314 :

I guess this is a hack until the promised "Optimize this function by only using runtime dependencies of the original package set" is implemented?

No, in fact, this is exactly the opposite. The hack is to rely on runtime dependencies, as runtime dependencies are not known statically (without building packages). The current implementation is a conservative approximation of runtime dependencies. There is still one issue with the current function, as it does not consider the closure of dependencies, but only explicit dependencies.

nbp · 2015-11-16T12:11:59Z

This extra patches are changing the design to fit better with the ideal model shown in the slide of my presentation. [1][2]

So far, while evaluating packages with nix-env -f ./. -qaP --drv-path I saw multiple issues.

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.
So far, the code is unable to retrieve the list of dependencies of 3190 packages, which means that for these packages we would not patch them if one of their dependencies got modified.

[1] http://nbp.github.io/slides/NixCon/2015.ShippingSecurityUpdates/
[2] https://www.youtube.com/watch?v=RhcKXS00zEE&list=PL_IxoDz1Nq2Y7mIxMZ28mVtjRbbnlVdmy&index=13

Ericson2314 · 2015-11-18T18:06:28Z

pkgs/top-level/all-packages-wrapper.nix

+      abiSec;
+
+
+  # The package compositions.  Yes, this isn't properly indented.


Uh, no it is. :)

Ericson2314 · 2015-11-18T18:19:00Z

It won't render all-packages.nix for me to do a line comment, but the documentation the of self (the parameter) says self is affected by user overrides, but since it is fixed within pkgFun I don't think it is.

Ericson2314 · 2015-11-18T18:24:27Z

lib is let-bound in all-packages.nix, but since it is also inherited in the body of the function this could be made more concise by doing just lib = import ../lib;. Since with pkgs; comes after the let-binding of lib, either way were a user to override lib that would take priority.

Ericson2314 · 2015-11-18T18:30:06Z

@nbp I've only glanced at your slides, so can't yet speak to that. But even in terms of the refactoring alone, this is a HUGE improvement. Thanks so much! (and @Mathnerd314 too!)

Ericson2314 · 2015-11-18T18:34:55Z

Oh and @Mathnerd314 got a good portion of my comments too, hah, but they didn't show up on the diff.

Ericson2314 · 2015-11-19T03:50:58Z

pkgs/top-level/all-packages-wrapper.nix

+  # packageOverrides attributes to refer to the original attributes
+  # (e.g. "foo = ... pkgs.foo ...").
+  configOverrides =
+      if config ? packageOverrides && bootStdenv == null # don't apply config overrides in stdenv boot


nit: 4 space indent

Mathnerd314 · 2015-11-20T17:09:37Z

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.

This is because nix-env uses pointer equality for skipping duplicate derivations, rather than output path or derivation path: https://github.com/NixOS/nix/blob/master/src/libexpr/get-drvs.cc#L193. Personally, I would remove that check; outputting all possible aliases is consistent with the behavior of command-not-found.

Mathnerd314 · 2015-11-20T19:21:40Z

I rebased the patches; still testing it.

nbp · 2015-11-20T19:35:04Z

from @Ericson2314

If we remember the hashes of things we are going to rebuild, and we find something we wouldn't rebuild that has the same hash as something we are rebuilding, we could "rebuild" it anyways. That will work for simply rebound aliases, but not for alises that involve overriding (an example of introduction >= elimination, but introduction > 0).

I intended to do something similar to ensure that we do not add such aliases to nixpkgs.

I do not think this is a good choice to do this automatically as this effectively means that for installing any sotfware you will have to evaluate all of them.

from @Mathnerd314

Aliases are shown in the list of packages, while they are not listed without the quickfix stage applied.

This is because nix-env uses pointer equality for skipping duplicate derivations, rather than output path or derivation path: https://github.com/NixOS/nix/blob/master/src/libexpr/get-drvs.cc#L193. Personally, I would remove that check; outputting all possible aliases is consistent with the behavior of command-not-found.

That's what I thought at first, but then I realized that these differences are related to aliased packages, and that for each newly reported entry, we have a potential security issue (as described in #10851 (comment))

So even if this behaviour of nix-env is not consistent, we should still fix these issues.

Ericson2314 · 2015-12-01T21:19:56Z

@nbp Sorry for taking a while to respond, but I think that all makes sense now. Thanks for explaining it.

I wonder if we should revive #10307 after all, as a diff this big goes stale so quick.

Ericson2314 · 2015-12-01T23:04:41Z

Also Ericson2314@03486c8 here is a commit just with the improved documentation for overridePackages you might want to add to this.

jb55 · 2016-02-18T20:38:44Z

I got codetriaged here. What's the status of this?

Ericson2314 · 2016-02-19T15:09:34Z

I'm not sure about the status of the main security part, but the improved fix-point part has been good to go for a while, hense my advocacy to split this in two.

PyroLagus · 2016-02-21T17:48:21Z

Wait, so if the improved fix-point is already finished, what do we have to do to make it work and patch security updates? I'd really like to help on this, but I have no idea how.

nbp · 2016-10-31T19:52:28Z

@teh:

Explain what the output of the static analysis means,

Done (see above)

and how we can fix it

Unfortunately highly depends on each of the Nix expression. But as soon as you understand how the static analysis works, you can probably figure your-self how to fix it. Otherwise fill free to ask questions here.

Add some docs how this can be used by end-users

This is made to be transparent, so technically the only documentation needed to to point people to the right channel / repository-branch, and that would be all. But for that we need prototype to distribute the content of security-updates branch.

A working demo

@domenkozar and I had a discussion about that almost a month ago. On my side, I have to fix enough of these issues such that we can reduce the number of recompilation.

spacekitteh · 2016-11-04T13:59:49Z

@nbp what sort of fixes are holding up a working demo? Anything that actually breaks the mechanism? Or is it that you're trying to fix all of nixpkgs of static analysis errors?

nbp · 2016-11-05T16:34:46Z

@spacekitteh ,

what sort of fixes are holding up a working demo?

There is no mandatory fixes but this would be a nicer experience to demo a larger set of fixable packages.

From the infrastructure, the main blocker for getting a working demo is to get the security-updates branch built in Hydra, i-e making rules to build the base branch and merge it with the security branch.

Then, we need some task-force to get some fixes (or dummy fixes) applied in the security updates branch.

The static analysis is only a tool made to highlight the parts which would not work properly. I have not yet measured, but I would expect that we have already have a large number of packages with no issues. Among the prerequisite are that I should write some tests to ensure that we can successfully update real packages such as openssl.

Anything that actually breaks the mechanism?

The mechanism can already be tested locally. For that you need to check out the security-updates branch twice, in 2 different directories, and call the default.nix file from one with a quickfixPackages argument which imports the pkgs/top-level/index.nix file of the other. The other directory contains all the program which have security patches.

Currently, we could merge the security-updates branch as-is, as the patching mechanism is disabled unless the quickfixPackages argument is provided.

Or is it that you're trying to fix all of nixpkgs of static analysis errors?

That is something that I would hardly be able to complete alone. Writing documentation to explain how to fix them would be hard as people tend to innovate every-time they write new packages.

One of the restriction of the security-updates branch is that all packages should use callPackage, and that callPackage should provide all the input of the package. We should not use callPackages, as this suggest that people can use rec to get other packages, instead of getting them from the arguments of callPackage.

infinisil · 2019-01-27T19:51:36Z

Closing because this seems to be stalled (for 2 years now), feel free to reopen when work is being done again.

nixos-discourse · 2019-05-19T06:59:26Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/in-overlays-when-to-use-self-vs-super/2968/5

nixos-discourse · 2022-10-22T10:33:08Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-10-19-nixpkgs-architecture-team-meeting-14/22578/3

nixos-discourse · 2023-06-19T10:13:18Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/timely-updates-for-nixos/29231/7

nbp added 0.kind: enhancement 2.status: work-in-progress 1.severity: security labels Nov 6, 2015

nbp added this to the 16.03 milestone Nov 6, 2015

nbp mentioned this pull request Nov 18, 2015

WIP: Refactor fixpoints and overriding in all-packages #11110

Closed

nbp self-assigned this Nov 18, 2015

Ericson2314 reviewed Nov 18, 2015
View reviewed changes

Ericson2314 reviewed Nov 19, 2015
View reviewed changes

nbp mentioned this pull request Nov 20, 2015

Split all-packages.nix #10307

Closed

nbp added 9.needs: changelog 9.needs: documentation 8.has: clean-up 9.needs: clean-up labels Nov 22, 2015

nbp mentioned this pull request Nov 25, 2015

w3m and ranger fixes #11222

Merged

spacekitteh mentioned this pull request Nov 5, 2016

There needs to be an automated tool to harvest the NVD vulnerability database and match it with the current nixpkgs tree. #20176

Closed

nbp mentioned this pull request Dec 17, 2016

Add overlays mechanism to Nixpkgs. #21243

Merged

4 tasks

nbp mentioned this pull request Feb 26, 2017

RFC to describe the RFC process NixOS/rfcs#1

Merged

4 tasks

globin modified the milestones: 17.09, 17.03 Mar 13, 2017

nbp mentioned this pull request Oct 31, 2017

overlay: doc update for self/super -> final/previous #30958

Closed

8 tasks

nbp mentioned this pull request Mar 3, 2018

[RFC 0003] SOS: Simple Override Strategy. NixOS/rfcs#3

Closed

7c6f434c added the 11.by: nixpkgs-member label Mar 10, 2018

bobvanderlinden mentioned this pull request Jun 10, 2018

Pull requests becoming stale #41793

Closed

vcunat mentioned this pull request Aug 11, 2018

Security patches? NixOS/nix#614

Closed

timbertson mentioned this pull request Sep 10, 2018

glibc 2.27 breaks locale support #38991

Closed

matthewbauer modified the milestones: 17.09, 18.09 Oct 1, 2018

samueldr modified the milestones: 18.09, 19.03 Oct 6, 2018

veprbl added the 2.status: merge conflict label Nov 28, 2018

infinisil closed this Jan 27, 2019

infinisil removed this from the 19.03 milestone Jan 27, 2019

garbas moved this from TODO to Done in Security Apr 8, 2020

nbp mentioned this pull request Apr 9, 2021

Pop, object system with multiple-inheritance for nix #116275

Open

10 tasks

SomeoneSerge mentioned this pull request Jun 2, 2023

gst-plugins-bad: remove opencv to improve build time #235481

Merged

12 tasks

2xsaiko mentioned this pull request Jun 9, 2023

use-case: deeply replace packages nixpkgs-architecture/pkgs-modules#12

Merged

spikespaz mentioned this pull request Feb 7, 2024

nix: overlay polish for prev parameter hyprwm/Hyprland#4558

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shipping Security Updates #10851

Shipping Security Updates #10851

nbp commented Nov 6, 2015

PyroLagus commented Nov 7, 2015

nbp commented Nov 8, 2015

nbp commented Nov 16, 2015

Ericson2314 Nov 18, 2015

Ericson2314 commented Nov 18, 2015

Ericson2314 commented Nov 18, 2015

Ericson2314 commented Nov 18, 2015

Ericson2314 commented Nov 18, 2015

Ericson2314 Nov 19, 2015

Mathnerd314 commented Nov 20, 2015

Mathnerd314 commented Nov 20, 2015

nbp commented Nov 20, 2015

Ericson2314 commented Dec 1, 2015

Ericson2314 commented Dec 1, 2015

jb55 commented Feb 18, 2016

Ericson2314 commented Feb 19, 2016

PyroLagus commented Feb 21, 2016

nbp commented Oct 31, 2016

spacekitteh commented Nov 4, 2016 •

edited

nbp commented Nov 5, 2016

infinisil commented Jan 27, 2019

nixos-discourse commented May 19, 2019

nixos-discourse commented Oct 22, 2022

nixos-discourse commented Jun 19, 2023

		abiSec;


		# The package compositions. Yes, this isn't properly indented.

Shipping Security Updates #10851

Shipping Security Updates #10851

Conversation

nbp commented Nov 6, 2015

Motivations

Goals

Approach

Nixpkgs as a pure function

Recompile only fixed packages

Building patches

Status

References

PyroLagus commented Nov 7, 2015

nbp commented Nov 8, 2015

nbp commented Nov 16, 2015

Ericson2314 Nov 18, 2015

Choose a reason for hiding this comment

Ericson2314 commented Nov 18, 2015

Ericson2314 commented Nov 18, 2015

Ericson2314 commented Nov 18, 2015

Ericson2314 commented Nov 18, 2015

Ericson2314 Nov 19, 2015

Choose a reason for hiding this comment

Mathnerd314 commented Nov 20, 2015

Mathnerd314 commented Nov 20, 2015

nbp commented Nov 20, 2015

Ericson2314 commented Dec 1, 2015

Ericson2314 commented Dec 1, 2015

jb55 commented Feb 18, 2016

Ericson2314 commented Feb 19, 2016

PyroLagus commented Feb 21, 2016

nbp commented Oct 31, 2016

spacekitteh commented Nov 4, 2016 • edited

nbp commented Nov 5, 2016

infinisil commented Jan 27, 2019

nixos-discourse commented May 19, 2019

nixos-discourse commented Oct 22, 2022

nixos-discourse commented Jun 19, 2023

spacekitteh commented Nov 4, 2016 •

edited