home-assistant: Respect nixpkgs Python package overrides instead of replacing them #104161

Shados · 2020-11-18T12:59:41Z

Motivation for this change

Currently, home-assistant's definition replaces any existing Python package overrides instead of just layering on top of them, preventing HA from using any packages in existing overrides. This changes that.

Things done

SuperSandro2000 · 2020-11-23T13:19:25Z

Can you fix the eval errors of ofborg?

Shados · 2020-11-24T15:02:42Z

Oh, my bad. Amusingly I forgot to consider and test the normal case of not having any upstream Python packageOverrides. I've fixed that now.

mweinelt · 2020-12-06T17:22:14Z

Do you have an example to help me understand what does not work currently? I don't think I entirely grasp the problem.

Shados · 2020-12-18T12:57:31Z

@mweinelt Without this patch, any Python package overrides specified in a nixpkgs overlay won't make it into the python3 that HA uses, only those overrides explicitly provided to HA via its packageOverrides argument will.

e.g. overrides specified as per below won't be available to HA currently, but will with this patch:

{ ... }:
{
  nixpkgs.overlays = [
    (self: super: {
      python3 = super.python3.override { packageOverrides = ...; };
    })
  ];
}

mweinelt · 2020-12-18T22:39:09Z

Sorry, I need @dotlambda, who came up with this construct, to review this. I understand the problem, but not the solution.

dotlambda

I thought about whether packageOverrides can be removed completely. Is the scenario that you have to override something in defaultOverrides likely to occur?

dotlambda · 2021-06-16T07:46:25Z

pkgs/servers/home-assistant/default.nix

-    # Put packageOverrides at the start so they are applied after defaultOverrides
-    packageOverrides = lib.foldr lib.composeExtensions (self: super: { }) ([ packageOverrides ] ++ defaultOverrides);
-  };
+  py = python3.override(oa: {


This is not how override works, is it? It doesn't expect a function, at least according to https://nixos.org/manual/nixpkgs/unstable/#sec-pkg-override.

Oh, after reading lib/customisation.nix it seems like this is possible:

nixpkgs/lib/customisation.nix

Lines 67 to 93 in 8e84caf

makeOverridable = f: origArgs:

let

result = f origArgs;

# Creates a functor with the same arguments as f

copyArgs = g: lib.setFunctionArgs g (lib.functionArgs f);

# Changes the original arguments with (potentially a function that returns) a set of new attributes

overrideWith = newArgs: origArgs // (if lib.isFunction newArgs then newArgs origArgs else newArgs);

# Re-call the function but with different arguments

overrideArgs = copyArgs (newArgs: makeOverridable f (overrideWith newArgs));

# Change the result of the function call by applying g to it

overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;

in

if builtins.isAttrs result then

result // {

override = overrideArgs;

overrideDerivation = fdrv: overrideResult (x: overrideDerivation x fdrv);

${if result ? overrideAttrs then "overrideAttrs" else null} = fdrv:

overrideResult (x: x.overrideAttrs fdrv);

}

else if lib.isFunction result then

# Transform the result into a functor while propagating its arguments

lib.setFunctionArgs result (lib.functionArgs result) // {

override = overrideArgs;

}

else result;

(pay attention to the if in the definition of overrideWIth)
But then I think we should use

Suggested change

py = python3.override(oa: {

py = python3.override (oldAttrs: {

because that's common naming.

Not actually oldArgs?

True, that makes even more sense. Or origArgs, as it's called above.

dotlambda · 2021-06-16T07:46:52Z

pkgs/servers/home-assistant/default.nix

-  };
+  py = python3.override(oa: {
+    # Put packageOverrides at the start so they are applied after
+    # defaultOverrides and upstream overrides


Maybe

Suggested change

# defaultOverrides and upstream overrides

# defaultOverrides and overlay overrides

is a better wording?

dotlambda · 2021-06-16T08:05:02Z

pkgs/servers/home-assistant/default.nix

+    packageOverrides = lib.foldr lib.composeExtensions (self: super: { }) (
+      [ packageOverrides ]
+      ++ defaultOverrides
+      ++ (lib.optional (oa ? packageOverrides) oa.packageOverrides));


Suggested change

++ (lib.optional (oa ? packageOverrides) oa.packageOverrides));

++ lib.optional (oa ? packageOverrides) oa.packageOverrides;

or

Suggested change

++ (lib.optional (oa ? packageOverrides) oa.packageOverrides));

++ oa.packageOverrides or [ ];

dotlambda · 2021-06-16T08:06:11Z

I understand the problem, but not the solution.

If you override packageOverrides twice, its value from the second overriding, i.e. the one in pkgs/servers/home-assistant/default.nix is used. If we also want to respect existing packageOverrides from overlays, we need to add existing ones to the list of packageOverrides we pass to override.

stale · 2022-01-08T23:07:21Z

I marked this as stale due to inactivity. → More info

SuperSandro2000 · 2022-11-08T00:14:29Z

Is this still relevant?

mweinelt · 2023-02-11T00:34:23Z

Doesn't apply anymore since #182112 and I wonder if it is still required thereafter.

home-assistant: Respect nixpkgs Python package overrides

59b0d4d

Shados force-pushed the fix-ha-python-overrides branch from bc2ac87 to 59b0d4d Compare November 24, 2020 15:02

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Nov 24, 2020

SuperSandro2000 requested review from dotlambda and Mic92 November 25, 2020 02:03

Mic92 requested a review from mweinelt November 25, 2020 10:34

mweinelt added this to To do in Home Assistant via automation May 4, 2021

dotlambda reviewed Jun 16, 2021

View reviewed changes

dotlambda requested changes Jun 16, 2021

View reviewed changes

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 8, 2022

mweinelt moved this from Incoming to Needs work in Home Assistant Feb 12, 2022

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 8, 2022

mweinelt closed this Feb 11, 2023

Home Assistant automation moved this from Needs work to Done Feb 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

home-assistant: Respect nixpkgs Python package overrides instead of replacing them #104161

home-assistant: Respect nixpkgs Python package overrides instead of replacing them #104161

Shados commented Nov 18, 2020

SuperSandro2000 commented Nov 23, 2020

Shados commented Nov 24, 2020

mweinelt commented Dec 6, 2020

Shados commented Dec 18, 2020

mweinelt commented Dec 18, 2020

dotlambda left a comment

dotlambda Jun 16, 2021 •

edited

dotlambda Jun 16, 2021

mweinelt Jun 16, 2021 •

edited

dotlambda Jun 16, 2021 •

edited

dotlambda Jun 16, 2021

dotlambda Jun 16, 2021

dotlambda commented Jun 16, 2021

stale bot commented Jan 8, 2022

SuperSandro2000 commented Nov 8, 2022

mweinelt commented Feb 11, 2023

	makeOverridable = f: origArgs:
	let
	result = f origArgs;

	# Creates a functor with the same arguments as f
	copyArgs = g: lib.setFunctionArgs g (lib.functionArgs f);
	# Changes the original arguments with (potentially a function that returns) a set of new attributes
	overrideWith = newArgs: origArgs // (if lib.isFunction newArgs then newArgs origArgs else newArgs);

	# Re-call the function but with different arguments
	overrideArgs = copyArgs (newArgs: makeOverridable f (overrideWith newArgs));
	# Change the result of the function call by applying g to it
	overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;
	in
	if builtins.isAttrs result then
	result // {
	override = overrideArgs;
	overrideDerivation = fdrv: overrideResult (x: overrideDerivation x fdrv);
	${if result ? overrideAttrs then "overrideAttrs" else null} = fdrv:
	overrideResult (x: x.overrideAttrs fdrv);
	}
	else if lib.isFunction result then
	# Transform the result into a functor while propagating its arguments
	lib.setFunctionArgs result (lib.functionArgs result) // {
	override = overrideArgs;
	}
	else result;

	py = python3.override(oa: {
	py = python3.override (oldAttrs: {

	# defaultOverrides and upstream overrides
	# defaultOverrides and overlay overrides

	++ (lib.optional (oa ? packageOverrides) oa.packageOverrides));
	++ lib.optional (oa ? packageOverrides) oa.packageOverrides;

	++ (lib.optional (oa ? packageOverrides) oa.packageOverrides));
	++ oa.packageOverrides or [ ];

home-assistant: Respect nixpkgs Python package overrides instead of replacing them #104161

home-assistant: Respect nixpkgs Python package overrides instead of replacing them #104161

Conversation

Shados commented Nov 18, 2020

Motivation for this change

Things done

SuperSandro2000 commented Nov 23, 2020

Shados commented Nov 24, 2020

mweinelt commented Dec 6, 2020

Shados commented Dec 18, 2020

mweinelt commented Dec 18, 2020

dotlambda left a comment

Choose a reason for hiding this comment

dotlambda Jun 16, 2021 • edited

Choose a reason for hiding this comment

dotlambda Jun 16, 2021

Choose a reason for hiding this comment

mweinelt Jun 16, 2021 • edited

Choose a reason for hiding this comment

dotlambda Jun 16, 2021 • edited

Choose a reason for hiding this comment

dotlambda Jun 16, 2021

Choose a reason for hiding this comment

dotlambda Jun 16, 2021

Choose a reason for hiding this comment

dotlambda commented Jun 16, 2021

stale bot commented Jan 8, 2022

SuperSandro2000 commented Nov 8, 2022

mweinelt commented Feb 11, 2023

dotlambda Jun 16, 2021 •

edited

mweinelt Jun 16, 2021 •

edited

dotlambda Jun 16, 2021 •

edited