[WIP] Home Assistant #33675

fleaz · 2018-01-09T22:31:17Z

Motivation for this change

Left to do:

NixOS module
NixOS tests
Handling dynamic pip installs
Build HA with tests enabled

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Tests are currently disabled due to many errors. There are also pending commits on the 'dev' branch which fixes some tests. Also home-assistant installs more dependencies dynamically on every start via pip. This still needs to be fixed.

dotlambda · 2018-01-10T09:47:13Z

In order to prevent home-assistant from later pulling in dependencies, we could maybe replace the function install_package() by something like

def install_package(package: str, upgrade: bool=True,
                    target: Optional[str]=None,
                    constraints: Optional[str]=None) -> bool:
    return check_package_exists(package)

I haven't tested this though.
There also seems to be the option skip_pip, e.g. here. I guess it can be triggered using --skip-pip

Also, I think there should be an option to specify additional components, which also requires the possibility to specify additional dependencies.

FRidh · 2018-01-10T10:36:31Z

pkgs/development/python-modules/astral/default.nix

+buildPythonPackage rec {
+  pname = "astral";
+  version = "1.4";
+  name  = "${pname}-${version}";


FRidh · 2018-01-10T10:36:43Z

pkgs/development/python-modules/vincenty/default.nix

+{ stdenv, fetchPypi, buildPythonApplication }:
+
+buildPythonApplication rec {
+  name = "${pname}-${version}";


FRidh · 2018-01-10T10:37:15Z

pkgs/development/python-modules/vincenty/default.nix

@@ -0,0 +1,25 @@
+{ stdenv, fetchPypi, buildPythonApplication }:
+
+buildPythonApplication rec {


python-modules and python-packages.nix should contain only libraries ( buildPythonPackage) and not applications (buildPythonApplication).

FRidh · 2018-01-10T10:37:33Z

pkgs/servers/home-assistant/default.nix

+{ stdenv, fetchFromGitHub, fetchurl, python3Packages, git, nmap }:
+
+python3Packages.buildPythonApplication rec {
+  name = "${pname}-${version}";


FRidh · 2018-01-10T10:37:40Z

pkgs/servers/home-assistant/default.nix

+    sha256 = "164a87dh4zxw1a5nmlgwjc1nls0d4jjhdy5pzz43pgnwhhflbph3";
+  };
+
+  patchPhase = ''


buildPythonPackage inherits from toPythonModule which automatically generates the name for the package with "name ? "${attrs.pname}-${attrs.version}" in the file mk-python-derivation.nix

FRidh · 2018-01-11T18:11:00Z

The following implements skip-pip:

diff --git a/pkgs/servers/home-assistant/default.nix b/pkgs/servers/home-assistant/default.nix
index c2ea5c28229..984925bfd7d 100644
--- a/pkgs/servers/home-assistant/default.nix
+++ b/pkgs/servers/home-assistant/default.nix
@@ -19,6 +19,7 @@ python3Packages.buildPythonApplication rec {
   propagatedBuildInputs = with python3Packages; [ astral certifi netifaces pip vincenty webcolors pyyaml yarl
                                                   aiohttp-cors voluptuous async-timeout requests aiohttp jinja2
                                                   pytz chardet setuptools_scm typing git nmap ];
+  makeWrapperArgs = [ "--add-flags --skip-pip" ];
 
 
  #checkInputs = [ python3Packages.pytest python3Packages.sqlalchemy ];

home-assistant is the opposite of Nix:
home-assistant/core#11134 (comment)

We may need to find a way to include additional dependencies.

FRidh · 2018-01-11T18:11:39Z

Earlier home-assistant PR #19508

fleaz · 2018-01-12T14:46:07Z

@FRidh Thanks for the link to the old PR. I will have a look at it.

Re. "--skip-pip":
This way HA won't install anything on demand afaik. Every time a user would add a new component to her HA configuration, she would need to read the sourcecode of the activated component to get the list of needed python requirements and write them to the module configuration of her nixos home-assistant package. Not sure if this is very user friendly

@flokli told me about a feature were you can give the application some sort of "virtual environment" so home-assistant would be able to install it's dependencies with pip. Maybe this is better than listing all the requirements per hand?

(I start to think that HA was not the best choice for a first package when you want to learn packaging under NixOS cough)

dotlambda · 2018-01-13T22:18:36Z

I've gone ahead and tried running hass --open-ui but got many errors about modules that could not be found.
Therefore I packaged

netdisco,
user-agents and last but not least
home-assistant-frontend

For the last one, I use PyPI and avoid generating any JS, CSS and HTML foo from source.

user-agents depends on ua-parser, whose test I was not able to make run. They require to fetch a submodule but then still some errors occur, e.g.

======================================================================
ERROR: _regexes (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: _regexes
Traceback (most recent call last):
  File "/nix/store/4wqa1whb42khic67rv9jgip30nxmaii8-python3-3.6.4/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/tmp/nix-build-python3.6-ua-parser-0.7.3.drv-0/ua-parser-0.7.3/ua_parser/_regexes.py", line 8, in <module>
    from .user_agent_parser import (
  File "/tmp/nix-build-python3.6-ua-parser-0.7.3.drv-0/ua-parser-0.7.3/ua_parser/user_agent_parser.py", line 549, in <module>
    from ._regexes import USER_AGENT_PARSERS, DEVICE_PARSERS, OS_PARSERS
ImportError: cannot import name 'USER_AGENT_PARSERS'

Plus, I added some already packaged python packages to home-assistant's propagatedBuildInputs.

You can find the changes here: master...dotlambda:home-assistant-old
I'm now able to run hass --open-ui without any errors and with --skip-pip enabled.

What do you think, @FRidh? Should I make PRs for the 4 new python modules?
I would of course still have to add meta blocks.

FRidh · 2018-01-14T18:57:21Z

@dotlambda how many more packages are going to be added? I mean, open-ui is one thing, but what about extra modules? Should we package all of them? Who is going to maintain all these expressions? Did you read #19508?

dotlambda · 2018-01-14T19:13:39Z

No, of course we shouldn't package all dependencies.
I think that anyone wanting to use home-assistant should just package all modules they need for their use case. This way, the maintance effort will also be distributed across multiple maintainers.
And then, one can specify those using an argument called extraBuildInputs, which can be set using home-assistant.override { extraBuildInputs = [ ... ]; } or an equivalent option of the NixOS module.
Do you think that's a feasible approach, @FRidh?

flokli · 2018-01-14T19:50:28Z

I also think calling home-assistant using --skip-pip from its module in combination with a module configuration to specify extra Packages is a good idea.

If the list of packages grows too much, we might do some plugins to packages translation in there, but as a start, this might be enough.

Good to already see some of those packages landing in nixpkgs :-)

It would maybe make sense to put home-assistant-frontend behind a frontendSupport argument of the package itself (defaulting to true), as it's pretty much the first thing home-assistant wants to start ;-)

dotlambda · 2018-01-14T21:13:49Z

Why did you add vincenty to propagatedBuildInputs, @f-breidenstein? I think the dependency was removed in 2016: home-assistant/core@4b0df51

dotlambda · 2018-01-14T21:39:10Z

I have created a new version of the derivation for home-assistant, which you can find here: master...dotlambda:home-assistant.
This uses --skip-pip and my proposed extraBuildnputs.
I added all dependencies listed in setup.py as well as those that were not found when running hass --open-ui. To avoid depending on pip, the appropriate import was removed.
I didn't make the frontend optional as @flokli proposed since hass_frontend is imported as soon as hass is run, regardless of the arguments.
The main thing that still has to be done is writing a NixOS module.

I hope you don't mind that I kept you as a maintainer, @f-breidenstein.

What do you think, @FRidh?

bachp · 2018-01-14T22:03:58Z

Would it be possible to use a tool to automatically generate nix expressions from home-assistants requirements_all.txt? Somthing like https://github.com/johbo/pip2nix.

I guess home-assistant wouldn't be very useful if most of the components are missing dependencies.

dotlambda · 2018-01-14T22:17:23Z

Yes that is possible, @bachp. See https://github.com/garbas/nixpkgs-python. But then, it cannot be included in nixpkgs from my understanding.
I think it is more helpful to include a basic home-assistant package in Nixpkgs, which allows us to write a NixOS module for it. It is very easy to spot missing dependencies because hass will print error messages like ModuleNotFoundError: No module named 'netdisco'.
Maybe we could add a list of possible extraBuildInputs (together with the components that require them) on https://nixos.org/nixos/manual/options.html.

fleaz · 2018-01-15T12:15:02Z

@dotlambda, iirc I packaged everything which was mentioned in the requirements part of the setup.py from HA, but yes, it seems like it isn't there anymore.

@ALL: Sorry for the low amount of input to this discussion from my side, but regarding my limitied knownledge about nix and nixOS, I'm probably not very helpful by packaging such a beast like HA.

dotlambda · 2018-01-18T00:25:05Z

Any thoughts on this, @FRidh?

FRidh · 2018-01-18T10:33:32Z

@dotlambda I think having the following would be good:

add a skip-pip ? true parameter. By default it won't fetch, but if one wants it they can override the expression easily and use that feature.
add a extraPackages ? x: [] parameter, which is a lambda like withPackages that selects packages from the same package set that is used to build home-assistant. One can then add packages to home-assistant by first overriding the python package set, and then overriding the home-asssistant expression to add packages with the lambda. The reason for the lambda is to guarantee a working package set.

dotlambda · 2018-01-20T20:21:18Z

I have started hass another time (without an existing configuration) and took note of all components that are loaded:

history
recorder
http
frontend
api
websocket_api
system_log
introduction
persistent_notification
updater
group
tts
tts.google
config
logbook
conversation
sun
sensor
sensor.yr
cloud
automation
script
discovery
map
config.script
config.core
config.hassbian
config.customize
config.group
config.automation

These should correspond to the default config: https://github.com/home-assistant/home-assistant/blob/master/homeassistant/config.py#L64
I think we should include all of their dependencies as listet in https://github.com/home-assistant/home-assistant/blob/master/requirements_all.txt by default.

FRidh · 2018-01-23T16:21:20Z

Closing in favor of #34188. Thanks for the work!

fleaz and others added 3 commits January 9, 2018 23:13

Add f-breidenstein as maintainer

4f68f59

vincenty: init at 0.1.4

a474a71

astral: init at 1.4

f9a7fc5

fleaz requested a review from FRidh as a code owner January 9, 2018 22:31

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jan 9, 2018

home-assistant: init at 0.60.1

be48259

Tests are currently disabled due to many errors. There are also pending commits on the 'dev' branch which fixes some tests. Also home-assistant installs more dependencies dynamically on every start via pip. This still needs to be fixed.

FRidh requested changes Jan 10, 2018

View reviewed changes

fleaz added 3 commits January 11, 2018 18:10

home-assistant: Change 'patchPhase' to 'postPatch'

2d3d600

vincenty: Correct 'pythonApplication' to 'pythonPackage'

6d270c5

astral, home-assistant, vincenty: Remove 'name'

5f70c81

buildPythonPackage inherits from toPythonModule which automatically generates the name for the package with "name ? "${attrs.pname}-${attrs.version}" in the file mk-python-derivation.nix

FRidh added the 6.topic: python label Jan 11, 2018

This was referenced Jan 14, 2018

pythonPackages.user-agents: init at 1.1.0 #33855

Merged

python3Packages.netdisco: init at 1.2.3 #33859

Merged

python3Packages.pyhomematic: init at 0.1.38 #33862

Merged

FRidh added the 2.status: work-in-progress label Jan 18, 2018

This was referenced Jan 23, 2018

python3Packages.yarl: 1.0.0 -> 1.1.0 #34147

Merged

home-assistant: init at 0.62.1 #34188

Merged

FRidh closed this Jan 23, 2018

Centaurioun mentioned this pull request Mar 2, 2023

[Snyk] Fix for 15 vulnerabilities Centaurioun/nixpkgs#137

Merged

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Home Assistant #33675

[WIP] Home Assistant #33675

fleaz commented Jan 9, 2018

dotlambda commented Jan 10, 2018

FRidh Jan 10, 2018

FRidh Jan 10, 2018

FRidh Jan 10, 2018

FRidh Jan 10, 2018

FRidh Jan 10, 2018

FRidh commented Jan 11, 2018

FRidh commented Jan 11, 2018

fleaz commented Jan 12, 2018

dotlambda commented Jan 13, 2018 •

edited

FRidh commented Jan 14, 2018

dotlambda commented Jan 14, 2018 •

edited

flokli commented Jan 14, 2018

dotlambda commented Jan 14, 2018

dotlambda commented Jan 14, 2018

bachp commented Jan 14, 2018

dotlambda commented Jan 14, 2018

fleaz commented Jan 15, 2018

dotlambda commented Jan 18, 2018

FRidh commented Jan 18, 2018

dotlambda commented Jan 20, 2018 •

edited

FRidh commented Jan 23, 2018

		@@ -0,0 +1,25 @@
		{ stdenv, fetchPypi, buildPythonApplication }:

		buildPythonApplication rec {

[WIP] Home Assistant #33675

[WIP] Home Assistant #33675

Conversation

fleaz commented Jan 9, 2018

Motivation for this change

Things done

dotlambda commented Jan 10, 2018

FRidh Jan 10, 2018

Choose a reason for hiding this comment

FRidh Jan 10, 2018

Choose a reason for hiding this comment

FRidh Jan 10, 2018

Choose a reason for hiding this comment

FRidh Jan 10, 2018

Choose a reason for hiding this comment

FRidh Jan 10, 2018

Choose a reason for hiding this comment

FRidh commented Jan 11, 2018

FRidh commented Jan 11, 2018

fleaz commented Jan 12, 2018

dotlambda commented Jan 13, 2018 • edited

FRidh commented Jan 14, 2018

dotlambda commented Jan 14, 2018 • edited

flokli commented Jan 14, 2018

dotlambda commented Jan 14, 2018

dotlambda commented Jan 14, 2018

bachp commented Jan 14, 2018

dotlambda commented Jan 14, 2018

fleaz commented Jan 15, 2018

dotlambda commented Jan 18, 2018

FRidh commented Jan 18, 2018

dotlambda commented Jan 20, 2018 • edited

FRidh commented Jan 23, 2018

dotlambda commented Jan 13, 2018 •

edited

dotlambda commented Jan 14, 2018 •

edited

dotlambda commented Jan 20, 2018 •

edited