Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openzwave & pythonPackages.python_openzwave: init at 20180404 and 0.4.4 #38577

Merged
merged 4 commits into from Apr 10, 2018

Conversation

etu
Copy link
Contributor

@etu etu commented Apr 7, 2018

Motivation for this change

OpenZWave can be used with a wide range of USB-Sticks used to controll Z-Wave devices.

Python_OpenZWave can be used with homeassistant that is already packaged to be able to control and script Z-Wave devices.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@etu etu requested a review from FRidh as a code owner April 7, 2018 22:48
@etu
Copy link
Contributor Author

etu commented Apr 7, 2018

cc @dotlambda @f-breidenstein maybe?

Copy link
Member

@dotlambda dotlambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what /etc/openzwave usually contains?

Did you test that this actually works with Home Assistant?

, udev }:

let
version = "2018-04-04";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not a stable version?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the latest stable is V1.5 and was relaseaed 2016-08-15. And the repository seems to be quite active since then.

https://github.com/OpenZWave/open-zwave/releases/tag/V1.5

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like almost all distros package 1.5: https://repology.org/metapackage/openzwave/versions.
But pip install python_openzwave installs from master by default: https://github.com/OpenZWave/python-openzwave#python-openzwave-04x-is-here-.
So I guess using an unstable version should be fine.
You could however ask them what they want distributions to do on their mailing list.

makeFlagsArray+=("DESTDIR=$out")
makeFlagsArray+=("PREFIX=$out")
makeFlagsArray+=("pkgconfigdir=lib/pkgconfig")
makeFlagsArray+=("sysconfigdir=etc/openzwave")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can set this in makeFlags, no need to use preBuild. See https://nixos.org/nixpkgs/manual/#build-phase.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is much nicer, thanks 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem to work like the docs suggests:

  makeFlags = [
    "DESTDIR=$(out)"
    "PREFIX=$(out)"
    "pkgconfigdir=lib/pkgconfig"
    "sysconfigdir=etc/openzwave"
  ];

Doesn't build and it seems like the issue is with $out because it outputs this:

build flags: SHELL=/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin/bash DESTDIR=\$\(out\) PREFIX=\$\(out\) pkgconfigdir=lib/pkgconfig sysconfigdir=etc/openzwave

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a problem sometimes. Try without parantheses or use "DESTDIR=\${out}".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't help at all :D

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried myself and it gives me a different error when leaving out parantheses: mv: cannot stat '/nix/store/nhy5v0l6yccgi2wfp01d457dp9g6ql9w-openzwave-2018-04-04//nix/store/nhy5v0l6yccgi2wfp01d457dp9g6ql9w-openzwave-2018-04-04/bin': No such file or directory.

Copy link
Contributor Author

@etu etu Apr 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's the fixupPhase. If it's not getting the prefix and destdir it just don't put any files in the store path.


preBuild = ''
makeFlagsArray+=("DESTDIR=$out")
makeFlagsArray+=("PREFIX=$out")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try removing this line and you shouldn't need the fixup anymore.

Copy link
Contributor Author

@etu etu Apr 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing that created a whole range of other issues, because there's a thing that doesn't respect the DESTDIR...

So currently it creates:

/nix/store/xyz-openzwave-2018-04-04/lib/pkgconfig/libopenzwave.pc
/nix/store/xyz-openzwave-2018-04-04/nix/store/xyz-openzwave-2018-04-04/<rest-of-the-files>

I haven't found a good workaround for that :/

But I can dig some more now when I have gotten some sleep. I've might also been very tierd and fuzzy during the late hours yesterday and don't remember all the details.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've gone through all the things now.

I've managed to set PREFIX= (as in empty string to not get the default /usr/local/bin).

And got rid of most of fixup but had to patch the pkgconfig-file so the python library can be built.

I have tested it with the python library on x86_64 with home assistant as well. So at the moment it works and is slightly less ugly :)


meta = with stdenv.lib; {
description = "C++ library to control Z-Wave Networks via a USB Z-Wave Controller";
homepage = https://github.com/OpenZWave/open-zwave;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe http://www.openzwave.net/ instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've honestly missed that that one exists because it's not coming up top when I've been doing searches :D

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda /etc/openzwave contains a bunch of XML files that are configuration for OpenZWave. It's needed to be able to use OpenZWave at all. All utilities and libs in all languages depends on it.

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda As for your other question:

Did you test that this actually works with Home Assistant?

Yes, I have it running here on my desktop as well as a raspberry pi. I had to change one little thing in the home-assistant derivation (for some depenedency), and had to add some configuration myself to the config file.

Diff of home-assistant:

diff --git a/pkgs/servers/home-assistant/default.nix b/pkgs/servers/home-assistant/default.nix
index 75f8b309a40..e1814317463 100644
--- a/pkgs/servers/home-assistant/default.nix
+++ b/pkgs/servers/home-assistant/default.nix
@@ -72,6 +72,7 @@ in with py.pkgs; buildPythonApplication rec {
   propagatedBuildInputs = [
     # From setup.py
     requests pyyaml pytz pip jinja2 voluptuous typing aiohttp async-timeout astral certifi attrs
+    pydispatcher
     # From http, frontend and recorder components
     sqlalchemy aiohttp-cors hass-frontend
   ] ++ componentBuildInputs ++ extraBuildInputs;

configuration:

  # Enable Home Assistant, open port and add the hass user to the dialout group
  services.home-assistant.enable = true;
  services.home-assistant.package = pkgs.home-assistant.override {
    extraPackages = ps: [ pkgs.python36Packages.python_openzwave ];
  };

  networking.firewall.allowedTCPPorts = [ 8123 ];
  users.extraUsers.hass.extraGroups = [ "dialout" ];

Yaml for home-assistant:

zwave:
  usb_path: /dev/serial/by-id/usb-0658_0200-if00
  config_path: /nix/store/dlgdch7iwdd6ky4z1b4wpzrn1mg8w4i5-openzwave-2018-04-04/etc/openzwave/

As you can see it's not perfect, especially not the part that it's required to configure the config_path. It's an optional parameter so it looks in default places. I've yet to figure out how to set these defaults.

First I thought that I did by patching Options.cpp in openzwave, but that didn't cut it. Might need a patch in the python lib or in home-assistant to do it automatically...

makeFlagsArray+=("PREFIX=")
makeFlagsArray+=("pkgconfigdir=lib/pkgconfig")
makeFlagsArray+=("sysconfigdir=etc/openzwave")
'';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dotlambda At least I've got "rid" of the prefix and lots of file moving in the fixup, but still not sure how to use makeFlags... At least it's quite alot cleaner overall now....

@dotlambda
Copy link
Member

extraPackages = ps: [ pkgs.python36Packages.python_openzwave ];

That shoud be extraPackages = ps: with ps; [ python_openzwave pydispatcher ];.
However, after running pkgs/servers/home-assistant/parse-requirements.py, these dependencies will be added automatically when using the services.home-assistant.config option or extraComponents = [ "zwave" ].

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

  services.home-assistant.package = pkgs.home-assistant.override {
    extraPackages = ps: [ python_openzwave ];
    extraComponents = [ "zwave" ];
  };

results in: error: undefined variable 'python_openzwave' at /etc/nixos/configuration.nix:44:27

and if I remove my patch from home-assistant/default.nix and use this configuration:

  # Enable Home Assistant, open port and add the hass user to the dialout group
  services.home-assistant.enable = true;
  services.home-assistant.package = pkgs.home-assistant.override {
    extraPackages = ps: [ pkgs.python36Packages.python_openzwave ];
    extraComponents = [ "zwave" ];
  };

  networking.firewall.allowedTCPPorts = [ 8123 ];
  users.extraUsers.hass.extraGroups = [ "dialout" ];

It just doesn't work inside of home-assistant, and I can find the following error in the debug log:

Error during setup of component zwave

Traceback (most recent call last):
  File "/nix/store/vv7ay3lp7l19japagig92s93f8cjlggg-homeassistant-0.65.5/lib/python3.6/site-packages/homeassistant/setup.py", line 145, in _async_setup_component
    component.setup, hass, processed_config)
  File "/nix/store/b8gd0cbvkm59x8flbc53bvsvmskyig5a-python3-3.6.4/lib/python3.6/asyncio/futures.py", line 327, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/nix/store/b8gd0cbvkm59x8flbc53bvsvmskyig5a-python3-3.6.4/lib/python3.6/asyncio/tasks.py", line 250, in _wakeup
    future.result()
  File "/nix/store/b8gd0cbvkm59x8flbc53bvsvmskyig5a-python3-3.6.4/lib/python3.6/asyncio/futures.py", line 243, in result
    raise self._exception
  File "/nix/store/b8gd0cbvkm59x8flbc53bvsvmskyig5a-python3-3.6.4/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/nix/store/vv7ay3lp7l19japagig92s93f8cjlggg-homeassistant-0.65.5/lib/python3.6/site-packages/homeassistant/components/zwave/__init__.py", line 250, in setup
    from pydispatch import dispatcher
ModuleNotFoundError: No module named 'pydispatch'

@dotlambda
Copy link
Member

You need to copy exactly that: extraPackages = ps: with ps; [ python_openzwave pydispatcher ];

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

Oh, yeah, that did work :)

@@ -20532,6 +20532,8 @@ with pkgs;

moltengamepad = callPackage ../misc/drivers/moltengamepad { };

openzwave = callPackage ../misc/drivers/openzwave { };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this belongs in pkgs/development/libraries.

@dotlambda
Copy link
Member

How about so?

  installFlags = [
    "PREFIX="
    "pkgconfigdir=lib/pkgconfig"
    "sysconfigdir=etc/openzwave"
  ];

  preInstall = ''
    installFlagsArray+=("DESTDIR=$out")
  '';

@etu Please also try fixing the Fontconfig error: Cannot load default config file errors.

Maybe @peterhoeg or @FRidh want to have a look at this? I'm certainly not an expert when it comes to C libraries.

@dotlambda
Copy link
Member

@etu I added a commit which should fix extraComponents = [ "zwave" ];.

names.append(req[len('python-'):])
for name in names:
# treat "-" and "_" equally
name = re.sub('[-_]', '[-_]', name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replacement is a regex as well?

Copy link
Member

@dotlambda dotlambda Apr 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, because I need to replace both, - and _, at the same time. I don't think string.replace() can do that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could do

name = name.replace('_', '-')
name = name.replace('-', '[-_]')

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda Re fontconfig: Here I have a build log of openzwave and python_openzwave: https://gist.github.com/etu/6bdba2f432a839566e6806b5d72e3773 -- which warnings do you refer to?

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda
I've copied the files of that commit to my own nixpkgs and tried to change the config to:

  # Enable Home Assistant, open port and add the hass user to the dialout group
  services.home-assistant.enable = true;
  services.home-assistant.package = pkgs.home-assistant.override {
    extraComponents = [ "zwave" ];
  };

  networking.firewall.allowedTCPPorts = [ 8123 ];
  users.extraUsers.hass.extraGroups = [ "dialout" ];

That resulted in a python error log from home-assistant that complaints about a missing python module named openzwave. So I'm not sure I trust extraComponents at all :p

@dotlambda
Copy link
Member

Maybe that error only appears when enabling sandboxing.
log.txt

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda Seems so, I can reproduce it by enabling sandboxing.

@dotlambda
Copy link
Member

dotlambda commented Apr 8, 2018

@etu I investigated why extraComponents = [ "zwave" ]; fails for you. Can you try again with the commit I pushed?

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda

This patch does get rid of the fontconfig warnings, but I'm not sure where to put the environment variables because preInstall feels like the wrong place.

diff --git a/pkgs/development/libraries/openzwave/default.nix b/pkgs/development/libraries/openzwave/default.nix
index 0d5feae2419..3c18849f1c5 100644
--- a/pkgs/development/libraries/openzwave/default.nix
+++ b/pkgs/development/libraries/openzwave/default.nix
@@ -1,5 +1,5 @@
 { stdenv, fetchFromGitHub
-, doxygen, libxml2, git, which, pkgconfig, graphviz-nox
+, doxygen, libxml2, git, which, pkgconfig, graphviz-nox, fontconfig
 , udev }:
 
 let
@@ -15,7 +15,7 @@ in stdenv.mkDerivation rec {
     sha256 = "0yby8ygzjn5zp5vhysxaadbzysqanwd2zakz379299qs454pr2h9";
   };
 
-  nativeBuildInputs = [ doxygen libxml2 git which pkgconfig graphviz-nox ];
+  nativeBuildInputs = [ doxygen libxml2 git which pkgconfig graphviz-nox fontconfig ];
   buildInputs = [ udev ];
 
   installFlags = [
@@ -26,6 +26,9 @@ in stdenv.mkDerivation rec {
 
   preInstall = ''
     installFlagsArray+=("DESTDIR=$out")
+
+    export FONTCONFIG_FILE=${fontconfig.out}/etc/fonts/fonts.conf
+    export FONTCONFIG_PATH=${fontconfig.out}/etc/fonts/
   '';
 
   postPatch = ''

I will try your change.

@dotlambda
Copy link
Member

You can make these a part of the Nix expression, i.e.

preInstall = ''
  installFlagsArray+=("DESTDIR=$out")'
'';

FONTCONFIG_FILE = "${fontconfig.out}/etc/fonts/fonts.conf";
FONTCONFIG_PATH = "${fontconfig.out}/etc/fonts/";

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda Oh, ok. Rebased and pushed.

I've tested that change in home-assistant/default.nix and no such luck. Now it's missing pydispatcher instead 😕

@etu
Copy link
Contributor Author

etu commented Apr 8, 2018

@dotlambda I've just confirmed that my patch here: https://github.com/NixOS/nixpkgs/pull/38577/files#diff-588a05cff755242c814f80436a1184faR23

Works properly in home-assistant. So one doesn't need to configure that. That makes everything better I'd say 😄

@peterhoeg
Copy link
Member

Sorry about being late to the party. Please don't merge this yet, I have a few comments.

@peterhoeg
Copy link
Member

peterhoeg commented Apr 9, 2018

I made a few changes to address a few issues - you can see them here: peterhoeg/nixpkgs@75aaa71...peterhoeg:zw

  • udev is an alias for systemd, so just use that
  • while technically not wrong, fiddling with installFlags as well as the installFlagsArray bash variable is a little ugly so instead we override the phase which while being a bit more verbose, makes it a lot clearer what is going on (IMO)
  • substituteInPlace i/o sed as it's much easier to read
  • pcfile inside ozw_config wasn't being substituted correctly
  • do not override /etc/openzwave location so we can set it from nixos
  • do not adjust pkg_config_path. If openzwave had its .pc files under usr/lib, that should be fixed instead but that's not the case.
  • drop git as input - we are not using it

@etu
Copy link
Contributor Author

etu commented Apr 9, 2018

@peterhoeg Oh, nice. I'm gonna test through it all tonight (in like 12-13 hours when I get home from work and other things). You can push this to my branch if you want :)

@peterhoeg peterhoeg changed the title Package OpenZWave and Python_OpenZWave openzwave & pythonPackages.python_openzwave: init at 20180404 and 0.4.4 Apr 9, 2018
@peterhoeg
Copy link
Member

Pushed. You probably want to squash those commits into the relevant commits on your side.

@etu
Copy link
Contributor Author

etu commented Apr 9, 2018

@peterhoeg git is used by openzwave build to find version-number from it's git-repo. So without it I got a warning that it doesn't exist. I don't think it matters very much for the actual usage though.

Getting warnings like this:

which: no git in (/nix/store/kp7q1wgj2n8s56vkpnamz708wvdr4qhy-doxygen-1.8.14/bin:/nix/store/gdlda7gxf1iwg6j0b4m40mg8w4gi2rmw-freetype-2.7.1-dev/bin:/nix/store/9wj6nikq0h48gmh8dhdr6dyqd54xgzk0-bzip2-1.0.6.0.1-bin/bin:/nix/store/k0k1pv36m1himk885kjl7hll8l9nqrmk-libpng-apng-1.6.34-dev/bin:/nix/store/abkbqmm7zspm22fbppxb2mpql27yzf2m-fontconfig-2.12.1-bin/bin:/nix/store/4s2z334lh2nyp25vgkf5rsjyppvzcf81-graphviz-2.40.1/bin:/nix/store/ikqbd879irr8dvrg5ln8dw8735n6d3i1-libxml2-2.9.7-dev/bin:/nix/store/r5l4lqv24cgj0phb27fybg0213bc5p11-libxml2-2.9.7-bin/bin:/nix/store/378vwwffwdp4pir08mfd6iz3nmj4zf7w-pkg-config-0.29.2/bin:/nix/store/dix5whv5hlhjjrvrr2l7w85qkrw7hxxp-which-2.21/bin:/nix/store/7inxdf88zz2msxmd3m6dhhxdsxkbad5k-patchelf-0.9/bin:/nix/store/n9kdl63krsqix92b9hz5hwfy2322zi3h-paxctl-0.9/bin:/nix/store/4r5kszyy0iirc5agfah45lvz7mnnsrb4-gcc-wrapper-7.3.0/bin:/nix/store/cm5znbhvylrh702hhsy9hnad7wk3v7xv-gcc-7.3.0/bin:/nix/store/jy6igfw7xawlhsyk7f7cjng8hd0dqdpf-glibc-2.26-131-bin/bin:/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29/bin:/nix/store/1is4c0vfcs0q5i3ygij21y6z713lihw9-binutils-wrapper-2.28.1/bin:/nix/store/fzcs0fn6bb04m82frhlb78nc03ny3w55-binutils-2.28.1/bin:/nix/store/jy6igfw7xawlhsyk7f7cjng8hd0dqdpf-glibc-2.26-131-bin/bin:/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29/bin:/nix/store/8mii4rkzpmkcv3hyb4334sfm6jhjgscr-systemd-237/bin:/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29/bin:/nix/store/3ixa5cn578zz4pr81a0rdkbi4dal748a-findutils-4.6.0/bin:/nix/store/p8x2s5ch40yybrkl57swp4b0rs2bnaki-diffutils-3.6/bin:/nix/store/r0wkp0vkw6wjfl0skmdpykabgj1bkyn2-gnused-4.4/bin:/nix/store/xd4m1v7q7kn902p8hvnfn79c2wjijlif-gnugrep-3.1/bin:/nix/store/hdpv6al52bz14mf8ffnjkp62f8haq3vd-gawk-4.2.1/bin:/nix/store/rng0r3whml1jql1s6ps11v7f8cxmna04-gnutar-1.30/bin:/nix/store/mrwabgy72d3ab6c80d21gmsw5lkixjnz-gzip-1.9/bin:/nix/store/9wj6nikq0h48gmh8dhdr6dyqd54xgzk0-bzip2-1.0.6.0.1-bin/bin:/nix/store/qz5lb9xwxy8rws569ra7v0i1asbk4g1l-gnumake-4.2.1/bin:/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin:/nix/store/h69mmb2cix9lyk4162wa72p0jcmw157x-patch-2.7.6/bin:/nix/store/qh62aip5awsbkgs4yzjwpdgzwpal72fd-xz-5.2.3-bin/bin)

@peterhoeg
Copy link
Member

Correct, you get the warning but $src is not a checked out git repo but simply a directory, so it cannot do anything there anyway.

@etu
Copy link
Contributor Author

etu commented Apr 9, 2018

@peterhoeg I've rebased and squashed your changed. And I'll fixed some dependency issues for the CLI tools for the python library and have pushed it. I hope it looks good now :)

@peterhoeg
Copy link
Member

Looks good. @dotlambda/@FRidh do you want this to go via staging?

@dotlambda
Copy link
Member

I manually checked that it won't create merge conflict, so I guess merging into master is fine.

@dotlambda dotlambda merged commit 492ab20 into NixOS:master Apr 10, 2018
@dotlambda
Copy link
Member

Thanks a lot @etu!

@etu etu deleted the openzwave branch April 10, 2018 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants