pythonPackages.image-match: init at 1.1.2 #43723

CMCDragonkai · 2018-07-18T08:04:44Z

Motivation for this change

Added image-match.

One problem with this is that image-match requires elasticsearch at below 6.0.0 according to setup.py. Since there is no elasticsearch below 6.0.0 in nixpkgs, I just ignored that dependency. I use image-match without touching the elasticsearch functionality.

I asked if they could bump up the version constraint anyway: rhsimplex/image-match#97

Furthermore the I could not use fetchPypi due to: #43721

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

FRidh

I use image-match without touching the elasticsearch functionality.

That's ok.

Meta is missing. Also, I like you to be maintainer of packages you add.

FRidh · 2018-07-18T08:05:15Z

pkgs/development/python-modules/image-match/default.nix

+    sha256 = "0vlmpidmhkpgdzw2k03x5layhijcrjpmyfd93yv2ls77ihz00ix5";
+  };
+
+  buildInputs = [ pytestrunner ];


checkInputs

The pytestrunner is part of the setup_requires, I thought that meant it was required regardless of tests.

You're correct, buildInputs it is.

Looking at their setup.py there seems to be no need for having it in setup_requires.

FRidh · 2018-07-18T08:05:29Z

pkgs/development/python-modules/image-match/default.nix

+
+  # ignore elasticsearch requirement due to version incompatibility
+  installFlags = [ "--no-deps" ];
+  doCheck = false;


include motivation for disabling tests

CMCDragonkai · 2018-07-18T08:17:14Z

Added everything.

FRidh · 2018-07-18T08:22:45Z

@GrahamcOfBorg build pythonPackages.image-match python3Packages.image-match

GrahamcOfBorg · 2018-07-18T08:24:21Z

Success on x86_64-linux (full log)

Attempted: pythonPackages.image-match, python3Packages.image-match

Partial log (click to expand)

Successfully installed image-match-1.1.2
/build/source
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/k6whvmybmf3ijnacgqbdldc56r9yakf5-python3.6-image-match-1.1.2
strip is /nix/store/90vmpr41dzsx350k5argycaf693hnl1l-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/k6whvmybmf3ijnacgqbdldc56r9yakf5-python3.6-image-match-1.1.2/lib
patching script interpreter paths in /nix/store/k6whvmybmf3ijnacgqbdldc56r9yakf5-python3.6-image-match-1.1.2
checking for references to /build in /nix/store/k6whvmybmf3ijnacgqbdldc56r9yakf5-python3.6-image-match-1.1.2...
/nix/store/4ymx0ysdvlpph3mh7zmx0p4j398q51b0-python2.7-image-match-1.1.2
/nix/store/k6whvmybmf3ijnacgqbdldc56r9yakf5-python3.6-image-match-1.1.2

GrahamcOfBorg · 2018-07-18T08:25:31Z

Success on x86_64-darwin (full log)

Attempted: pythonPackages.image-match, python3Packages.image-match

Partial log (click to expand)

Processing ./image_match-1.1.2-py3-none-any.whl
Installing collected packages: image-match
Successfully installed image-match-1.1.2
/private/tmp/nix-build-python3.6-image-match-1.1.2.drv-0/source
post-installation fixup
strip is /nix/store/8axizw5mf6dx7wip65nbkyrmlkhmjhc5-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/2v0s60m06qkgbm0iska585wcjrw9k6qy-python3.6-image-match-1.1.2/lib
patching script interpreter paths in /nix/store/2v0s60m06qkgbm0iska585wcjrw9k6qy-python3.6-image-match-1.1.2
/nix/store/kq70mhdmgqsqk3sq00s2bfb7idf4wbmg-python2.7-image-match-1.1.2
/nix/store/2v0s60m06qkgbm0iska585wcjrw9k6qy-python3.6-image-match-1.1.2

GrahamcOfBorg · 2018-07-18T09:23:32Z

Failure on aarch64-linux (full log)

Attempted: pythonPackages.image-match, python3Packages.image-match

Partial log (click to expand)

cannot build derivation '/nix/store/kz0br1c8zpdwr5xwjqkgwgnyn2hhrwl8-python3.6-partd-0.3.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/1kp728zwsijrmvkym64f6666bj4rv4yr-python3.6-dask-0.18.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/bzd05rg6kwlf6bgvk2cxs9lfi1903dx6-python3.6-scikit-image-0.12.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/mjqvkpwwscfzw52mwxgv2644nzh408kz-python3.6-image-match-1.1.2.drv': 1 dependencies couldn't be built
building of '/nix/store/1hg80yzbpk5md5amqsmc7sq0psxj211r-python2.7-pandas-0.23.1.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/zhn8hw8xpd7ksd860srga1s15m9qwiis-python2.7-partd-0.3.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9505xb5zmxp025hk6dh4fbwjrbw45gav-python2.7-dask-0.18.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/w6wfw7a6wr58iy4g2qzzipxirwsdqk79-python2.7-scikit-image-0.12.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/7vc263sqm984gy4h7plyy7z4sj3lmwjf-python2.7-image-match-1.1.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/7vc263sqm984gy4h7plyy7z4sj3lmwjf-python2.7-image-match-1.1.2.drv', '/nix/store/mjqvkpwwscfzw52mwxgv2644nzh408kz-python3.6-image-match-1.1.2.drv' failed

CMCDragonkai · 2018-07-18T13:03:17Z

Fixed license issue.

CMCDragonkai · 2018-07-19T06:24:46Z

@FRidh For any derivation using this package as an *input. They also have to make sure to set:

      installFlags = [ "--no-deps" ];
      doCheck = false;

On their derivation as well, since the settings for this derivation don't appear to propagate.

This is not really nice. Maybe we should have elasticsearch 5.0.0 packaged as well in nixpkgs then?

FRidh · 2018-07-19T07:03:11Z

pkgs/development/python-modules/image-match/default.nix

+  ];
+
+  # ignore elasticsearch requirement due to version incompatibility
+  installFlags = [ "--no-deps" ];


it's better to patch setup.py / requirements.txt / whatever file to remove the specific dependency.

CMCDragonkai · 2018-07-19T07:11:04Z

Good idea. Which phase would that be? And do I use sed?

FRidh · 2018-07-19T07:16:55Z

`postPatch` and `substituteInPlace`

…

On Thu, Jul 19, 2018 at 9:11 AM, Roger Qiu ***@***.***> wrote: Good idea. Which phase would that be? And do I use sed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43723 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACB878TyUi8rUPqWgvU2U19xKgG2FmOKks5uIDELgaJpZM4VUILz> .

CMCDragonkai · 2018-07-19T07:59:16Z

@FRidh I have done that now by removing the elasticsearch requirement from the setup.py. Tests still will fail due to the lack of elasticsearch, but at least this is more robust.

Note that I noticed that I have to actually add in the dask dependency for some reason. I wonder if it is meant to be a propagated dependency or only a buildInput?

CMCDragonkai · 2018-07-20T02:16:12Z

I added dask to buildInputs instead of propagatedBuildInputs. This works, and was based on how scikitimage derivation was done.

CMCDragonkai · 2018-07-20T03:50:45Z

@FRidh Can you confirm that moving dask to buildInputs is the correct idea? The package does build without it, however while other packages can use image-match without the dask dependency, they cannot install it unless they also specify dask in their buildInputs. So maybe it's better that dask should be propagated, even if it isn't used.

Basically any subsequent derivation buildPythonPackage or buildPythonApplication will fail to install the dependency since it will try to build the setup.py which will try to install dask. If they put dask in their buildInputs it will work.

However it doesn't make much sense that whenever scikitimage is installed, it only works if dask is also brought in. If this keeps happening, then scikitimage should just have dask as well.

FRidh · 2018-07-20T05:54:45Z

I've pushed 5fba38d. See that commit and previous ones on what I did with dask.

CMCDragonkai · 2018-07-20T06:18:46Z

I see. However that means dask is no longer needed in the package function in default.nix. I think you forgot to remove it.

CMCDragonkai requested a review from FRidh as a code owner July 18, 2018 08:04

FRidh requested changes Jul 18, 2018

View reviewed changes

CMCDragonkai force-pushed the image-match branch from e2134a0 to c073f32 Compare July 18, 2018 08:14

GrahamcOfBorg added the 6.topic: python label Jul 18, 2018

CMCDragonkai force-pushed the image-match branch from c073f32 to 664077d Compare July 18, 2018 13:03

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jul 18, 2018

FRidh reviewed Jul 19, 2018

View reviewed changes

CMCDragonkai force-pushed the image-match branch from 664077d to 7fdefd1 Compare July 19, 2018 07:58

pythonPackages.image-match: init at 1.1.2

627145c

CMCDragonkai force-pushed the image-match branch from 7fdefd1 to 627145c Compare July 19, 2018 11:10

CMCDragonkai mentioned this pull request Jul 20, 2018

pythonPackages.imgaug: init at 0.2.6 #43720

Merged

9 tasks

FRidh closed this Jul 20, 2018

CMCDragonkai deleted the image-match branch August 8, 2018 11:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.image-match: init at 1.1.2 #43723

pythonPackages.image-match: init at 1.1.2 #43723

CMCDragonkai commented Jul 18, 2018 •

edited

FRidh left a comment

FRidh Jul 18, 2018

CMCDragonkai Jul 18, 2018

FRidh Jul 18, 2018

FRidh Jul 18, 2018

CMCDragonkai commented Jul 18, 2018

FRidh commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

CMCDragonkai commented Jul 18, 2018

CMCDragonkai commented Jul 19, 2018

FRidh Jul 19, 2018

CMCDragonkai commented Jul 19, 2018

FRidh commented Jul 19, 2018 via email

CMCDragonkai commented Jul 19, 2018

CMCDragonkai commented Jul 20, 2018

CMCDragonkai commented Jul 20, 2018 •

edited

FRidh commented Jul 20, 2018

CMCDragonkai commented Jul 20, 2018

pythonPackages.image-match: init at 1.1.2 #43723

pythonPackages.image-match: init at 1.1.2 #43723

Conversation

CMCDragonkai commented Jul 18, 2018 • edited

Motivation for this change

Things done

FRidh left a comment

Choose a reason for hiding this comment

FRidh Jul 18, 2018

Choose a reason for hiding this comment

CMCDragonkai Jul 18, 2018

Choose a reason for hiding this comment

FRidh Jul 18, 2018

Choose a reason for hiding this comment

FRidh Jul 18, 2018

Choose a reason for hiding this comment

CMCDragonkai commented Jul 18, 2018

FRidh commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

GrahamcOfBorg commented Jul 18, 2018

CMCDragonkai commented Jul 18, 2018

CMCDragonkai commented Jul 19, 2018

FRidh Jul 19, 2018

Choose a reason for hiding this comment

CMCDragonkai commented Jul 19, 2018

FRidh commented Jul 19, 2018 via email

CMCDragonkai commented Jul 19, 2018

CMCDragonkai commented Jul 20, 2018

CMCDragonkai commented Jul 20, 2018 • edited

FRidh commented Jul 20, 2018

CMCDragonkai commented Jul 20, 2018

CMCDragonkai commented Jul 18, 2018 •

edited

CMCDragonkai commented Jul 20, 2018 •

edited