haskellPackages.tensorflow-mnist-input-data: fetch #46103

teto · 2018-09-05T15:19:48Z

Motivation for this change

Prefetch data so that the package doesn't try to connect to the internet.
See #46029

nix-build -A haskellPackages.tensorflow-mnist ~/nixpkgs still fails though because of

Setup: Encountered missing dependencies:
tensorflow-mnist-input-data -any

definition of the package tensorflow-mnist seems

       broken = true;
     }) {tensorflow-mnist-input-data = null;};

@peti git blame says that code is generated. Anyway to not set tensorflow-mnist-input-data to null ?

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Prefetch data so that the package doesn't try to connect to the internet.

GrahamcOfBorg · 2018-09-05T15:33:53Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskellPackages.tensorflow-mnist-input-data

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-05T15:35:42Z

Success on x86_64-linux (full log)

Attempted: haskellPackages.tensorflow-mnist-input-data

Partial log (click to expand)

wrong ELF type
shrinking RPATHs of ELF executables and libraries in /nix/store/fp6v86qrabyin8lrxm8zcrpdzvsbf1p1-tensorflow-mnist-input-data-0.1.0.0-data
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/fp6v86qrabyin8lrxm8zcrpdzvsbf1p1-tensorflow-mnist-input-data-0.1.0.0-data
checking for references to /build in /nix/store/fp6v86qrabyin8lrxm8zcrpdzvsbf1p1-tensorflow-mnist-input-data-0.1.0.0-data...
shrinking RPATHs of ELF executables and libraries in /nix/store/7b13qcpm3sf87jz5g8cixykkk78hgl1z-tensorflow-mnist-input-data-0.1.0.0-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/7b13qcpm3sf87jz5g8cixykkk78hgl1z-tensorflow-mnist-input-data-0.1.0.0-doc
checking for references to /build in /nix/store/7b13qcpm3sf87jz5g8cixykkk78hgl1z-tensorflow-mnist-input-data-0.1.0.0-doc...
/nix/store/a53hbml3pfp0qjsb73m6n16ylajw8jx4-tensorflow-mnist-input-data-0.1.0.0

basvandijk

Thanks for this fix. I should have tested building in a sandbox before committing the tensorflow packages.

basvandijk · 2018-09-06T04:51:15Z

pkgs/development/haskell-modules/configuration-tensorflow.nix

+      urlPrefix = "http://yann.lecun.com/exdb/mnist/";
+
+      # File names relative to 'urlPrefix' and their sha256.
+      fileInfos = [


I think using an attribute set here which maps file names to their hashes is a bit clearer and safer to use.

Note I do something similar in the opencv package.

basvandijk · 2018-09-06T04:53:34Z

pkgs/development/haskell-modules/configuration-tensorflow.nix

@@ -71,6 +95,9 @@ in
        base bytestring Cabal cryptonite directory filepath HTTP
        network-uri
      ];
+      preConfigure = pkgs.lib.strings.concatStringsSep "\n" (
+          map (x: "cp ${x} data/$(stripHash ${x})") downloads


Maybe use symbolic links instead of copying the files to prevent duplication.

basvandijk · 2018-09-06T06:05:45Z

I've no idea where the broken = true and the tensorflow-mnist-input-data = null override in:

"tensorflow-mnist" = 
  ...
       broken = true;
     }) {tensorflow-mnist-input-data = null;};

are coming from. tensorflow-mnist is not mentioned in configuration-hackage2nix.yaml nor in cabal2nix.

basvandijk · 2018-09-06T06:30:48Z

After reading the source I understand that packages are marked broken when they have missing dependencies. The tensorflow-mnist-input-data dependency of tensorflow-mnist is probably considered missing since it's not published on Hackage.

The best solution is to have an override for tensorflow-mnist where we get rid of the broken = true flag and point tensorflow-mnist-input-data to the actual package.

We should also report an issue at https://github.com/tensorflow/haskell/issues to publish tensorflow-mnist-input-data to Hackage.

basvandijk · 2018-09-06T07:06:28Z

I've now made these changes in this PR. I also reverted my libtensorflow: 1.9 -> 1.10 upgrade since that broke the Haskell tensorflow packages and libtensorflow is only used by the Haskell packages so this revert doesn't break anything else.

I also removed the tensorflow packages from dont-distribute-packages so they will be build on Hydra again after the next hackage2nix run.

@GrahamcOfBorg build haskellPackages.tensorflow-mnist

GrahamcOfBorg · 2018-09-06T07:06:53Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskellPackages.tensorflow-mnist

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-06T07:15:48Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskellPackages.tensorflow-mnist-input-data

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

…specify the data files This is more understandable and safer than using a nested list.

…of copying This prevents duplication in the nix store.

…w-mnist-input-data

This reverts commit 93ed13f. libtensorflow is only used by the Haskell tensorflow packages and they don't work with tensorflow-1.10 yet. So the easiest solution is to just revert this commit and add it back when they do gain support.

This reverts commit 7139911. libtensorflow is only used by the Haskell tensorflow packages and they don't work with tensorflow-1.10 yet. So the easiest solution is to just revert this commit and add it back when they do gain support.

…successfully

GrahamcOfBorg · 2018-09-06T07:16:33Z

Success on x86_64-linux (full log)

Attempted: haskellPackages.tensorflow-mnist-input-data

Partial log (click to expand)

wrong ELF type
shrinking RPATHs of ELF executables and libraries in /nix/store/7dhijl4mh1b5rnbvdn9s2lxxqp1ifkq9-tensorflow-mnist-input-data-0.1.0.0-data
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/7dhijl4mh1b5rnbvdn9s2lxxqp1ifkq9-tensorflow-mnist-input-data-0.1.0.0-data
checking for references to /build in /nix/store/7dhijl4mh1b5rnbvdn9s2lxxqp1ifkq9-tensorflow-mnist-input-data-0.1.0.0-data...
shrinking RPATHs of ELF executables and libraries in /nix/store/h711g6cmliz4qvwx32i4gpdlfl3sdai2-tensorflow-mnist-input-data-0.1.0.0-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/h711g6cmliz4qvwx32i4gpdlfl3sdai2-tensorflow-mnist-input-data-0.1.0.0-doc
checking for references to /build in /nix/store/h711g6cmliz4qvwx32i4gpdlfl3sdai2-tensorflow-mnist-input-data-0.1.0.0-doc...
/nix/store/jag6yf2x30x2j5kmljk4xsm2d83brig0-tensorflow-mnist-input-data-0.1.0.0

GrahamcOfBorg · 2018-09-06T07:25:26Z

Success on x86_64-darwin (full log)

Attempted: haskellPackages.tensorflow-mnist

Partial log (click to expand)

not in the system search path.
post-installation fixup
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/csdn5ah2dry8k8l3g6mxzaa9njkyg4dd-tensorflow-mnist-0.1.0.0/lib  /nix/store/csdn5ah2dry8k8l3g6mxzaa9njkyg4dd-tensorflow-mnist-0.1.0.0/bin
patching script interpreter paths in /nix/store/csdn5ah2dry8k8l3g6mxzaa9njkyg4dd-tensorflow-mnist-0.1.0.0
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/pv261j64n6wi5kfjc0azyh3nwwdcc4ma-tensorflow-mnist-0.1.0.0-data
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/17gml5pz7cmibflqphfy6pn3bcwqvbrh-tensorflow-mnist-0.1.0.0-doc
/nix/store/csdn5ah2dry8k8l3g6mxzaa9njkyg4dd-tensorflow-mnist-0.1.0.0

GrahamcOfBorg · 2018-09-06T07:26:50Z

Success on x86_64-linux (full log)

Attempted: haskellPackages.tensorflow-mnist

Partial log (click to expand)

wrong ELF type
shrinking RPATHs of ELF executables and libraries in /nix/store/pxjiq3v71n6bzrgjr9xrp9228303xb8f-tensorflow-mnist-0.1.0.0-data
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/pxjiq3v71n6bzrgjr9xrp9228303xb8f-tensorflow-mnist-0.1.0.0-data
checking for references to /build in /nix/store/pxjiq3v71n6bzrgjr9xrp9228303xb8f-tensorflow-mnist-0.1.0.0-data...
shrinking RPATHs of ELF executables and libraries in /nix/store/fjglc800w2gcchald3q5wsmrnv3i33kz-tensorflow-mnist-0.1.0.0-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/fjglc800w2gcchald3q5wsmrnv3i33kz-tensorflow-mnist-0.1.0.0-doc
checking for references to /build in /nix/store/fjglc800w2gcchald3q5wsmrnv3i33kz-tensorflow-mnist-0.1.0.0-doc...
/nix/store/bp3zq3vj9a6jj9y45pcgi3j0f79w9a89-tensorflow-mnist-0.1.0.0

GrahamcOfBorg · 2018-09-06T07:28:55Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskellPackages.tensorflow-mnist-input-data

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-06T07:30:31Z

Success on x86_64-linux (full log)

Attempted: haskellPackages.tensorflow-mnist-input-data

Partial log (click to expand)

wrong ELF type
shrinking RPATHs of ELF executables and libraries in /nix/store/7dhijl4mh1b5rnbvdn9s2lxxqp1ifkq9-tensorflow-mnist-input-data-0.1.0.0-data
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/7dhijl4mh1b5rnbvdn9s2lxxqp1ifkq9-tensorflow-mnist-input-data-0.1.0.0-data
checking for references to /build in /nix/store/7dhijl4mh1b5rnbvdn9s2lxxqp1ifkq9-tensorflow-mnist-input-data-0.1.0.0-data...
shrinking RPATHs of ELF executables and libraries in /nix/store/h711g6cmliz4qvwx32i4gpdlfl3sdai2-tensorflow-mnist-input-data-0.1.0.0-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/h711g6cmliz4qvwx32i4gpdlfl3sdai2-tensorflow-mnist-input-data-0.1.0.0-doc
checking for references to /build in /nix/store/h711g6cmliz4qvwx32i4gpdlfl3sdai2-tensorflow-mnist-input-data-0.1.0.0-doc...
/nix/store/jag6yf2x30x2j5kmljk4xsm2d83brig0-tensorflow-mnist-input-data-0.1.0.0

teto · 2018-09-06T07:42:34Z

wow that's super cool !! thanks.

haskellPackages.tensorflow-mnist-input-data: fetch

11e2009

Prefetch data so that the package doesn't try to connect to the internet.

teto requested review from basvandijk, peti and ryantm as code owners September 5, 2018 15:19

GrahamcOfBorg added 6.topic: haskell 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Sep 5, 2018

basvandijk suggested changes Sep 6, 2018

View reviewed changes

basvandijk mentioned this pull request Sep 6, 2018

Publish tensorflow-mnist-input-data to Hackage tensorflow/haskell#215

Closed

basvandijk approved these changes Sep 6, 2018

View reviewed changes

basvandijk added 6 commits September 6, 2018 09:15

haskellPackages.tensorflow-mnist-input-data: use an attribute set to …

6bd8799

…specify the data files This is more understandable and safer than using a nested list.

haskellPackages.tensorflow-mnist-input-data: link data files instead …

641025d

…of copying This prevents duplication in the nix store.

haskellPackages.tensorflow-mnist: fix missing dependency on tensorflo…

a5f08c1

…w-mnist-input-data

Revert "libtensorflow: 1.9.0 -> 1.10.0"

6e14b9f

This reverts commit 7139911. libtensorflow is only used by the Haskell tensorflow packages and they don't work with tensorflow-1.10 yet. So the easiest solution is to just revert this commit and add it back when they do gain support.

haskell: build the tensorflow packages on hydra since they now build …

1fc5a6d

…successfully

basvandijk force-pushed the haskell_mnist branch from e1abbb1 to 1fc5a6d Compare September 6, 2018 07:16

basvandijk merged commit 7158e74 into NixOS:master Sep 6, 2018

teto deleted the haskell_mnist branch September 6, 2018 07:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

haskellPackages.tensorflow-mnist-input-data: fetch #46103

haskellPackages.tensorflow-mnist-input-data: fetch #46103

teto commented Sep 5, 2018

GrahamcOfBorg commented Sep 5, 2018

GrahamcOfBorg commented Sep 5, 2018

basvandijk left a comment

basvandijk Sep 6, 2018 •

edited

basvandijk Sep 6, 2018

basvandijk commented Sep 6, 2018

basvandijk commented Sep 6, 2018 •

edited

basvandijk commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

teto commented Sep 6, 2018

haskellPackages.tensorflow-mnist-input-data: fetch #46103

haskellPackages.tensorflow-mnist-input-data: fetch #46103

Conversation

teto commented Sep 5, 2018

Motivation for this change

Things done

GrahamcOfBorg commented Sep 5, 2018

GrahamcOfBorg commented Sep 5, 2018

basvandijk left a comment

Choose a reason for hiding this comment

basvandijk Sep 6, 2018 • edited

Choose a reason for hiding this comment

basvandijk Sep 6, 2018

Choose a reason for hiding this comment

basvandijk commented Sep 6, 2018

basvandijk commented Sep 6, 2018 • edited

basvandijk commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

GrahamcOfBorg commented Sep 6, 2018

teto commented Sep 6, 2018

basvandijk Sep 6, 2018 •

edited

basvandijk commented Sep 6, 2018 •

edited