lumo: init at 1.9.0-alpha #44076

hlolli · 2018-07-25T04:22:09Z

Motivation for this change

This would be the 4th attempt to get lumo into nixpkgs, the other 3 being
#33691
#25770
#38619

This is built completely from source. Jar dependencies are fetched from deps.nix, generated by deps.edn via clj2nix. Node modules are fetched via yarn2nix, other projects using yarn2nix also put the yarn.lock and package.json into the nixpkgs tree, despite the lockfile size can get pretty big (here 200k).

Lumo is built just like it is built with boot. Boot turned out to be unable to be used in offline mode, because it ignores java classpaths. Therefore I used the clojure cli tool where I needed to call clojure to build clojurescript sources.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

ryantm · 2018-09-09T20:59:28Z

@GrahamcOfBorg build lumo

GrahamcOfBorg · 2018-09-09T20:59:41Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: lumo

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-09T21:00:03Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: lumo

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-09T21:24:29Z

Success on x86_64-linux (full log)

Attempted: lumo

Partial log (click to expand)

Finished bundling. Nexe binary can be found in build/lumo
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/wfiml0x4cyj3yipy1mckhm1fkxxr8v0q-lumo-1.9.0-alpha
shrinking /nix/store/wfiml0x4cyj3yipy1mckhm1fkxxr8v0q-lumo-1.9.0-alpha/bin/lumo
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/wfiml0x4cyj3yipy1mckhm1fkxxr8v0q-lumo-1.9.0-alpha/bin
patching script interpreter paths in /nix/store/wfiml0x4cyj3yipy1mckhm1fkxxr8v0q-lumo-1.9.0-alpha
checking for references to /build in /nix/store/wfiml0x4cyj3yipy1mckhm1fkxxr8v0q-lumo-1.9.0-alpha...
/nix/store/wfiml0x4cyj3yipy1mckhm1fkxxr8v0q-lumo-1.9.0-alpha

ryantm · 2018-09-09T21:47:12Z

I don't really know enough to know if this is good. It seems like a lot of custom setup and building is going on here. Maybe this is okay given that this is a compiler. I'll squash and merge this in soon if no one objects.

lukateras

I get:

Archive:  /nix/store/hz99ddmp9v7ka3dm9p4b9cp7yv7wkfxk-org_clojure_test_check-0.10.0-alpha2.jar
  inflating: ./target/META-INF/MANIFEST.MF
   creating: ./target/clojure/test/
   creating: ./target/clojure/test/check/
   creating: ./target/clojure/test/check/random/
   creating: ./target/clojure/test/check/random/longs/
  inflating: ./target/clojure/test/check.cljc
  inflating: ./target/clojure/test/check/results.cljc
  inflating: ./target/clojure/test/check/clojure_test.cljc
  inflating: ./target/clojure/test/check/impl.cljc
  inflating: ./target/clojure/test/check/properties.cljc
  inflating: ./target/clojure/test/check/generators.cljc
  inflating: ./target/clojure/test/check/random.clj
  inflating: ./target/clojure/test/check/random/doubles.cljs
  inflating: ./target/clojure/test/check/random/longs/bit_count_impl.cljs
  inflating: ./target/clojure/test/check/random/longs.cljs
  inflating: ./target/clojure/test/check/random.cljs
  inflating: ./target/clojure/test/check/rose_tree.cljc
   creating: ./target/META-INF/maven/org.clojure/test.check/
  inflating: ./target/META-INF/maven/org.clojure/test.check/pom.xml
  inflating: ./target/META-INF/maven/org.clojure/test.check/pom.properties
java.io.FileInputStream
#'user/write-transit-json
#'user/process-caches
/build/lumo-1.9.0-alpha/scripts/bundle.js:83
  );
  ^

SyntaxError: Unexpected token )
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:549:28)
    at Object.Module._extensions..js (module.js:586:10)
    at Module.load (module.js:494:32)
    at tryModuleLoad (module.js:453:12)
    at Function.Module._load (module.js:445:3)
    at Module.runMain (module.js:611:10)
    at run (bootstrap_node.js:387:7)
    at startup (bootstrap_node.js:153:9)
builder for '/nix/store/34mkwq4m76nqaavnxx21z258zh5qn9r3-lumo-1.9.0-alpha.drv' failed with exit code 1
error: build of '/nix/store/34mkwq4m76nqaavnxx21z258zh5qn9r3-lumo-1.9.0-alpha.drv' failed

hlolli · 2018-11-03T12:57:25Z

@yegortimoshenko

Sorry for the delay, I looked at this syntax error and it seems to be in the package itself

https://github.com/anmonteiro/lumo/blob/1.9.0-alpha/scripts/bundle.js#L76-L84

there's a comma after the array.push call, which shouldn't be there, but I'm surprised that js forgiveness isn't so forgiving on your computer. Are you using node11? Should I fix this with sed?

garrett-hopper · 2018-11-09T02:38:46Z

@hlolli, it probably makes more sense to use a patch.
fix-comma.patch.txt

hlolli · 2018-11-10T01:16:51Z

@yegortimoshenko saga continues, I bumped to 1.9.0 release, added patch for that comma thingy plus some fs promises that were only recently added in node version 10.7.0 (and are experimental). It compiles in nodejs 8 (the current nodejs version on nixpkgs) and all later versions, if you are testing this on a node version lower than 8, then I can only cross fingers that it will compile and run (are we supposed to support so old node versions anyway?).

I had to prevent strip, this is an issue from the binary bundler and npm package nexe, the solution not to prevent to strip is way to hacky. I won't have much time to look into this in next weeks, so I hope this gets finally accepted.

ryantm · 2018-11-10T03:12:53Z

@GrahamcOfBorg build lumo

GrahamcOfBorg · 2018-11-10T03:13:10Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: lumo

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-11-10T03:23:46Z

Failure on x86_64-darwin (full log)

Attempted: lumo

Partial log (click to expand)

    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:915:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
(node:82372) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:82372) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
### Compiling Macro Namespaces
scripts/aot-bundle-macros.sh: line 16: /private/tmp/nix-build-lumo-1.9.0.drv-0/lumo-1.9.0/build/lumo: No such file or directory
builder for '/nix/store/6z601a0hnqmkk1slzcn18xa7dlw1k05x-lumo-1.9.0.drv' failed with exit code 127
error: build of '/nix/store/6z601a0hnqmkk1slzcn18xa7dlw1k05x-lumo-1.9.0.drv' failed

GrahamcOfBorg · 2018-11-10T04:04:49Z

Success on x86_64-linux (full log)

Attempted: lumo

Partial log (click to expand)

- Entry: 'LUMO__INTERNAL__CLASSPATH/bundle.min.js' written to: build/lumo
✔ Entry: 'LUMO__INTERNAL__CLASSPATH/bundle.min.js' written to: build/lumo
- Finished in 10.096s
✔ Finished in 10.096s
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/x5a2xg4s719smir3i6qv36qiwsmgwkkg-lumo-1.9.0
shrinking /nix/store/x5a2xg4s719smir3i6qv36qiwsmgwkkg-lumo-1.9.0/bin/lumo
patching script interpreter paths in /nix/store/x5a2xg4s719smir3i6qv36qiwsmgwkkg-lumo-1.9.0
checking for references to /build in /nix/store/x5a2xg4s719smir3i6qv36qiwsmgwkkg-lumo-1.9.0...

lukateras · 2018-11-10T11:56:55Z

nodejs is Node.js 8.12.0, nodejs-10_x is Node.js 10.12.0. If Lumo only works with the latter, you could either explicitly require that version in arg attrset:

 { stdenv, fetchurl, clojure,
-  nodejs, yarn2nix, jre, unzip,
+ nodejs-10_x, yarn2nix, jre, unzip,
   python, openssl }:

Or inject override in callPackage:

- lumo = callPackage ../development/interpreters/clojurescript/lumo {}; 
+ lumo = callPackage ../development/interpreters/clojurescript/lumo {
   nodejs = nodejs-10_x;
 };

hlolli · 2018-11-10T12:03:54Z

I thought it wasn't allowed :), I think injecting it in callPackage makes sense. But it does work for both nodejs 8 and 10, I used nodejs-10_x on my computer as well.

That darwin build failed, the platforms = all was a mistake. If I have darwin with nix, I'll try to debug it later, so I'll change it, targeting linux only for now.

So 3 things, I remove the patch, remove platform =all to linux and add the inject override. Squash and push.

hlolli · 2018-11-10T14:57:30Z

@infinisil done #50181

infinisil · 2018-11-10T15:02:05Z

@hlolli I said two separate commits. PR's are annoying for that because then one depends on the other :)

hlolli · 2018-11-10T15:04:23Z

ah I see :) should be quick fix, 1min...

hlolli · 2018-11-10T15:05:47Z

ok I wasn't quick enough to close my PR :) it got merged. uh.. it should be fine then? @infinisil

infinisil · 2018-11-10T15:09:43Z

@GrahamcOfBorg build lumo

GrahamcOfBorg · 2018-11-10T15:09:59Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: lumo

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-11-10T15:10:02Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: lumo

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Mic92 · 2018-11-10T15:18:08Z

pkgs/development/interpreters/clojurescript/lumo/package.json

@@ -0,0 +1,110 @@
+{
+  "name": "lumo-cljs",


Would it be possible to add lumo-cljs to pkgs/development/node-packages/node-packages-v8.json.
Then we can share its dependencies with other node packages. 17k of dependencies is not acceptable for a single package. Git and nix do not like large files like this.

Could be a good idea. Don't know if this has changed, but there was such a lack of example useage of yarn2nix in nixpkgs that I felt a bit on my own. That said, if you read the other 3 PR for lumo (lumo = lumo-cljs (lumo was already reserved on npm)), then all that the npm-module does is downloading a binary file from github, a binary file that can not run on nixos no matter the amount of binary hacking. So the npm module itself is useless in terms of lumo in Nixos. But I guess that the npm dependencies to build lumo could be shared, that would mean that I would need to make 40 nix derivations before we can build lumo in nixpkgs?

Another thing, if two nix epression want to fetch the same url with the same sha sum, isn't that resource shared in nix-store, even tough the reference is seperate?

It's also not 17k, it's 1038 fetchurl's to yarn registry. It's a whole lot I admit.

It is not just about sharing the source (which is the case) but also evaluation time and channel size (each user would then need to store an 1mb extra of expressions just for a single package).

If you the package would be included in nodePackages can its dependencies not also used from there?

$ nix repl '<nixpkgs>' nixpkgs> :b nodePackages.eslint > "${nodePackages.eslint}" /nix/store/7sk5kg95rdy25rjdxvknxl3rmkgjcxxg-node-eslint-5.8.0 > ls -la /nix/store/7sk5kg95rdy25rjdxvknxl3rmkgjcxxg-node-eslint-5.8.0/lib/node_modules/eslint/node_modules/

We could even delete unusable binaries from the package itself by adding an override.
Many of the dependencies listed here are already include in node-packages.

if the package is in node dependencies, I will have to test if the build tool (yarn/npm) will no just try to make lockfile, or fetch dep graph fro the mirror. I had this problem with npm2nix, that I couldn't stop it from making http requests. The thing that I worry about too, is that I would only be adding lumo-cljs to nix's node-packages for its build dependencies but not runtime, and the readme in pkgs/development/node-packages said it only wants top-level packages. Lumo isn't a traditional package, npm is part of the process of building it, it serves more as a way to install the binary globaly than as node module. You are right that I can diminsih the size quite a bit, it's now total for 6 files 633.6kb and I made a comment about this size in my PR message. I could try to make this smaller. And I can try adding lumo-cljs as non top-level npm module.

"Libraries should only be added to the package set if there is a non-NPM package that requires it." so in the Lumo's case it should be fine :)

We do the same already for meguca.

Can you look over my PR now, this looks quite strange, the version name actually takes relative path, and that's appended to the nodePackages attribute name. But it works fine like this (hopefully GrahamcOfBorg aggrees).

GrahamcOfBorg · 2018-11-10T15:36:04Z

Success on x86_64-linux (full log)

Attempted: lumo

Partial log (click to expand)

- Entry: 'LUMO__INTERNAL__CLASSPATH/bundle.min.js' written to: build/lumo
✔ Entry: 'LUMO__INTERNAL__CLASSPATH/bundle.min.js' written to: build/lumo
- Finished in 2.769s
✔ Finished in 2.769s
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/gk8fvb9m508162c9665v7jc2abn82blj-lumo-1.9.0
shrinking /nix/store/gk8fvb9m508162c9665v7jc2abn82blj-lumo-1.9.0/bin/lumo
patching script interpreter paths in /nix/store/gk8fvb9m508162c9665v7jc2abn82blj-lumo-1.9.0
checking for references to /build in /nix/store/gk8fvb9m508162c9665v7jc2abn82blj-lumo-1.9.0...

infinisil · 2018-11-11T00:26:51Z

Awesome, thanks :)

hlolli · 2018-11-11T00:32:17Z

😰 I made changes to @Mic92's review, from which time GrahamcOfBorg didn't run

infinisil · 2018-11-11T00:49:33Z

@hlolli I tested it locally, the build and everything works fine :) (GrahamcOfBorg just build the derivation anyways)

cocreature · 2018-11-11T20:43:22Z

This broke the tarball job on hydra https://hydra.nixos.org/build/83951478/nixlog/1 due to the import of <nixpkgs>.

infinisil · 2018-11-11T20:47:58Z

pkgs/development/interpreters/clojurescript/lumo/deps.nix

+        "http://oss.sonatype.org/content/repositories/public/"
+        "http://repo.typesafe.com/typesafe/releases/"
+      ];
+  pkgs = import <nixpkgs> {};


Ohh yeah, this is the problem here. Does clj2nix have a way to not use import <nixpkgs> {} and instead take an argument? Otherwise this needs to be fixed manually.

oh shit, this is my first nixos plugin, so it's no surprise I did something wrong. Yes, I'll find a solution, inherit pkgs or worst case, pass fetchMavenArtifact as argument. Can this wait 24h?

Or maybe I do this after eating (2H)

You could just add pkgs: at the top of this file, remove line 10 instead.

No worries, stuff breaks all the time, I'm just glad @cocreature noticed it this early :)

infinisil · 2018-11-11T23:05:12Z

pkgs/development/interpreters/clojurescript/lumo/default.nix

+    '';
+
+
+    cljdeps   = import ./deps.nix;


Then change this to cljdeps = import ./deps.nix pkgs; while getting pkgs by adding it to the argument list at the top.

good idea, I do that!

samueldr · 2018-11-11T23:06:34Z

@GrahamcOfBorg build lumo

(A bit late, but I want to confirm whether it has issues with the import.)

GrahamcOfBorg · 2018-11-11T23:06:50Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: lumo

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-11-11T23:06:54Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: lumo

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-11-11T23:07:29Z

Success on x86_64-linux (full log)

Attempted: lumo

Partial log (click to expand)

these paths will be fetched (17.06 MiB download, 53.27 MiB unpacked):
  /nix/store/3gs08y9ja2vyaabhh94acjzqaz0bzlas-lumo-1.9.0
copying path '/nix/store/3gs08y9ja2vyaabhh94acjzqaz0bzlas-lumo-1.9.0' from 'https://cache.nixos.org'...
/nix/store/3gs08y9ja2vyaabhh94acjzqaz0bzlas-lumo-1.9.0

hlolli force-pushed the lumo branch 2 times, most recently from be73af9 to 1b83e02 Compare July 25, 2018 09:44

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Jul 25, 2018

hlolli mentioned this pull request Aug 7, 2018

closh init at 0.2.2 #44636

Closed

5 tasks

lukateras self-assigned this Oct 13, 2018

lukateras mentioned this pull request Oct 13, 2018

lumo: init #33691

Closed

8 tasks

lukateras requested changes Oct 13, 2018

View reviewed changes

This was referenced Nov 9, 2018

Remove Unnecessary Comma anmonteiro/lumo#448

Closed

Create a NixOS package anmonteiro/lumo#311

Open

hlolli mentioned this pull request Nov 9, 2018

remove spurious comma anmonteiro/lumo#450

Closed

hlolli force-pushed the lumo branch from e5f3736 to 3a447df Compare November 10, 2018 01:08

GrahamcOfBorg added 10.rebuild-darwin: 1-10 and removed 10.rebuild-darwin: 0 labels Nov 10, 2018

hlolli force-pushed the lumo branch from 3a447df to d4c444b Compare November 10, 2018 12:08

hlolli force-pushed the lumo branch from d4c444b to ca7eee7 Compare November 10, 2018 14:53

hlolli mentioned this pull request Nov 10, 2018

add myself (hlolli) to maintainers list #50181

Merged

Mic92 reviewed Nov 10, 2018

View reviewed changes

Lumo init at 1.9.0

0e134bf

hlolli force-pushed the lumo branch from ca7eee7 to 0e134bf Compare November 10, 2018 20:53

GrahamcOfBorg added 10.rebuild-darwin: 1-10 and removed 10.rebuild-darwin: 0 labels Nov 10, 2018

infinisil approved these changes Nov 11, 2018

View reviewed changes

infinisil merged commit a7da00d into NixOS:master Nov 11, 2018

infinisil reviewed Nov 11, 2018

View reviewed changes

infinisil mentioned this pull request Nov 12, 2018

use newer clj2nix which passes pkgs as an argument to deps.nix #50270

Merged

9 tasks

This was referenced Nov 12, 2018

rambox: 0.6.1 -> 0.6.2 #50262

Closed

turtl-server: init at 2018-11-05 #50409

Closed

lumo: init at 1.9.0-alpha #44076

lumo: init at 1.9.0-alpha #44076

Conversation

hlolli commented Jul 25, 2018 • edited by lukateras

Motivation for this change

Things done

ryantm commented Sep 9, 2018

GrahamcOfBorg commented Sep 9, 2018

GrahamcOfBorg commented Sep 9, 2018

GrahamcOfBorg commented Sep 9, 2018

ryantm commented Sep 9, 2018

lukateras left a comment

Choose a reason for hiding this comment

hlolli commented Nov 3, 2018

garrett-hopper commented Nov 9, 2018

hlolli commented Nov 10, 2018

ryantm commented Nov 10, 2018

GrahamcOfBorg commented Nov 10, 2018

GrahamcOfBorg commented Nov 10, 2018

GrahamcOfBorg commented Nov 10, 2018

lukateras commented Nov 10, 2018

hlolli commented Nov 10, 2018

hlolli commented Nov 10, 2018

infinisil commented Nov 10, 2018

hlolli commented Nov 10, 2018

hlolli commented Nov 10, 2018

infinisil commented Nov 10, 2018

GrahamcOfBorg commented Nov 10, 2018

GrahamcOfBorg commented Nov 10, 2018

Mic92 Nov 10, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hlolli Nov 10, 2018 • edited

Choose a reason for hiding this comment

Mic92 Nov 10, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GrahamcOfBorg commented Nov 10, 2018

infinisil commented Nov 11, 2018

hlolli commented Nov 11, 2018

infinisil commented Nov 11, 2018

cocreature commented Nov 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samueldr commented Nov 11, 2018

GrahamcOfBorg commented Nov 11, 2018

GrahamcOfBorg commented Nov 11, 2018

GrahamcOfBorg commented Nov 11, 2018

hlolli commented Jul 25, 2018 •

edited by lukateras

Mic92 Nov 10, 2018 •

edited

hlolli Nov 10, 2018 •

edited

Mic92 Nov 10, 2018 •

edited