rambox: build from source #31137

lukateras · 2017-11-02T15:56:39Z

Motivation for this change

This makes rambox build from source. It also includes Sencha Cmd derivation (build tool used by rambox) and fetchNodeModules fetcher both of which are currently not available at top-level (not good enough).

@gnidorah, please check this out and tell if it works for you, and if you see any errors in DevTools console in normal workflow.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Mic92 · 2017-11-02T16:35:19Z

Checksum seems be wrong:

...
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: 7zip-bin-win@2.1.0 (node_modules/7zip-bin-win):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for 7zip-bin-win@2.1.0: wanted {"os":"win32","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: 7zip-bin-win@2.1.0 (node_modules/electron-builder-squirrel-windows/node_modules/7zip-bin-win):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for 7zip-bin-win@2.1.0: wanted {"os":"win32","arch":"any"} (current: {"os":"linux","arch":"x64"})
added 908 packages in 52.274s
output path '/nix/store/aknfmf7rqs2kv5wgbslh3q595rjd6q4m-node_modules' has r:sha256 hash '0gllw4zbrx7fagcrqfzsvw3rlzfqvmf4x04436xwfphl1h3pyw5q' when '09z6r17v1nikcfzbs06mvv5pl4zdclf6xcryfiy9iv8nd0rv8nzw' was expected
cannot build derivation '/nix/store/7ngmcvgmfcn5h343h72bqak62h1kqhr8-rambox-bare-0.5.13.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8p3jb1bwgkp1il3wlvry6q0lva07q3dd-rambox-0.5.13.drv': 1 dependencies couldn't be built
error: build of '/nix/store/8p3jb1bwgkp1il3wlvry6q0lva07q3dd-rambox-0.5.13.drv' failed
The invocation of "nix-shell --keep-going -j4 --option build-use-sandbox true -p rambox" failed

lukateras · 2017-11-02T17:18:10Z

@Mic92 Interesting, fetchNodeModules is deterministic for me. When I refetch after removing result and running nixos-collect-garbage, it still has the same checksum.

Could you show the full log? Also I think this relies on f1e8138, NPM until recently was not deterministic, see npm/npm#17979 (comment).

ghost · 2017-11-02T17:18:41Z

@yegortimoshenko Wow, you've made a great work!!! 👍 Works fine (aside of url opening under wayland). No errors in console so far.

ghost · 2017-11-02T17:19:37Z

pkgs/applications/networking/instant-messengers/rambox/sencha-bare.nix

+    rm $out/shell-wrapper.sh $out/Uninstaller
+  '';
+
+  meta = with stdenv.lib; {


sencha license?

Yeah, it should be licenses.unfree.

ghost · 2017-11-02T17:20:02Z

pkgs/applications/networking/instant-messengers/rambox/default.nix

-    description = "Free and Open Source messaging and emailing app that combines common web applications into one";
-    homepage = http://rambox.pro;
-    license = licenses.mit;
-    maintainers = [ maintainers.gnidorah ];


you don't want to add yourself to maintainers? :-)

ghost · 2017-11-02T17:20:54Z

pkgs/applications/networking/instant-messengers/rambox/default.nix

-
-  postFixup = ''
-    paxmark m $out/opt/Rambox/rambox
-    wrapProgram $out/opt/Rambox/rambox --prefix PATH : ${xdg_utils}/bin


this is still needed for url opening under wayland, since xdg-utils is not a part of e.g. sway compositor

Fixed, thanks!

grahamc · 2017-11-02T17:51:56Z

pkgs/applications/networking/instant-messengers/rambox/env.js

@@ -0,0 +1 @@
+var auth0Cfg={clientID:"y9am0DVawe2tvlA3ucD7OufpJHZZMjsO",domain:"rambox.auth0.com"};


are we allowed to put this client ID in our repo?

No, not explicitly.

A new client ID requires an account at https://auth0.com/. I could register one on my own but I'm not sure if that's appropriate. Chromium derivation also has API credentials:

nixpkgs/pkgs/applications/networking/browsers/chromium/common.nix

Lines 197 to 203 in e8f1ddc

# Google API keys, see:

# http://www.chromium.org/developers/how-tos/api-keys

# Note: These are for NixOS/nixpkgs use ONLY. For your own distribution,

# please get your own set of keys.

google_api_key = "AIzaSyDGi15Zwl11UNe6Y-5XW_upsfyw31qwZPI";

google_default_client_id = "404761575300.apps.googleusercontent.com";

google_default_client_secret = "9rIFQjfnkykEmqb6FfjJQD1D";

Are they registered using a special email address or is it just someone who happens to maintain the derivation?

Nevermind, used my own account for now. Fixed!

Mic92 · 2017-11-02T18:37:38Z

@yegortimoshenko I have build you pull request both rebased against master and your pull request branch:

https://gist.github.com/Mic92/f47265189009d4cbd202c2dafec3ecaf

The only difference I can spot is that I use nixUnstable, which is also used on hydra.

lukateras · 2017-11-02T20:41:24Z

@Mic92 This is probably due to some install script. I've disabled those and forced npm to install only production dependencies (reduces size and potentially less prone to breakage) but for all platforms (previously e.g. Linux-specific dependency would not be fetched on macOS which leads to a different checksum).

Now there should be very little surface for non-determinism. Could you please try again?

lukateras · 2017-11-02T22:44:45Z

I've made a few other fixes: switched file layout/naming, added another patch to completely disable auto-update functionality along with passing --without-update flag, added assertion to fetchNodeModules, fixed locale support.

Mic92 · 2017-11-03T00:34:30Z

Now I get:

building path(s) '/nix/store/shm37242gb6m7nvnjg27a1x83ib6vrcz-node_modules'
building '/nix/store/fx8113433ghsr1y8d8j5j2kid41w1shi-node_modules.drv'...
npm WARN using --force I sure hope you know what you are doing.
added 107 packages in 7.064s
output path '/nix/store/shm37242gb6m7nvnjg27a1x83ib6vrcz-node_modules' has r:sha256 hash '04wmwnjm5010n564gq7l8w1cs411wy3p4hk05a2s63z7waxsaw63' when '0w912a3pmd87rwxi5kq248b2yicissvzyq6iafim6ckvvcvky5xf' was expected
cannot build derivation '/nix/store/249wmafhwzfv5b20j0197scix58bi7sj-rambox-bare-0.5.13.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9x5fhfxah9iwr6p5r74i5jijghv9irz3-rambox-0.5.13.drv': 1 dependencies couldn't be built
error: build of '/nix/store/9x5fhfxah9iwr6p5r74i5jijghv9irz3-rambox-0.5.13.drv' failed

I have uploaded node-modules for inspection: https://dl.thalheim.io/wCrMmJa1V6fYMGbzdZ7IUw/node-modules.tar.gz

$ tar --xattrs -czpf node-modules.tar.gz /tmp/nix-build-node_modules.drv-0

You can diff it with diffoscope

lukateras · 2017-11-03T01:38:36Z

Could you show /nix/store/shm37242gb6m7nvnjg27a1x83ib6vrcz-node_modules?

fetchNodeModules moves rather than copies to $out. Or this line could be replaced with cp -r node_modules $out so that build directory has the output.

ghost · 2017-11-03T08:26:13Z

pkgs/applications/networking/instant-messengers/rambox/bare.nix

+    homepage = http://rambox.pro;
+    license = licenses.gpl3;
+    maintainers = with maintainers; [ gnidorah ];
+    platforms = platforms.linux;


someone with Mac to test it there? :-)

That'd take some effort: Sencha CMD derivation should be extended to Darwin first. And I'd wager that most macOS users are more comfortable with packaging native to their platform anyway.

I agree with you. Just some offtopic: some nixpkgs macOS users don't agree, see discussion #20670 (comment)

Mic92 · 2017-11-03T15:53:42Z

@yegortimoshenko

$ tar --xattrs -czpf ~/upload/node-modules.tar.gz /nix/store/mfb04xbqdmbh5cpmwqrhvq8jbmfb81lx-node_modules

https://dl.thalheim.io/wCrMmJa1V6fYMGbzdZ7IUw/node-modules.tar.gz

lukateras · 2017-11-03T20:04:03Z

@Mic92 Thanks a lot! My build dir is different from yours for some reason:

diff -urNZ /nix/store/shm37242gb6m7nvnjg27a1x83ib6vrcz-node_modules/asap/package.json nix/store/mfb04xbqdmbh5cpmwqrhvq8jbmfb81lx-node_modules/asap/package.json
--- /nix/store/shm37242gb6m7nvnjg27a1x83ib6vrcz-node_modules/asap/package.json	1970-01-01 00:00:01.000000000 +0000
+++ nix/store/mfb04xbqdmbh5cpmwqrhvq8jbmfb81lx-node_modules/asap/package.json	1970-01-01 00:00:01.000000000 +0000
@@ -2,7 +2,7 @@
   "_args": [
     [
       "asap@2.0.6",
-      "/tmp/nix-build-node_modules.drv-0"
+      "/build"
     ]
   ],
   "_from": "asap@2.0.6",
@@ -26,7 +26,7 @@
   ],
   "_resolved": "https://registry.npmjs.org/asap/-/asap-2.0.6.tgz",
   "_spec": "2.0.6",
-  "_where": "/tmp/nix-build-node_modules.drv-0",
+  "_where": "/build",
   "browser": {
     "./asap": "./browser-asap.js",
     "./asap.js": "./browser-asap.js",

Of course this metadata should not affect the checksum. I've updated the fetcher to strip it using jq.
Also see npm/npm#10393.

Mic92 · 2017-11-03T20:44:44Z

@yegortimoshenko this is due nixUnstable, I suppose. Hydra will have the same build path. But looks like it could be normalized. Why do they store this information at all?

lukateras · 2017-11-04T13:16:39Z

They store this information to speed up next npm install invocations. This should be ready to merge.

Mic92 · 2017-11-04T16:32:42Z

But it will not build in hydra because the wrong checksum.

lukateras · 2017-11-04T16:56:01Z

But checks have passed. I've already fixed the fetcher, it now normalizes package.json files. Could you please link me to the failing build? Can't find it on hydra.nixos.org.

ghost · 2017-11-05T06:15:00Z

@Mic92 Will hydra build it? Rambox dependency Sencha is unfree. We can "build" w/o Sencha, like Rambox CI scripts do https://github.com/NixOS/nixpkgs/pull/20657/files#diff-5a9e329a438665eff30913e6b6d5bb72R46 but that's not cool

Mic92 · 2017-11-07T09:34:43Z

@yegortimoshenko thanks, with the normalization it works. @gnidorah hydra will not build it with sencha beeing marked as unfree. But the build seems pretty fast to me, so maybe it does not matter.

Mic92 · 2017-11-07T09:41:15Z

pkgs/applications/networking/instant-messengers/rambox/fetchNodeModules.nix

+      jq -S 'delpaths(keys | map(select(startswith("_")) | [.]))' $f > $f.tmp
+      mv $f.tmp $f
+    done
+    mv node_modules $out


This snippet is actually pretty cool. We can use it to build other node.js applications without adding thousands of npm dependencies to nixpkgs.

@Mic92 Thanks! I'll test it with another package or two and then I'll move it to build-support. Pretty sure it's deterministic now, but ignoring scripts might be a problem with some builds. Running scripts will likely cause different checksums for different platforms/architectures.

I also have a fetcher for Maven/Leiningen project dependencies:
https://github.com/yegortimoshenko/overlay/blob/master/pkgs/fetchMaven/default.nix

ghost · 2017-11-07T10:43:58Z

@yegortimoshenko Thank you much!

GrahamcOfBorg added 10.rebuild-linux: 0 10.rebuild-darwin: 0 labels Nov 2, 2017

ghost reviewed Nov 2, 2017

View reviewed changes

grahamc reviewed Nov 2, 2017

View reviewed changes

lukateras force-pushed the rambox/build-from-source branch 2 times, most recently from 597a258 to e7fc6fd Compare November 2, 2017 20:37

lukateras force-pushed the rambox/build-from-source branch 2 times, most recently from 8e96d14 to d48662b Compare November 2, 2017 22:41

ghost reviewed Nov 3, 2017

View reviewed changes

rambox: build from source

cc04a30

lukateras force-pushed the rambox/build-from-source branch from d48662b to cc04a30 Compare November 3, 2017 20:02

Mic92 reviewed Nov 7, 2017

View reviewed changes

Mic92 merged commit 1925b4c into NixOS:master Nov 7, 2017

ghost mentioned this pull request Nov 22, 2017

rambox: build from sources #20657

Closed

lukateras deleted the rambox/build-from-source branch December 18, 2017 05:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rambox: build from source #31137

rambox: build from source #31137

lukateras commented Nov 2, 2017 •

edited

Mic92 commented Nov 2, 2017 •

edited

lukateras commented Nov 2, 2017 •

edited

ghost commented Nov 2, 2017

ghost Nov 2, 2017

lukateras Nov 2, 2017

ghost Nov 2, 2017

lukateras Nov 2, 2017

ghost Nov 2, 2017

lukateras Nov 2, 2017

grahamc Nov 2, 2017

lukateras Nov 2, 2017

lukateras Nov 2, 2017

Mic92 commented Nov 2, 2017 •

edited

lukateras commented Nov 2, 2017 •

edited

lukateras commented Nov 2, 2017 •

edited

Mic92 commented Nov 3, 2017 •

edited

lukateras commented Nov 3, 2017 •

edited

ghost Nov 3, 2017

lukateras Nov 3, 2017

ghost Nov 5, 2017 •

edited by ghost

Mic92 commented Nov 3, 2017

lukateras commented Nov 3, 2017 •

edited

Mic92 commented Nov 3, 2017 •

edited

lukateras commented Nov 4, 2017 •

edited

Mic92 commented Nov 4, 2017

lukateras commented Nov 4, 2017 •

edited

ghost commented Nov 5, 2017 •

edited by ghost

Mic92 commented Nov 7, 2017 •

edited

Mic92 Nov 7, 2017

lukateras Nov 7, 2017 •

edited

ghost commented Nov 7, 2017

		@@ -0,0 +1 @@
		var auth0Cfg={clientID:"y9am0DVawe2tvlA3ucD7OufpJHZZMjsO",domain:"rambox.auth0.com"};

	# Google API keys, see:
	# http://www.chromium.org/developers/how-tos/api-keys
	# Note: These are for NixOS/nixpkgs use ONLY. For your own distribution,
	# please get your own set of keys.
	google_api_key = "AIzaSyDGi15Zwl11UNe6Y-5XW_upsfyw31qwZPI";
	google_default_client_id = "404761575300.apps.googleusercontent.com";
	google_default_client_secret = "9rIFQjfnkykEmqb6FfjJQD1D";

rambox: build from source #31137

rambox: build from source #31137

Conversation

lukateras commented Nov 2, 2017 • edited

Motivation for this change

Things done

Mic92 commented Nov 2, 2017 • edited

lukateras commented Nov 2, 2017 • edited

ghost commented Nov 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Nov 2, 2017 • edited

lukateras commented Nov 2, 2017 • edited

lukateras commented Nov 2, 2017 • edited

Mic92 commented Nov 3, 2017 • edited

lukateras commented Nov 3, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Nov 5, 2017 • edited by ghost

Choose a reason for hiding this comment

Mic92 commented Nov 3, 2017

lukateras commented Nov 3, 2017 • edited

Mic92 commented Nov 3, 2017 • edited

lukateras commented Nov 4, 2017 • edited

Mic92 commented Nov 4, 2017

lukateras commented Nov 4, 2017 • edited

ghost commented Nov 5, 2017 • edited by ghost

Mic92 commented Nov 7, 2017 • edited

Choose a reason for hiding this comment

lukateras Nov 7, 2017 • edited

Choose a reason for hiding this comment

ghost commented Nov 7, 2017

lukateras commented Nov 2, 2017 •

edited

Mic92 commented Nov 2, 2017 •

edited

lukateras commented Nov 2, 2017 •

edited

Mic92 commented Nov 2, 2017 •

edited

lukateras commented Nov 2, 2017 •

edited

lukateras commented Nov 2, 2017 •

edited

Mic92 commented Nov 3, 2017 •

edited

lukateras commented Nov 3, 2017 •

edited

ghost Nov 5, 2017 •

edited by ghost

lukateras commented Nov 3, 2017 •

edited

Mic92 commented Nov 3, 2017 •

edited

lukateras commented Nov 4, 2017 •

edited

lukateras commented Nov 4, 2017 •

edited

ghost commented Nov 5, 2017 •

edited by ghost

Mic92 commented Nov 7, 2017 •

edited

lukateras Nov 7, 2017 •

edited