yarn: add yarnpkg binary to outputs and use nodejs dependency #35063

nicknovitski · 2018-02-17T00:45:02Z

Motivation for this change

My company makes a few open source nodejs tools that call out to the nodejs package manage yarn, but some users of those tools also have hadoop installed on their systems, which also provides a binary named yarn. As that link indicates, this conflict was addressed by the node yarn developers by adding an additional identical binary called yarnpkg, and we use that solution by having our tools run yarn by calling yarnpkg.

For assorted reasons, I'd like us to start packaging yarn internally using nix, but in order to test and work on those externally shipped tools, we need the yarnpkg binary, and I don't see any reason why this package can't include it.

~~While making that change, I noticed that the yarn commands were still looking for the "system" node, so I added an argument to the makeWrapper call to fix that.~~ Never mind about that! I was confused.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

nicknovitski · 2018-02-17T03:00:15Z

Oh, wait, there's also a nodePackages.yarn, which doesn't have any of these problems ~~and actually seems to build from source~~. That's strange.

nicknovitski · 2018-02-17T04:10:19Z

I instead deleted the yarn/default.nix entirely and just aliased the attribute to nodePackages.yarn. That fixed the problem for me.

Further motivation: I want to be able to usefully apply source patches to this package. Unfortunately, yarn's tar.gz files both in npm and github releases are a concatenated "dist" built with their own gulp, webpack, and shell scripts.

I overrode the package to get the actual source, and the default build steps worked just fine.

joachifm · 2018-02-17T06:32:51Z

@GrahamcOfBorg build yarn

GrahamcOfBorg · 2018-02-17T06:33:14Z

Success on x86_64-linux (full log)

Partial log (click to expand)

patching script interpreter paths in .
./yarn/bin/yarn: interpreter directive changed from "/bin/sh" to "/nix/store/fcxliihmhm2ak9z4890gk0qw63zsbrcx-bash-4.4-p12/bin/sh"
./yarn/bin/yarnpkg: interpreter directive changed from "/usr/bin/env node" to "/nix/store/0qp6kmkrx0mqyqrfj781yw3kp2wbkf7y-nodejs-6.12.3/bin/node"
./yarn/bin/yarn.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/0qp6kmkrx0mqyqrfj781yw3kp2wbkf7y-nodejs-6.12.3/bin/node"
./yarn/lib/cli.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/0qp6kmkrx0mqyqrfj781yw3kp2wbkf7y-nodejs-6.12.3/bin/node"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/wxsxbvnnzxgyhx5s4kd1yazkm5gwqdcb-node-yarn-1.3.2
patching script interpreter paths in /nix/store/wxsxbvnnzxgyhx5s4kd1yazkm5gwqdcb-node-yarn-1.3.2
checking for references to /tmp/nix-build-node-yarn-1.3.2.drv-0 in /nix/store/wxsxbvnnzxgyhx5s4kd1yazkm5gwqdcb-node-yarn-1.3.2...
/nix/store/wxsxbvnnzxgyhx5s4kd1yazkm5gwqdcb-node-yarn-1.3.2

GrahamcOfBorg · 2018-02-17T06:33:23Z

Success on x86_64-darwin (full log)

Partial log (click to expand)

unpacking source archive /nix/store/112njfl8y65ynwdr9njffid32z7cgsvc-yarn-1.3.2.tgz
pinpointing versions of dependencies...
patching script interpreter paths in .
./yarn/bin/yarn: interpreter directive changed from "/bin/sh" to "/nix/store/19cc89zq5n68ibr53izy4zgcr5qg2rza-bash-4.4-p12/bin/sh"
./yarn/bin/yarn.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/06302fw8ik0nai9asn1zga64g88c2ifc-nodejs-6.12.3/bin/node"
./yarn/bin/yarnpkg: interpreter directive changed from "/usr/bin/env node" to "/nix/store/06302fw8ik0nai9asn1zga64g88c2ifc-nodejs-6.12.3/bin/node"
./yarn/lib/cli.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/06302fw8ik0nai9asn1zga64g88c2ifc-nodejs-6.12.3/bin/node"
post-installation fixup
patching script interpreter paths in /nix/store/fxyghd2h5hrd8f465xpvrx66xz18gg3l-node-yarn-1.3.2
/nix/store/fxyghd2h5hrd8f465xpvrx66xz18gg3l-node-yarn-1.3.2

GrahamcOfBorg · 2018-02-17T06:33:40Z

Success on aarch64-linux (full log)

Partial log (click to expand)

patching script interpreter paths in .
./yarn/lib/cli.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/g2pg448750pf27js1p5niwv831j973wv-nodejs-6.12.3/bin/node"
./yarn/bin/yarn: interpreter directive changed from "/bin/sh" to "/nix/store/shjmvsnwrlg7rd8m0kzgpygqq4if312z-bash-4.4-p12/bin/sh"
./yarn/bin/yarnpkg: interpreter directive changed from "/usr/bin/env node" to "/nix/store/g2pg448750pf27js1p5niwv831j973wv-nodejs-6.12.3/bin/node"
./yarn/bin/yarn.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/g2pg448750pf27js1p5niwv831j973wv-nodejs-6.12.3/bin/node"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/k2n0xn43cn3m8sm7sj80xr1n5raf3fg7-node-yarn-1.3.2
patching script interpreter paths in /nix/store/k2n0xn43cn3m8sm7sj80xr1n5raf3fg7-node-yarn-1.3.2
checking for references to /build in /nix/store/k2n0xn43cn3m8sm7sj80xr1n5raf3fg7-node-yarn-1.3.2...
/nix/store/k2n0xn43cn3m8sm7sj80xr1n5raf3fg7-node-yarn-1.3.2

joachifm · 2018-02-17T06:35:27Z

cc @offlinehacker

Mic92 · 2018-02-17T09:57:57Z

pkgs/development/tools/yarn/default.nix

@@ -1,26 +0,0 @@
-{ stdenv, nodejs, fetchzip, makeWrapper }:
-
-stdenv.mkDerivation rec {


I think, this package was intentionally added by @offlinehacker

Mic92 · 2018-02-17T10:02:04Z

I have not yet understood, how building yarn from npm instead of the source tarballs solves conflict with colliding executables.

nicknovitski · 2018-02-17T18:04:37Z

Sorry, I wasn't clear: that collision is not addressed by this PR, which in effect just adds a new executable to the package's output.

(e: I mistakenly thought at first that it fixed another problem.)

I hope that building from the full source will help me create and maintain a private patched yarn package, but to be honest I'm having trouble making that work at the moment.

nicknovitski · 2018-02-17T18:09:03Z

Also, I may not have explained this properly: the "source" tarballs used by the deleted packaged and served from npm do not contain the actual source as it is stored in git; they're produced by a build process.

nicknovitski · 2018-05-04T00:39:15Z

Updated for version 1.6.0.

Mic92 · 2018-05-04T09:28:46Z

@GrahamcOfBorg build nodePackages.yarn

Mic92 · 2018-05-04T09:30:12Z

pkgs/development/node-packages/default-v6.nix

+      owner = "yarnpkg";
+      repo = "yarn";
+      rev = "v${oldAttrs.version}";
+      sha256 = "1h1qcgyx53hxkmg1z34k9b8gc3w0l670mb3jgmqlcbs6vjlqxbr8";


How do you plan to keep this checksum up-to-date when node-packages-v6.nix is updated automatically?

It would need to be updated like any other package, which is not ideal.

Mic92 · 2018-05-04T09:31:54Z

pkgs/development/tools/yarn/default.nix

-  installPhase = ''
-    mkdir -p $out/{bin,libexec/yarn/}
-    cp -R . $out/libexec/yarn
-    makeWrapper $out/libexec/yarn/bin/yarn.js $out/bin/yarn


Could your problem be also solved by adding a yarnpkg symlink?

This makeWrapper call also does not work on macOS because shebangs executables are not allowed to be scripts itself. It should use the nodejs binary instead for wrapping.

GrahamcOfBorg · 2018-05-04T13:46:00Z

Success on aarch64-linux (full log)

Attempted: nodePackages.yarn

Partial log (click to expand)

patching script interpreter paths in .
./yarn/lib/cli.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/rlijl90m74dzd0irc938wrzw9mspmxyd-nodejs-6.14.1/bin/node"
./yarn/bin/yarnpkg: interpreter directive changed from "/usr/bin/env node" to "/nix/store/rlijl90m74dzd0irc938wrzw9mspmxyd-nodejs-6.14.1/bin/node"
./yarn/bin/yarn: interpreter directive changed from "/bin/sh" to "/nix/store/a4qslf0yfs44mhnbagarn232sjnckhy0-bash-4.4-p19/bin/sh"
./yarn/bin/yarn.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/rlijl90m74dzd0irc938wrzw9mspmxyd-nodejs-6.14.1/bin/node"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/hisb09fy9g52m9ajk96aq6bzcgqnz0xp-node-yarn-1.6.0
patching script interpreter paths in /nix/store/hisb09fy9g52m9ajk96aq6bzcgqnz0xp-node-yarn-1.6.0
checking for references to /build in /nix/store/hisb09fy9g52m9ajk96aq6bzcgqnz0xp-node-yarn-1.6.0...
/nix/store/hisb09fy9g52m9ajk96aq6bzcgqnz0xp-node-yarn-1.6.0

GrahamcOfBorg · 2018-05-04T13:46:26Z

Success on x86_64-linux (full log)

Attempted: nodePackages.yarn

Partial log (click to expand)

patching script interpreter paths in .
./yarn/bin/yarn.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/gkcz8nnk0phqq8fmnyjjmkg7ds7qwd74-nodejs-6.14.1/bin/node"
./yarn/bin/yarn: interpreter directive changed from "/bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
./yarn/bin/yarnpkg: interpreter directive changed from "/usr/bin/env node" to "/nix/store/gkcz8nnk0phqq8fmnyjjmkg7ds7qwd74-nodejs-6.14.1/bin/node"
./yarn/lib/cli.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/gkcz8nnk0phqq8fmnyjjmkg7ds7qwd74-nodejs-6.14.1/bin/node"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/n551dzfr51dhy6sbi6gfgfqjwzqhldxi-node-yarn-1.6.0
patching script interpreter paths in /nix/store/n551dzfr51dhy6sbi6gfgfqjwzqhldxi-node-yarn-1.6.0
checking for references to /build in /nix/store/n551dzfr51dhy6sbi6gfgfqjwzqhldxi-node-yarn-1.6.0...
/nix/store/n551dzfr51dhy6sbi6gfgfqjwzqhldxi-node-yarn-1.6.0

GrahamcOfBorg · 2018-05-04T13:48:03Z

Success on x86_64-darwin (full log)

Attempted: nodePackages.yarn

Partial log (click to expand)

unpacking source archive /nix/store/cl1gssw8i9hc8pwpgwaaf75z5k8fk4nd-yarn-1.6.0.tgz
pinpointing versions of dependencies...
patching script interpreter paths in .
./yarn/bin/yarn: interpreter directive changed from "/bin/sh" to "/nix/store/r8bx3qf1bpncb14i9gzma4vr089pc3pv-bash-4.4-p19/bin/sh"
./yarn/bin/yarn.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/71rmmi2sq48ffnpqg80ilkv8xj2vkngi-nodejs-6.14.1/bin/node"
./yarn/bin/yarnpkg: interpreter directive changed from "/usr/bin/env node" to "/nix/store/71rmmi2sq48ffnpqg80ilkv8xj2vkngi-nodejs-6.14.1/bin/node"
./yarn/lib/cli.js: interpreter directive changed from "/usr/bin/env node" to "/nix/store/71rmmi2sq48ffnpqg80ilkv8xj2vkngi-nodejs-6.14.1/bin/node"
post-installation fixup
patching script interpreter paths in /nix/store/hbc2izis3s0vlj64g7s1kx2qnq996gbj-node-yarn-1.6.0
/nix/store/hbc2izis3s0vlj64g7s1kx2qnq996gbj-node-yarn-1.6.0

nicknovitski · 2018-05-04T18:40:45Z

I'm closing this.

My goal was to make it possible to add patches from my company's fork of yarn on top of the yarn package, but changing its src attribute to the actual source of the project still doesn't reach that goal: it turns out that buildNodePackage has an empty unpack phase by default, so the patch phase will always fail to find the files to patch.

I've made a private package which I believe achieves this goal. After I'm sure, I'll upstream any changes I think would actually be improvements.

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Feb 17, 2018

nicknovitski force-pushed the yarnpkg branch from 122d4da to c2e197a Compare February 17, 2018 04:05

Mic92 reviewed Feb 17, 2018

View reviewed changes

yarn: include full source and yarnpkg binary

b214b7e

nicknovitski force-pushed the yarnpkg branch from c2e197a to b214b7e Compare May 4, 2018 00:38

Mic92 reviewed May 4, 2018

View reviewed changes

nicknovitski closed this May 4, 2018

nicknovitski deleted the yarnpkg branch May 4, 2018 18:40

nicknovitski mentioned this pull request Jul 30, 2018

yarn: add yarnpkg bin alias output #44133

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

yarn: add yarnpkg binary to outputs and use nodejs dependency #35063

yarn: add yarnpkg binary to outputs and use nodejs dependency #35063

nicknovitski commented Feb 17, 2018 •

edited

nicknovitski commented Feb 17, 2018 •

edited

nicknovitski commented Feb 17, 2018 •

edited

joachifm commented Feb 17, 2018

GrahamcOfBorg commented Feb 17, 2018

GrahamcOfBorg commented Feb 17, 2018

GrahamcOfBorg commented Feb 17, 2018

joachifm commented Feb 17, 2018

Mic92 Feb 17, 2018

Mic92 commented Feb 17, 2018

nicknovitski commented Feb 17, 2018 •

edited

nicknovitski commented Feb 17, 2018

nicknovitski commented May 4, 2018

Mic92 commented May 4, 2018

Mic92 May 4, 2018

nicknovitski May 4, 2018

Mic92 May 4, 2018

Mic92 May 4, 2018

GrahamcOfBorg commented May 4, 2018

GrahamcOfBorg commented May 4, 2018

GrahamcOfBorg commented May 4, 2018

nicknovitski commented May 4, 2018

		@@ -1,26 +0,0 @@
		{ stdenv, nodejs, fetchzip, makeWrapper }:

		stdenv.mkDerivation rec {

yarn: add yarnpkg binary to outputs and use nodejs dependency #35063

yarn: add yarnpkg binary to outputs and use nodejs dependency #35063

Conversation

nicknovitski commented Feb 17, 2018 • edited

Motivation for this change

Things done

nicknovitski commented Feb 17, 2018 • edited

nicknovitski commented Feb 17, 2018 • edited

joachifm commented Feb 17, 2018

GrahamcOfBorg commented Feb 17, 2018

GrahamcOfBorg commented Feb 17, 2018

GrahamcOfBorg commented Feb 17, 2018

joachifm commented Feb 17, 2018

Mic92 Feb 17, 2018

Choose a reason for hiding this comment

Mic92 commented Feb 17, 2018

nicknovitski commented Feb 17, 2018 • edited

nicknovitski commented Feb 17, 2018

nicknovitski commented May 4, 2018

Mic92 commented May 4, 2018

Mic92 May 4, 2018

Choose a reason for hiding this comment

nicknovitski May 4, 2018

Choose a reason for hiding this comment

Mic92 May 4, 2018

Choose a reason for hiding this comment

Mic92 May 4, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented May 4, 2018

GrahamcOfBorg commented May 4, 2018

GrahamcOfBorg commented May 4, 2018

nicknovitski commented May 4, 2018

nicknovitski commented Feb 17, 2018 •

edited

nicknovitski commented Feb 17, 2018 •

edited

nicknovitski commented Feb 17, 2018 •

edited

nicknovitski commented Feb 17, 2018 •

edited