erlang: refactor to generalize Erlang/OTP derivations #26381

gleber · 2017-06-04T19:51:42Z

Motivation for this change

This PR does not change anything feature-wise, and instead aims at unify how different Erlang versions are built.

This is a subset of #17240 (and is a heavily-inspired re-implementation of it). It contains:

move all Beam packages into beam-packages.nix file which lives by the side of all-packages.nix;
deduplicates contents of pkgs/development/interpreters/erlang/R{16-19}.nix into a generic builder;
supports OTP build features like odbc- and javasupport

It does not contain:

a fixpoint to build Erlang packages using different OTP versions;
Elixir generalization;
LFE generalization.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

/cc @LnL7 @ankhers @ericbmerritt @yurrriq

LnL7 · 2017-06-04T19:57:23Z

pkgs/top-level/all-packages.nix

+    erlang erlang_odbc erlang_javac erlang_odbc_javac
+    elixir
+    lfe
+    erlangR16 erlangR16_odbc


Do we really have to expose all these versions toplevel? See #24047 (comment).

Probably not, but let's leave these two aspects separate. I want this PR to be purely a refactoring.

yurrriq · 2017-06-04T20:00:19Z

I'm mobile atm. Does this include the generic LFE builder from my fork?

gleber · 2017-06-04T20:17:40Z

@yurrriq It does not, on purpose. This PR is supposed to be purely a refactoring.

gleber · 2017-06-05T13:06:19Z

It builds on Ubuntu, but still fails on NixOs, debugging

ankhers · 2017-06-05T13:43:31Z

I'm also getting a failure on OSX 10.11.6

You can see the entire output of nox-review here

gleber · 2017-06-07T20:25:27Z

It compiles now both on Ubuntu and NixOs (or more specifically all Erlang/OTP derivations compile and nox preview has been running for 3 hours now and still did not fail).

Please review.

LnL7 · 2017-06-04T20:13:00Z

pkgs/top-level/beam-packages.nix

+rec {
+  lib = import ../development/beam-modules/lib.nix { inherit pkgs; };
+
+  inherit (pkgs) darwin wxGTK30;


these can be moved to the arguments, now they end up in the beam attrset

I replaced 'foo' with 'pkgs.foo' everywhere below.

Moved these to arguments of this file (i.e. lambda defined at top level of this file)

LnL7 · 2017-06-07T23:34:53Z

pkgs/development/interpreters/erlang/generic-builder.nix

+
+  buildInputs = [ perl gnum4 ncurses openssl autoconf libxslt libxml2 makeWrapper ]
+    ++ optionals wxSupport wxPackages2
+    ++ optional odbcSupport odbcPackages


use optionals since it's already a list, nested buildInputs work but it should be avoided

LnL7 · 2017-06-07T23:48:17Z

pkgs/development/beam-modules/lib.nix

+
+  Please note that "mkDerivation" defined here is the one called from R16.nix and similar files.
+  */
+  callErlang = drv: gargs: args: pkgs.callPackage drv (


Using makeOverridable for the arguments instead here would be more flexible and also makes the top-level definitions a bit nicer.

LnL7@de6ccfe

I have applied this patch

ankhers · 2017-06-08T00:38:27Z

nox-review passed on OSX 10.11.6

This applies to Erlang, Elixir and LFE packages. beam-packages provides interpreters and packages sets separately. This is in preparation of generalizing BEAM interpreters definitions.

Switch official Erlang versions to use a common builder.

gleber · 2017-06-08T09:32:14Z

@LnL7 All fixed. Please take a look again (PTAL)

gleber · 2017-06-08T20:07:14Z

@LnL7 I have added a commit which moves basho's fork over too. PTAL

ankhers · 2017-06-09T13:43:29Z

Failing again on OSX for the basho stuff. https://gist.github.com/ankhers/e36e7e6f4908446ec6de6c0ee30ff592

LnL7 · 2017-06-09T18:43:19Z

I'm seeing an issue with /usr/bin/env again for R16

/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/make/emd2exml /tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/HOWTO/DTRACE.md DTRACE.xml
/nix/store/wb34dgkpmnssjkq7yj4qbjqxpnapq0lw-bash-4.4-p12/bin/bash: /tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/make/emd2exml: /usr/bin/env: bad interpreter: No such file or directory
make[5]: *** [Makefile:86: DTRACE.xml] Error 126
make[5]: Leaving directory '/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/lib/runtime_tools/doc/src'
make[4]: *** [/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/make/otp_release_targets.mk:146: release_docs] Error 2
make[4]: Leaving directory '/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/lib/runtime_tools/doc/src'
make[3]: *** [/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/make/otp_subdir.mk:28: release_docs] Error 2
make[3]: Leaving directory '/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/lib/runtime_tools'
make[2]: *** [/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/make/otp_subdir.mk:28: release_docs] Error 2
make[2]: Leaving directory '/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/lib'
make[1]: *** [Makefile:408: release_docs] Error 2
make[1]: Leaving directory '/tmp/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src'
make: *** [Makefile:929: install-docs] Error 2
<3>builder for ‘/nix/store/p13fgcipawc4sxdw54yxb71zxqvaip45-erlang-16B02.drv’ failed with exit code 2

gleber · 2017-06-10T14:41:04Z

@LnL Fixed emd2exml failure for basho version.

@ankhers I also submitted a version which use clang instead of gcc on Darwin for the diameterc.

LnL7 · 2017-06-10T19:05:55Z

Another shebang issue (on darwin), not sure what's going on here.

../bin/diameterc -o gen -i ../ebin dict/base_rfc3588.dia
/nix/store/vkh1qq7m28v6wajva1k50a4i8x3l0r9c-bash-4.4-p12/bin/bash: ../bin/diameterc: /private/var/folders/xb/qd021gp54z3d0_rsjn6nhhn40000gn/T/nix-build-erlang-16B02.drv-0/otp-OTP_R16B02_basho8-src/bin/x86_64-ap: bad interpreter: No such file or directory
make[3]: *** [Makefile:123: gen/diameter_gen_base_rfc3588.erl] Error 126

gleber · 2017-06-11T12:18:58Z

@LnL7 That's this line:
https://github.com/NixOS/nixpkgs/pull/26381/files#diff-2b1b48d1f8b621ebb9fbae7dfc03bb6eR20

Previously Basho Erlang fork would require another Erlang to be already installed so that the diameterc script runs correctly during build phase.

There are three options:

disable diameterc use somehow (not sure if it is necessary in practice);
use anotehr erlang when building basho erlang fork (as it used to be);
find a way to predict where erlang build phase would put escript script.

I think I have found a solution using the last option. Can you retest?

gleber · 2017-06-11T12:23:11Z

I think I have found a less hacky way which should be more resilient. Can you retest it on OS X?

ankhers · 2017-06-11T17:41:59Z

Nox review just finished and everything built on OSX.

LnL7

Everything looks good now.

LnL7 · 2017-06-11T18:36:28Z

Thanks for finishing this!

gleber · 2017-06-11T18:53:53Z

Yay! There's still plenty to do

LnL7 reviewed Jun 4, 2017

View reviewed changes

LnL7 added 6.topic: erlang 8.has: clean-up labels Jun 4, 2017

gleber force-pushed the generalize-beam branch 5 times, most recently from c9d5451 to 39cf2ae Compare June 7, 2017 17:17

gleber changed the title ~~WIP: erlang: refactor to generalize Erlang/OTP derivations~~ erlang: refactor to generalize Erlang/OTP derivations Jun 7, 2017

LnL7 reviewed Jun 7, 2017

View reviewed changes

gleber added 2 commits June 8, 2017 11:28

erlang: Move all BEAM packages into beam-packages.

96f4cac

This applies to Erlang, Elixir and LFE packages. beam-packages provides interpreters and packages sets separately. This is in preparation of generalizing BEAM interpreters definitions.

erlang: Generalize Erlang/OTP derivations.

3426c88

Switch official Erlang versions to use a common builder.

gleber force-pushed the generalize-beam branch from 39cf2ae to ff80b17 Compare June 8, 2017 09:31

erlang: use makeOverridable for customizations

e88a89a

gleber force-pushed the generalize-beam branch from ac3d3f0 to 60682c0 Compare June 8, 2017 20:13

gleber force-pushed the generalize-beam branch from 60682c0 to a800173 Compare June 9, 2017 15:16

gleber force-pushed the generalize-beam branch from a800173 to d4577e5 Compare June 10, 2017 14:16

erlang: move erlang_basho_R16B02 to Erlang's general builder.

c9cb640

gleber force-pushed the generalize-beam branch from d4577e5 to c9cb640 Compare June 11, 2017 11:31

LnL7 mentioned this pull request Jun 11, 2017

WIP: generalize beam related interpreters and packages #17240

Closed

11 tasks

LnL7 approved these changes Jun 11, 2017

View reviewed changes

LnL7 merged commit 4010313 into NixOS:master Jun 11, 2017

gleber mentioned this pull request Jun 17, 2017

erlang: refactor: build packages per Erlang/OTP version. #26668

Merged

8 tasks

ankhers mentioned this pull request Jun 19, 2017

Generalize building of Elixir interpreter #26716

Merged

7 tasks

ankhers mentioned this pull request Jul 7, 2017

Cannot search beamPackages #26279

Closed

gleber mentioned this pull request Jul 8, 2017

BEAM packages support umbrella tracking issue (Erlang, Elixir, LFE, etc.) #11748

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

erlang: refactor to generalize Erlang/OTP derivations #26381

erlang: refactor to generalize Erlang/OTP derivations #26381

gleber commented Jun 4, 2017 •

edited

Loading

LnL7 Jun 4, 2017

gleber Jun 4, 2017

yurrriq commented Jun 4, 2017

gleber commented Jun 4, 2017

gleber commented Jun 5, 2017

ankhers commented Jun 5, 2017

gleber commented Jun 7, 2017

LnL7 Jun 4, 2017

gleber Jun 8, 2017

gleber Jun 8, 2017

LnL7 Jun 7, 2017

gleber Jun 8, 2017

LnL7 Jun 7, 2017

gleber Jun 8, 2017

ankhers commented Jun 8, 2017

gleber commented Jun 8, 2017

gleber commented Jun 8, 2017

ankhers commented Jun 9, 2017

LnL7 commented Jun 9, 2017

gleber commented Jun 10, 2017

LnL7 commented Jun 10, 2017 •

edited

Loading

gleber commented Jun 11, 2017

gleber commented Jun 11, 2017

ankhers commented Jun 11, 2017

LnL7 left a comment

LnL7 commented Jun 11, 2017

gleber commented Jun 11, 2017

erlang: refactor to generalize Erlang/OTP derivations #26381

erlang: refactor to generalize Erlang/OTP derivations #26381

Conversation

gleber commented Jun 4, 2017 • edited Loading

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yurrriq commented Jun 4, 2017

gleber commented Jun 4, 2017

gleber commented Jun 5, 2017

ankhers commented Jun 5, 2017

gleber commented Jun 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ankhers commented Jun 8, 2017

gleber commented Jun 8, 2017

gleber commented Jun 8, 2017

ankhers commented Jun 9, 2017

LnL7 commented Jun 9, 2017

gleber commented Jun 10, 2017

LnL7 commented Jun 10, 2017 • edited Loading

gleber commented Jun 11, 2017

gleber commented Jun 11, 2017

ankhers commented Jun 11, 2017

LnL7 left a comment

Choose a reason for hiding this comment

LnL7 commented Jun 11, 2017

gleber commented Jun 11, 2017

gleber commented Jun 4, 2017 •

edited

Loading

LnL7 commented Jun 10, 2017 •

edited

Loading