Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

erlang: refactor: build packages per Erlang/OTP version. #26668

Merged
merged 5 commits into from
Jun 22, 2017

Conversation

gleber
Copy link
Contributor

@gleber gleber commented Jun 17, 2017

This change introduces a fixpoint, which allows to do deep override when
building packages defined in pkgs/development/beam-modules/default.hex.
This allows to provide beam.packages.erlang{,R16,R17,R18,R19} which
contains the same packages built with different Erlang/OTP versions.

Top-level attribute beamPackages points at beam.packages.erlangR18, the
same applies to other top-level Erlang packages.

Motivation for this change

This is a continuation of #26381 based on #17240

This does not yet generalize Elixir nor LFE interpreters.

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip --against 9fb87f9c9d0eaeb97773e53e3740231d52b1104f" while being at 9f7ae27361b71fe94af88bd165da47df8fe1c1a4
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review pr 26668"
  • Tested execution of many binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

/cc @LnL7 @ericbmerritt @yurrriq @ankhers

@mention-bot
Copy link

@gleber, thanks for your PR! By analyzing the history of the files in this pull request, we identified @ericbmerritt and @LnL7 to be potential reviewers.

@gleber
Copy link
Contributor Author

gleber commented Jun 17, 2017

from IRC:

gleber_: Does that actually change anything? nix-review claims nothing changed.
<gleber_>: Ankhers: it does not change anything among top-level packages, but it does introduce many packages, which are not "exported"
<gleber_>: Ankhers: that's also one thing that I'd like to get feedback on. Is this really useful to have such "non-exported" packages defined at all? Should they be "exported"? Does this apply to only binaries, or to libraries too?

@yurrriq
Copy link
Member

yurrriq commented Jun 17, 2017

Just glancing at this on my phone for now, but does it enable easy building of arbitrary Hex packages as dependencies (and injecting them into beam.packages or whatever)? That's something I've wanted for a while.

@gleber
Copy link
Contributor Author

gleber commented Jun 18, 2017

@yurrriq I think it does provides some parts of what you want:

  • beam.packages.erlangRXX.buildHex makes it easy to build a package from Hex;
  • beam.packages.erlangRXX contains many packages from Hex (although this list is very outdated as you know).

There is no nice mechanism to do beam.packages.erlang.override to actually inject new packages into this package set, but it is probably not hard to add.

@LnL7 @ankhers @ericbmerritt This PR is ready to be reviewed.

@gleber gleber changed the title WIP: erlang: refactor: build packages per Erlang/OTP version. erlang: refactor: build packages per Erlang/OTP version. Jun 18, 2017
<section xml:id="beam-introduction">
<title>Introduction</title>
<para>
In this document and related Nix expressions we use the term
<emphasis>Beam</emphasis> to describe the environment. Beam is
<emphasis>Beam</emphasis> to describe the environment. BEAM is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably change all Beam instances to BEAM if we are going to change a couple of them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replace it only where it explicitly talks about BEAM the virtual machine, other than that Beam is used for stuff in Nixpkgs very consistently.

</para>
<para>
We also provide <literal>beam.packages.erlang.callPackage</literal>, which
simplifies writng Beam package definitions, by injecting all packages from
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor typo "writng".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

<literal>buildRebar3</literal>. We use this function to make a
derivation that understands how to build the rebar3 project. For
example, the epression we use to build the <link
There is a Nix functional called <literal>buildRebar3</literal> (defined
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be "There is a Nix function called..."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

sha256 =
"08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
builder = buildMix;
beamDeps = [ plug absinthe];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just stylistic, but maybe put a space between "absinthe" and the closing "]".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

};
drv = beamPackages.callPackage f {};

in
drv
</programlisting>
<section xml:id="building-in-a-shell">
<title>Building in a shell</title>
<title>Building in a shell (for mix projects)</title>
<para>
We can leveral the support of the Derivation, regardless of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is meant to say "We can leverage the support..."?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Member

@LnL7 LnL7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not played with it yet, but it looks good.

let
parse = drv: (builtins.parseDrvName drv).version;
in builtins.replaceStrings ["B" "-"] ["." "."] (
if stdenv.lib.isString x
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't really matter, but isString is a builtin so you don't need stdenv.lib.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get this:

~/c/nixpkgs ❯❯❯ nix-build --show-trace -A beam.packages.erlangR18.elixir /home/gleber/code/nixpkgs

error: while evaluating ‘versionAtLeast’ at /home/gleber/code/nixpkgs/lib/strings.nix:363:24, called from /home/gleber/code/nixpkgs/pkgs/development/beam-modules/default.nix:40:21:
while evaluating ‘versionOlder’ at /home/gleber/code/nixpkgs/lib/strings.nix:351:22, called from /home/gleber/code/nixpkgs/lib/strings.nix:363:29:
while evaluating ‘callPackageWith’ at /home/gleber/code/nixpkgs/lib/customisation.nix:95:35, called from /home/gleber/code/nixpkgs/pkgs/development/beam-modules/default.nix:6:9:
while evaluating ‘makeOverridable’ at /home/gleber/code/nixpkgs/lib/customisation.nix:54:24, called from /home/gleber/code/nixpkgs/lib/customisation.nix:99:8:
undefined variable ‘isString’ at /home/gleber/code/nixpkgs/pkgs/development/beam-modules/lib.nix:20:10

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made it into builtins.isString instead

lfe = callPackage ../development/interpreters/lfe { };
# Other Beam languages. These are built with `beam.interpreters.erlang`. To
# access for example elixir built with different version of Erlang, use
# `beam.packages.erlangR19.elixir`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

This change introduces a fixpoint, which allows to do deep override when
building packages defined in pkgs/development/beam-modules/default.hex.
This allows to provide beam.packages.erlang{,R16,R17,R18,R19} which
contains the same packages built with different Erlang/OTP versions.

Top-level attribute beamPackages points at beam.packages.erlangR18, the
same applies to other top-level Erlang packages.

TODO:
- beam.packages.erlang{R16,R17} is almost useless, since rebar/rebar3
  does not build using these versions;
- all packages in beam.packages which use buildMix are actually built
  with erlangR18;
- update documentation.
This makes beam.package.erlangR19.abnf to be actually built with R19,
instead of the default R18. It means that Elixir and LFE are provided in
two versions, one built with R18 and with R19.

Please note that Elixir does not build with R16 and R17 - trying to
access beam.packages.erlang{R16,R17}.elixir will throw an error.
The documentation got a bit stale compared to actual contents of
nixpkgs. This commit focuses on updating existing docs, not on making
sure all details of beam packages are covered.
@gleber
Copy link
Contributor Author

gleber commented Jun 19, 2017

Rebased, fixed all comments (by addressing them in appropriate commits, hence commit ids have changed).

@ankhers
Copy link
Contributor

ankhers commented Jun 19, 2017

nox-review passed on OSX.

@gleber
Copy link
Contributor Author

gleber commented Jun 19, 2017

Current state of PR built successfully built on NixOS with sandboxes:

$ nix-shell -p nox --run "nox-review pr 26668"
...
Result in /run/user/1000/nox-review-982bk1l1
total 0
lrwxrwxrwx 1 gleber users 57 Jun 19 20:36 result -> /nix/store/lnxk39nmhg03pi5lniz3v8kih6vz0sxa-hex2nix-0.0.5
lrwxrwxrwx 1 gleber users 56 Jun 19 20:36 result-2 -> /nix/store/4npkcs2ds1dga0903mjbvj4ya3h3rq77-rebar3-3.3.2
lrwxrwxrwx 1 gleber users 59 Jun 19 20:36 result-3 -> /nix/store/mfdf6z2hjxpyk1rrnj0knpzxxdnrmlva-relx-exe-3.18.0
lrwxrwxrwx 1 gleber users 53 Jun 19 20:36 result-4 -> /nix/store/wx7bvpfimz0n68dkgjkai0kajxsx1h0a-lfe-1.2.1
lrwxrwxrwx 1 gleber users 56 Jun 19 20:36 result-5 -> /nix/store/n3j5kw7glc12g9gdvj83h39airmfinp1-rebar3-3.3.2
 1

@yurrriq
Copy link
Member

yurrriq commented Jun 19, 2017

This is working great for me, but I want to use erlangR19 globally and I feel like this override is wrong or a bit clunky:

{
  nixpkgs.config.packageOverrides = pkgs: rec {
    beam = pkgs.beam // rec {
      interpreters = pkgs.beam.interpreters // {
        erlang = pkgs.beam.interpreters.erlangR19;
      };
      packages = pkgs.beam.packages // {
        erlang = pkgs.beam.packagesWith interpreters.erlang;
      };
    };
  };
}

Prior to the generalization, I just did this:

{
  nixpkgs.config.packageOverrides = pkgs: rec {
    erlang = pkgs.erlangR19;
  };
}

Maybe you can show me a better a way and maybe we can add that to the docs.

@gleber
Copy link
Contributor Author

gleber commented Jun 20, 2017

@yurrriq That's a good piece of feedback! I am not sure how this can be done better.

I see two problems:

  • if you override beam.interpreters.erlang it should switch beam.packages.erlang to the same version automagically (this is probably fixable by adding a fixpoint inside beam itself);
  • the fact that you need to manually go first into beam, then into interpreters to do the override of beam.interpreters.erlang is a bit ugly - I suspect some sort of override function can be added inside beam to make this easier.

I will take a look sometime this week.

@LnL7
Copy link
Member

LnL7 commented Jun 20, 2017

I think would be useful to include erlang in the beamPackages attrset, then people can refer to that to get the correct version.

@yurrriq Note that overriding the interpreter like that has undesirable side effects, it causes top-level packages like couchdb to potentially break.

@gleber
Copy link
Contributor Author

gleber commented Jun 20, 2017

@LnL7 I would be thankful if you provide guidance how to best structure ability to do the overrides on the levels where it is missing, as @yurrriq pointed out.

@ankhers
Copy link
Contributor

ankhers commented Jun 22, 2017

Is there anything else that needs to be done before this can be merged?

@yurrriq
Copy link
Member

yurrriq commented Jun 22, 2017

@ankhers, I think we should at least add some documentation re: clean(er) overrides, including the caveat @LnL7 pointed out, and maybe clean that up in general, perhaps as @gleber mentioned.

@yurrriq
Copy link
Member

yurrriq commented Jun 22, 2017

Re @LnL7's point: I agree. Really what I'm after is for my system-wide erl to be R19 and for LFE to be built with R19 too. I think I could instead include beam.interpreters.erlangR19 in my systemPackages and override (or build a separate) LFE to use R19. If nothing else, I think some related examples in the docs would be highly beneficial.

@ankhers
Copy link
Contributor

ankhers commented Jun 22, 2017

@yurrriq I thought this would prevent the need for override? I thought that this change allowed you to install beam.packages.erlangR19.lfe to have it build on R19.

@yurrriq
Copy link
Member

yurrriq commented Jun 22, 2017

Right, I think so. It'd be good to add a couple similar examples to the docs. /me checks the existing docs

@yurrriq
Copy link
Member

yurrriq commented Jun 22, 2017

I've got some pedantic edits (spelling, grammar, continuity, etc) for the docs and can add some examples like I've mentioned. Otherwise I think this is good to go.

Thinking on it more, I don't think we need to support such dramatic global overrides as I've got on my system now (see above).

@gleber, I'll open a PR to your branch, unless you have another suggestion.

@gleber
Copy link
Contributor Author

gleber commented Jun 22, 2017

Feel free to merge the PR as is - I think it already provides good value. I will continue refining it in various ways.

@yurrriq, sounds good. I will incorporate it into my work (regardless of the timing which PR gets merges/proposed first).

@yurrriq
Copy link
Member

yurrriq commented Jun 22, 2017

@gleber: gleber#1

Improve beam docs:
 * correct spelling
 * update per pandoc changes
 * capitalize titles
 * capitalize BEAM throughout and use "the BEAM" when referring to the virtual machine.
 * tweak grammar and phrasing
 * reformat build-tools-rebar3 section
 * add more links
 * re-wrap <para>s

Also update <programlisting>s
* normalize whitespace
* don't double quote homepage
* use $ in all shell snippets
@yurrriq
Copy link
Member

yurrriq commented Jun 22, 2017

We've merged in spelling/phrasing tweaks, including @ankhers's suggestion of normalizing BEAM throughout, but I didn't add any examples as discussed. I should be able to get to that later today.

@@ -26,7 +26,7 @@ pkgs.stdenv.mkDerivation {
extraHeader = ''xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" '';
in ''
{
pandoc '${inputFile}' -w docbook ${lib.optionalString useChapters "--chapters"} \
pandoc '${inputFile}' -w docbook ${lib.optionalString useChapters "--top-level-division=chapter"} \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this for?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got a warning that --chapters was deprecated in favor of --top-level-division=chapter.

Copy link
Member

@LnL7 LnL7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think improving the override is relevant enough to fix it now.

@LnL7 LnL7 merged commit aba574c into NixOS:master Jun 22, 2017
cuter = callPackage ../tools/erlang/cuter {};
relxExe = callPackage ../tools/erlang/relx-exe {};
};
in fix' (extends overrides packages)
Copy link
Contributor

@cstrahan cstrahan Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gleber For the sake of consistency, you might want to leverage makeExtensible here. It's presently used by both the Haskell package sets and linuxPackages sets.

The only change you'd need to make is replace this line with:

stdenv.lib.makeExtensible (extends overrides packages)

You could then remove the overrides argument, as you'll then be able to do beam.packages.erlang.extend (self: super: { /* ... */ })

Aside from aesthetics, I think there's a growing sentiment that "override" is a little bit overloaded, as it has historically been used to mean two different things:

  • overriding the inputs into an expression, and
  • overriding the packages within a package set

The use of "overriding" in the latter isn't exactly accurate, anyway -- you often want to "extend" the package set (and overriding can be seen as a subset of that concern). So the hope is that consistently using .override for modifying inputs and .extend for extending packages sets will make things more intuitive, and having one way of going about it (lib.makeExtensible) will ensure that each package set gets it right, for free.

But, I'm not much involved in BEAM in Nixpkgs, so I figured I'd make the suggestion and leave the choice at your discretion :).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for deep explanation, I am new to Nixpkgs so this kind of tribal knowledge is really hard for me to come by. I will include the suggested change into next PR.

A quick question: do you have any idea why makeOverridable does not have documentation? I could not find it neither in the code neither in the docs. Now, give that there are two distinct mechanisms which should be used for different situations, a documentation explaining both would be very useful.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for deep explanation

Sure thing -- thanks for the work on BEAM support 😄

do you have any idea why makeOverridable does not have documentation?

Probably just oversight. I've opened a #26869 to (hopefully soon) resolve this.

gleber added a commit to gleber/nixpkgs that referenced this pull request Jun 24, 2017
Use standardized implementation of attribute set extensibility mechanism
instead of manually re-implementing it.

Suggested by @cstrahan at NixOS#26668.
@gleber
Copy link
Contributor Author

gleber commented Jun 26, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants