-
-
Notifications
You must be signed in to change notification settings - Fork 15.5k
erlang: refactor: build packages per Erlang/OTP version. #26668
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@gleber, thanks for your PR! By analyzing the history of the files in this pull request, we identified @ericbmerritt and @LnL7 to be potential reviewers. |
from IRC: gleber_: Does that actually change anything? nix-review claims nothing changed. |
Just glancing at this on my phone for now, but does it enable easy building of arbitrary Hex packages as dependencies (and injecting them into |
@yurrriq I think it does provides some parts of what you want:
There is no nice mechanism to do @LnL7 @ankhers @ericbmerritt This PR is ready to be reviewed. |
88c66a8
to
815a276
Compare
doc/languages-frameworks/beam.xml
Outdated
<section xml:id="beam-introduction"> | ||
<title>Introduction</title> | ||
<para> | ||
In this document and related Nix expressions we use the term | ||
<emphasis>Beam</emphasis> to describe the environment. Beam is | ||
<emphasis>Beam</emphasis> to describe the environment. BEAM is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably change all Beam
instances to BEAM
if we are going to change a couple of them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I replace it only where it explicitly talks about BEAM the virtual machine, other than that Beam is used for stuff in Nixpkgs very consistently.
doc/languages-frameworks/beam.xml
Outdated
</para> | ||
<para> | ||
We also provide <literal>beam.packages.erlang.callPackage</literal>, which | ||
simplifies writng Beam package definitions, by injecting all packages from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor typo "writng".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
doc/languages-frameworks/beam.xml
Outdated
<literal>buildRebar3</literal>. We use this function to make a | ||
derivation that understands how to build the rebar3 project. For | ||
example, the epression we use to build the <link | ||
There is a Nix functional called <literal>buildRebar3</literal> (defined |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be "There is a Nix function called..."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
doc/languages-frameworks/beam.xml
Outdated
sha256 = | ||
"08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33"; | ||
builder = buildMix; | ||
beamDeps = [ plug absinthe]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just stylistic, but maybe put a space between "absinthe" and the closing "]".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
doc/languages-frameworks/beam.xml
Outdated
}; | ||
drv = beamPackages.callPackage f {}; | ||
|
||
in | ||
drv | ||
</programlisting> | ||
<section xml:id="building-in-a-shell"> | ||
<title>Building in a shell</title> | ||
<title>Building in a shell (for mix projects)</title> | ||
<para> | ||
We can leveral the support of the Derivation, regardless of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is meant to say "We can leverage the support..."?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have not played with it yet, but it looks good.
let | ||
parse = drv: (builtins.parseDrvName drv).version; | ||
in builtins.replaceStrings ["B" "-"] ["." "."] ( | ||
if stdenv.lib.isString x |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
doesn't really matter, but isString
is a builtin so you don't need stdenv.lib.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get this:
~/c/nixpkgs ❯❯❯ nix-build --show-trace -A beam.packages.erlangR18.elixir /home/gleber/code/nixpkgs
error: while evaluating ‘versionAtLeast’ at /home/gleber/code/nixpkgs/lib/strings.nix:363:24, called from /home/gleber/code/nixpkgs/pkgs/development/beam-modules/default.nix:40:21:
while evaluating ‘versionOlder’ at /home/gleber/code/nixpkgs/lib/strings.nix:351:22, called from /home/gleber/code/nixpkgs/lib/strings.nix:363:29:
while evaluating ‘callPackageWith’ at /home/gleber/code/nixpkgs/lib/customisation.nix:95:35, called from /home/gleber/code/nixpkgs/pkgs/development/beam-modules/default.nix:6:9:
while evaluating ‘makeOverridable’ at /home/gleber/code/nixpkgs/lib/customisation.nix:54:24, called from /home/gleber/code/nixpkgs/lib/customisation.nix:99:8:
undefined variable ‘isString’ at /home/gleber/code/nixpkgs/pkgs/development/beam-modules/lib.nix:20:10
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Made it into builtins.isString instead
lfe = callPackage ../development/interpreters/lfe { }; | ||
# Other Beam languages. These are built with `beam.interpreters.erlang`. To | ||
# access for example elixir built with different version of Erlang, use | ||
# `beam.packages.erlangR19.elixir`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice!
This change introduces a fixpoint, which allows to do deep override when building packages defined in pkgs/development/beam-modules/default.hex. This allows to provide beam.packages.erlang{,R16,R17,R18,R19} which contains the same packages built with different Erlang/OTP versions. Top-level attribute beamPackages points at beam.packages.erlangR18, the same applies to other top-level Erlang packages. TODO: - beam.packages.erlang{R16,R17} is almost useless, since rebar/rebar3 does not build using these versions; - all packages in beam.packages which use buildMix are actually built with erlangR18; - update documentation.
815a276
to
b664866
Compare
This makes beam.package.erlangR19.abnf to be actually built with R19, instead of the default R18. It means that Elixir and LFE are provided in two versions, one built with R18 and with R19. Please note that Elixir does not build with R16 and R17 - trying to access beam.packages.erlang{R16,R17}.elixir will throw an error.
The documentation got a bit stale compared to actual contents of nixpkgs. This commit focuses on updating existing docs, not on making sure all details of beam packages are covered.
b664866
to
0fccd5b
Compare
Rebased, fixed all comments (by addressing them in appropriate commits, hence commit ids have changed). |
|
Current state of PR built successfully built on NixOS with sandboxes: $ nix-shell -p nox --run "nox-review pr 26668" |
This is working great for me, but I want to use {
nixpkgs.config.packageOverrides = pkgs: rec {
beam = pkgs.beam // rec {
interpreters = pkgs.beam.interpreters // {
erlang = pkgs.beam.interpreters.erlangR19;
};
packages = pkgs.beam.packages // {
erlang = pkgs.beam.packagesWith interpreters.erlang;
};
};
};
} Prior to the generalization, I just did this: {
nixpkgs.config.packageOverrides = pkgs: rec {
erlang = pkgs.erlangR19;
};
} Maybe you can show me a better a way and maybe we can add that to the docs. |
@yurrriq That's a good piece of feedback! I am not sure how this can be done better. I see two problems:
I will take a look sometime this week. |
I think would be useful to include erlang in the beamPackages attrset, then people can refer to that to get the correct version. @yurrriq Note that overriding the interpreter like that has undesirable side effects, it causes top-level packages like couchdb to potentially break. |
Is there anything else that needs to be done before this can be merged? |
Re @LnL7's point: I agree. Really what I'm after is for my system-wide |
@yurrriq I thought this would prevent the need for override? I thought that this change allowed you to install |
Right, I think so. It'd be good to add a couple similar examples to the docs. /me checks the existing docs |
I've got some pedantic edits (spelling, grammar, continuity, etc) for the docs and can add some examples like I've mentioned. Otherwise I think this is good to go. Thinking on it more, I don't think we need to support such dramatic global overrides as I've got on my system now (see above). @gleber, I'll open a PR to your branch, unless you have another suggestion. |
Feel free to merge the PR as is - I think it already provides good value. I will continue refining it in various ways. @yurrriq, sounds good. I will incorporate it into my work (regardless of the timing which PR gets merges/proposed first). |
Improve beam docs: * correct spelling * update per pandoc changes * capitalize titles * capitalize BEAM throughout and use "the BEAM" when referring to the virtual machine. * tweak grammar and phrasing * reformat build-tools-rebar3 section * add more links * re-wrap <para>s Also update <programlisting>s * normalize whitespace * don't double quote homepage * use $ in all shell snippets
We've merged in spelling/phrasing tweaks, including @ankhers's suggestion of normalizing BEAM throughout, but I didn't add any examples as discussed. I should be able to get to that later today. |
@@ -26,7 +26,7 @@ pkgs.stdenv.mkDerivation { | |||
extraHeader = ''xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" ''; | |||
in '' | |||
{ | |||
pandoc '${inputFile}' -w docbook ${lib.optionalString useChapters "--chapters"} \ | |||
pandoc '${inputFile}' -w docbook ${lib.optionalString useChapters "--top-level-division=chapter"} \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's this for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I got a warning that --chapters
was deprecated in favor of --top-level-division=chapter
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think improving the override is relevant enough to fix it now.
cuter = callPackage ../tools/erlang/cuter {}; | ||
relxExe = callPackage ../tools/erlang/relx-exe {}; | ||
}; | ||
in fix' (extends overrides packages) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gleber For the sake of consistency, you might want to leverage makeExtensible
here. It's presently used by both the Haskell package sets and linuxPackages
sets.
The only change you'd need to make is replace this line with:
stdenv.lib.makeExtensible (extends overrides packages)
You could then remove the overrides
argument, as you'll then be able to do beam.packages.erlang.extend (self: super: { /* ... */ })
Aside from aesthetics, I think there's a growing sentiment that "override" is a little bit overloaded, as it has historically been used to mean two different things:
- overriding the inputs into an expression, and
- overriding the packages within a package set
The use of "overriding" in the latter isn't exactly accurate, anyway -- you often want to "extend" the package set (and overriding can be seen as a subset of that concern). So the hope is that consistently using .override
for modifying inputs and .extend
for extending packages sets will make things more intuitive, and having one way of going about it (lib.makeExtensible
) will ensure that each package set gets it right, for free.
But, I'm not much involved in BEAM in Nixpkgs, so I figured I'd make the suggestion and leave the choice at your discretion :).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for deep explanation, I am new to Nixpkgs so this kind of tribal knowledge is really hard for me to come by. I will include the suggested change into next PR.
A quick question: do you have any idea why makeOverridable
does not have documentation? I could not find it neither in the code neither in the docs. Now, give that there are two distinct mechanisms which should be used for different situations, a documentation explaining both would be very useful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for deep explanation
Sure thing -- thanks for the work on BEAM support 😄
do you have any idea why makeOverridable does not have documentation?
Probably just oversight. I've opened a #26869 to (hopefully soon) resolve this.
Use standardized implementation of attribute set extensibility mechanism instead of manually re-implementing it. Suggested by @cstrahan at NixOS#26668.
FYI #26813
|
This change introduces a fixpoint, which allows to do deep override when
building packages defined in pkgs/development/beam-modules/default.hex.
This allows to provide beam.packages.erlang{,R16,R17,R18,R19} which
contains the same packages built with different Erlang/OTP versions.
Top-level attribute beamPackages points at beam.packages.erlangR18, the
same applies to other top-level Erlang packages.
Motivation for this change
This is a continuation of #26381 based on #17240
This does not yet generalize Elixir nor LFE interpreters.
Things done
(nix.useSandbox on NixOS,
or option
build-use-sandbox
innix.conf
on non-NixOS)
nix-shell -p nox --run "nox-review wip --against 9fb87f9c9d0eaeb97773e53e3740231d52b1104f"
while being at 9f7ae27361b71fe94af88bd165da47df8fe1c1a4nix-shell -p nox --run "nox-review pr 26668"
./result/bin/
)/cc @LnL7 @ericbmerritt @yurrriq @ankhers