rebar3: 3.4.3 -> 3.6.1 #43430

tazjin · 2018-07-12T19:54:17Z

Left to do:

get erlang-nix/hex-pm-registry-snapshots/Add fonts from the DOSEmu project. #4 merged or host the registry snapshot elsewhere
~~update hex-packages.nix (currently running on my machine)~~ (will be a followup PR)
ensure that existing rebar-based packages still build correctly

Updates rebar3 to version 3.6.1, which amongst other things introduces
support for rebar3 on Erlang/OTP 21.

Changes made:

rebar3 and dependencies updated to new versions
rebar3 hermeticity patch updated to apply against new version
hex package registry snapshot updated

Motivation for this change

Nixpkgs already includes Erlang/OTP 21. The current included version of rebar3 is incompatible with the Erlang release, so an update is required.

This fixes #38966.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

gleber · 2018-07-12T20:20:06Z

@GrahamcOfBorg build rebar3 rebar3-open

tazjin · 2018-07-12T20:30:44Z

@gleber nox-review is showing some packages as failing (e.g. hex2nix itself). That package specifically has no version bounds on its dependencies and it seems that rebar3 will attempt to update its dependencies (erlware-commons in this case) to the latest version (which isn't yet in hex-packages.nix).

Do you know if it's expected behaviour that rebar3 wants non-constrained packages to be the latest version?

I'm currently generating an updated hex-packages.nix, but the process is failing every few minutes with a TLS error. Seems like it picks up again when using the -c flag though.

Edit: Hmm, nope, it doesn't pick up where it left off. It seems to always fail on the same package:

Fetching phoenix_jsroutes 0.0.4 details from hex.pm.

=ERROR REPORT==== 12-Jul-2018::22:35:23 ===
SSL: hello: ssl_alert.erl:88:Fatal error: internal error
escript: exception error: no case clause matching
                 {error,{conn_failed,{error,{tls_alert,"internal error"}}}}
  in function  h2n_fetcher:get_app_detail_from_hex_pm/1 (/build/source/_build/default/lib/hex2nix/src/h2n_fetcher.erl, line 175)
  in call from h2n_fetcher:decorate_app/2 (/build/source/_build/default/lib/hex2nix/src/h2n_fetcher.erl, line 146)
  in call from lists:filtermap/2 (lists.erl, line 1317)
  in call from h2n_fetcher:update_with_information_from_hex_pm/3 (/build/source/_build/default/lib/hex2nix/src/h2n_fetcher.erl, line 66)
  in call from hex2nix:do_main/1 (/build/source/_build/default/lib/hex2nix/src/hex2nix.erl, line 103)
  in call from escript:run/2 (escript.erl, line 760)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1

Going to investigate that in hex2nix now.

Second edit: Investigation in a rebar3 shell in hex2nix reveals that apparently any connection to hex.pm fails:

5> ibrowse:send_req("https://hex.pm/api/packages/soap"
                   , [{"User-Agent", "hex2nix"}, {"Accept", "application/json"}]
                   , get
                   , []
                   , h2n_util:get_ibrowse_http_env()).

=ERROR REPORT==== 12-Jul-2018::22:51:02 ===
SSL: hello: ssl_alert.erl:88:Fatal error: internal error
{error,{conn_failed,{error,{tls_alert,"internal error"}}}}

Opening the example page in a browser works fine, so it's not an expired certificate or random other misfortune.

Third edit: Current status.

tazjin · 2018-07-12T21:08:46Z

Okay, for some reason this TLS connection does not succeed in Erlang 19, but it does in Erlang 20.

Does anyone know if there's a reason for the default Erlang in Nix still being at 19?

Edit: It may be time to resume tomorrow, by now my local rebar3 doesn't work anymore :)

Second edit: With some surgery to get a local nix-shell with erlangR20 and rebar3-open being used for buildRebar3 I got hex2nix working in a shell after pinning ibrowse to 4.2.2. This may also be related to cmullaparthi/ibrowse#160. The package list generation is now running ...

Third edit: FWIW, the actual issue that I ran into here was with ibrowse and I submitted a fix at cmullaparthi/ibrowse#163. Package generation is still running and will probably take quite a while with the very aggressive anti-ratelimit settings - I'll leave it running over night.

tazjin · 2018-07-14T11:55:32Z

Package regeneration has failed multiple times on various random issues and it seems to be difficult to resume where hex2nix stopped.

I'm going to see if I can fix the issues relevant for this PR manually, and then work on a package set update in a new PR. Does that sound reasonable, @gleber?

gleber · 2018-07-14T12:06:01Z

@tazjin This sounds good.

tazjin · 2018-07-14T12:44:22Z

The failing build was hex2nix itself and I've now worked around that by pinning its dependencies in the patchPhase. nox-review is now happy with this PR!

@xeji This isn't WIP anymore, can you remove the label?

gleber · 2018-07-14T16:29:46Z

pkgs/development/beam-modules/hex-registry-snapshot.nix

    src = fetchFromGitHub {
-        owner = "erlang-nix";
+        owner = "tazjin";


Please revert back to erlang-nix

Fixing in next push, good catch! :)

gleber · 2018-07-14T16:31:23Z

pkgs/development/tools/erlang/hex2nix/default.nix

@@ -30,6 +30,20 @@ buildRebar3 rec {

    enableDebugInfo = true;

+    # hex2nix does not specify version constraints, in which case


Maybe it's better to submit these changes directly to hex2nix repo? It seems like the right thing to do.

That does actually make sense, I'll prepare a change. Currently JSX is imported manually at version 2.8.2 in the derivation for hex2nix, do you know if there's a reason for it? The latest packaged version is 2.8.0 and I'd prefer to stick to that until the packages are regenerated.

Cursory test of hex2nix compiled with JSX 2.8.0 shows that it still works. The changelog for JSX contains nothing that looks relevant between 2.8.0 and 2.8.2.

Change submitted here: erlang-nix/hex2nix#22

Unpinned dependency versions can cause build issues in Nix if the package registry index snapshot contains a newer version than what is packaged in nixpkgs. See NixOS/nixpkgs#43430 for context. The versions chosen here are the most recent ones currently packaged in nixpkgs, with the exception of ibrowse which has been pinned to a version that is not affected by cmullaparthi/ibrowse#160.

tazjin · 2018-07-14T17:46:55Z

Alright, updated the hex2nix-derivation after the merge of erlang-nix/hex2nix#22 now. I've appended the commit short hash to the version, but if it's released as a version 0.0.7 or something like that I'll bump the version to that instead.

Updates rebar3 to version 3.6.1, which amongst other things introduces support for rebar3 on Erlang/OTP 21. Changes made: * rebar3 and dependencies updated to new versions * rebar3 hermeticity patch updated to apply against new version * hex package registry snapshot updated

Updates hex2nix to a new version that includes pinned dependencies. This allows building hex2nix even if newer versions of dependencies than what is packaged in `hex-packages.nix` are available in the package snapshot. The version of ibrowse used has been downgraded due to an issue in the library. Custom builds of the ibrowse and jsx dependencies have been removed from the derivation.

gleber

Thanks for pushing it forward!

xeji · 2018-07-14T18:50:48Z

is this ready to merge now?

tazjin · 2018-07-14T19:04:21Z

@xeji Yep. I ran nox-review again and everything is working fine.

xeji · 2018-07-14T19:05:47Z

Great. Let's just check the bumped packages then since it's standard procedure:
@GrahamcOfBorg build rebar3 hex2nix

GrahamcOfBorg · 2018-07-14T19:07:22Z

Success on x86_64-linux (full log)

Attempted: rebar3, hex2nix

Partial log (click to expand)

mkdir -p ""/nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7/bin""
cp "/tmp/nix-build-hex2nix-0.0.6-a31eadd7.drv-0/source/_build/default/bin/hex2nix" ""/nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7/bin""
chmod a+x ""/nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7/bin"/hex2nix"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7
patching script interpreter paths in /nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7
/nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7/bin/hex2nix: interpreter directive changed from "/usr/bin/env escript" to "/nix/store/wvmgi7wch5l1j0382hf55bdsx3916z9l-erlang-19.3.6.6/bin/escript"
checking for references to /tmp/nix-build-hex2nix-0.0.6-a31eadd7.drv-0 in /nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7...
/nix/store/b3y98sxs05vkkk7mz6vlqfw4i1pk4r43-rebar3-3.6.1
/nix/store/ri2ibnrpjsxcf3qm1gsdpahkvgkqk9ql-hex2nix-0.0.6-a31eadd7

GrahamcOfBorg · 2018-07-14T19:08:03Z

Success on x86_64-darwin (full log)

Attempted: rebar3, hex2nix

Partial log (click to expand)

===> Compiling hex2nix
===> Building escript...
mkdir -p ""/nix/store/sm34rmva5048fkp2qnkj9yif65s404qw-hex2nix-0.0.6-a31eadd7/bin""
cp "/private/tmp/nix-build-hex2nix-0.0.6-a31eadd7.drv-0/source/_build/default/bin/hex2nix" ""/nix/store/sm34rmva5048fkp2qnkj9yif65s404qw-hex2nix-0.0.6-a31eadd7/bin""
chmod a+x ""/nix/store/sm34rmva5048fkp2qnkj9yif65s404qw-hex2nix-0.0.6-a31eadd7/bin"/hex2nix"
post-installation fixup
patching script interpreter paths in /nix/store/sm34rmva5048fkp2qnkj9yif65s404qw-hex2nix-0.0.6-a31eadd7
/nix/store/sm34rmva5048fkp2qnkj9yif65s404qw-hex2nix-0.0.6-a31eadd7/bin/hex2nix: interpreter directive changed from "/usr/bin/env escript" to "/nix/store/3w57r99256aik5iv5isf6ssil7wdbvr6-erlang-19.3.6.6/bin/escript"
/nix/store/svgxisy3fjai0nrx4pqqpjan6bs5jsc4-rebar3-3.6.1
/nix/store/sm34rmva5048fkp2qnkj9yif65s404qw-hex2nix-0.0.6-a31eadd7

GrahamcOfBorg · 2018-07-14T19:09:14Z

Success on aarch64-linux (full log)

Attempted: rebar3, hex2nix

Partial log (click to expand)

mkdir -p ""/nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7/bin""
cp "/build/source/_build/default/bin/hex2nix" ""/nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7/bin""
chmod a+x ""/nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7/bin"/hex2nix"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7
patching script interpreter paths in /nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7
/nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7/bin/hex2nix: interpreter directive changed from "/usr/bin/env escript" to "/nix/store/7617y3r3zkwrsjq587gsh4bv0mq4q3rb-erlang-19.3.6.6/bin/escript"
checking for references to /build in /nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7...
/nix/store/dwwh0w2gildd9cwjfwqrk9ql8rf381p3-rebar3-3.6.1
/nix/store/zlrxsw40pv48r2ac77g4imrc169g6k15-hex2nix-0.0.6-a31eadd7

Erlang/OTP 19 is relatively old and is starting to cause issues with, amongst other things, TLS compatibility. Check out NixOS#43430 for some additional information.

GrahamcOfBorg added 6.topic: erlang 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jul 12, 2018

tazjin force-pushed the rebar3-3.6.1 branch from 2fa749c to 3926134 Compare July 12, 2018 20:31

xeji added the 2.status: work-in-progress label Jul 12, 2018

xeji removed the 2.status: work-in-progress label Jul 14, 2018

gleber suggested changes Jul 14, 2018

View reviewed changes

tazjin mentioned this pull request Jul 14, 2018

Pin dependency versions erlang-nix/hex2nix#22

Merged

tazjin force-pushed the rebar3-3.6.1 branch from 1c05272 to 8772616 Compare July 14, 2018 17:45

tazjin added 2 commits July 14, 2018 20:05

tazjin force-pushed the rebar3-3.6.1 branch from 8772616 to 74ca35b Compare July 14, 2018 18:06

gleber approved these changes Jul 14, 2018

View reviewed changes

xeji merged commit fbcb869 into NixOS:master Jul 14, 2018

tazjin deleted the rebar3-3.6.1 branch July 14, 2018 19:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rebar3: 3.4.3 -> 3.6.1 #43430

rebar3: 3.4.3 -> 3.6.1 #43430

tazjin commented Jul 12, 2018 •

edited

gleber commented Jul 12, 2018

tazjin commented Jul 12, 2018 •

edited

tazjin commented Jul 12, 2018 •

edited

tazjin commented Jul 14, 2018

gleber commented Jul 14, 2018

tazjin commented Jul 14, 2018 •

edited

gleber Jul 14, 2018

tazjin Jul 14, 2018

gleber Jul 14, 2018

tazjin Jul 14, 2018

tazjin Jul 14, 2018

tazjin Jul 14, 2018

tazjin commented Jul 14, 2018

gleber left a comment

xeji commented Jul 14, 2018

tazjin commented Jul 14, 2018

xeji commented Jul 14, 2018

GrahamcOfBorg commented Jul 14, 2018

GrahamcOfBorg commented Jul 14, 2018

GrahamcOfBorg commented Jul 14, 2018

		@@ -30,6 +30,20 @@ buildRebar3 rec {

		enableDebugInfo = true;

		# hex2nix does not specify version constraints, in which case

rebar3: 3.4.3 -> 3.6.1 #43430

rebar3: 3.4.3 -> 3.6.1 #43430

Conversation

tazjin commented Jul 12, 2018 • edited

Motivation for this change

Things done

gleber commented Jul 12, 2018

tazjin commented Jul 12, 2018 • edited

tazjin commented Jul 12, 2018 • edited

tazjin commented Jul 14, 2018

gleber commented Jul 14, 2018

tazjin commented Jul 14, 2018 • edited

gleber Jul 14, 2018

Choose a reason for hiding this comment

tazjin Jul 14, 2018

Choose a reason for hiding this comment

gleber Jul 14, 2018

Choose a reason for hiding this comment

tazjin Jul 14, 2018

Choose a reason for hiding this comment

tazjin Jul 14, 2018

Choose a reason for hiding this comment

tazjin Jul 14, 2018

Choose a reason for hiding this comment

tazjin commented Jul 14, 2018

gleber left a comment

Choose a reason for hiding this comment

xeji commented Jul 14, 2018

tazjin commented Jul 14, 2018

xeji commented Jul 14, 2018

GrahamcOfBorg commented Jul 14, 2018

GrahamcOfBorg commented Jul 14, 2018

GrahamcOfBorg commented Jul 14, 2018

tazjin commented Jul 12, 2018 •

edited

tazjin commented Jul 12, 2018 •

edited

tazjin commented Jul 12, 2018 •

edited

tazjin commented Jul 14, 2018 •

edited