buildRebar3 and buildMix depend less on hex registry #25655

elitak · 2017-05-09T19:28:07Z

Motivation for this change

I was trying to build one of my own projects using packages newer than those in the hex registry and found it was impossible because both Mix and rebar3 were trying to pull in the updated registry after failing to find the requested versions.

Mix has a new environment flag that fixes the behavior, so that change was easy.

Rebar3, on the other hand, still insists on updating the registry even when the built dependencies are present in ./_build/default/lib. The best solution I could come up with short of fixing this in upstream (erlang/rebar3#958) was to change the rebar3 bootstrap to have it generate all of what was symlinked said directory to ./_checkouts instead, sans src subdirs, since rebar3 will try to recompile the source (always, because the derivations lack .rebar3/erlcinfo metadata) into a RO store directory.

My changes to the rebar3 bootstrapping could use review and probably more testing, but as it stands, ./pkgs/development/beam-modules/hex-packages.nix is plenty broken. Does anyone know where is the script that generated this package listing? I need to make a fix there so that packages that depend on broken, elided packages are also elided. It could use a refresh, too.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-05-09T19:28:09Z

@elitak, thanks for your PR! By analyzing the history of the files in this pull request, we identified @ericbmerritt and @yurrriq to be potential reviewers.

yurrriq · 2017-05-09T20:48:21Z

Quick reply for now: https://github.com/erlang-nix/hex-pm-registry-snapshots

elitak · 2017-05-09T21:33:35Z

@yurrriq I don't see the script that emits hex-packages.nix, if that's what you meant to link...

ericbmerritt · 2017-05-09T21:36:20Z

The script that updates the registry is update.sh. The code that actually creates hex-packages.nix is here https://github.com/erlang-nix/hex2nix.

elitak · 2017-05-09T21:37:06Z

Ok thanks.

elitak · 2017-05-10T04:34:31Z

Hex2nix isnt building now because I somehow broke rebar3 escriptize with these changes. This is turning into a real mess; I'm getting pretty aggravated by rebar3 not being able to reliably use dependencies I'm laying at its feet. The lack of documentation, code clarity, command-line options, and debug logging in rebar3 is wearing my patience thin.

I might just abandon this and port all my projects to Mix, unless someone who's familiar with the rebar3+nix-bootstrap can offer a better solution than my moving _build/default/lib/* to _checkouts/*, because even if I get that to work, it feels like way too many hacks to accommodate what should be a more robust build tool.

ericbmerritt · 2017-05-10T17:00:08Z

@elitak rebar3 doesn't want to give up dependency management in any way shape or form. I have had conversations with the maintainers and they are not interested in fixing that problem. We had to do quite a number of hacky things to get it working. You are running into all that stuff. Mix was much simpler.

elitak · 2017-05-10T17:05:57Z

Alright, I'm just backing out the rebar3 changes for now, leaving the innocuous buildMix change.

matthewbauer · 2018-04-21T21:00:11Z

@GrahamcOfBorg build hexPackages.airbrakify

GrahamcOfBorg · 2018-04-21T21:00:27Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: hexPackages.airbrakify

Partial log (click to expand)

Cannot nix-instantiate `hexPackages.airbrakify' because:
�[31;1merror:�[0m attribute 'hexPackages' in selection path 'hexPackages.airbrakify' not found

GrahamcOfBorg · 2018-04-21T21:00:29Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: hexPackages.airbrakify

Partial log (click to expand)

Cannot nix-instantiate `hexPackages.airbrakify' because:
�[31;1merror:�[0m attribute 'hexPackages' in selection path 'hexPackages.airbrakify' not found

GrahamcOfBorg · 2018-04-21T21:01:07Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: hexPackages.airbrakify

Partial log (click to expand)

Cannot nix-instantiate `hexPackages.airbrakify' because:
�[31;1merror:�[0m attribute 'hexPackages' in selection path 'hexPackages.airbrakify' not found

matthewbauer · 2018-04-21T21:03:30Z

@GrahamcOfBorg build beam.packages.erlang.airbrakify

GrahamcOfBorg · 2018-04-21T21:04:08Z

Failure on x86_64-darwin (full log)

Attempted: beam.packages.erlang.airbrakify

Partial log (click to expand)


Unsafe variable found at:
  lib/httpoison.ex:66

Generated httpoison app
installing
post-installation fixup
patching script interpreter paths in /nix/store/jv4lb2i4rrpsair2swr227xcf7xh3dcx-httpoison-0.8.3
cannot build derivation '/nix/store/r4vs9i6m0f905wwma8xhxvn5cx5f1mmg-airbrakify-0.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/r4vs9i6m0f905wwma8xhxvn5cx5f1mmg-airbrakify-0.0.1.drv' failed

GrahamcOfBorg · 2018-04-21T21:04:32Z

Failure on x86_64-linux (full log)

Attempted: beam.packages.erlang.airbrakify

Partial log (click to expand)

  lib/httpoison.ex:66

Generated httpoison app
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/s8kflyx9v0hf2yhcilg4z5gwip84kjwb-httpoison-0.8.3
patching script interpreter paths in /nix/store/s8kflyx9v0hf2yhcilg4z5gwip84kjwb-httpoison-0.8.3
checking for references to /build in /nix/store/s8kflyx9v0hf2yhcilg4z5gwip84kjwb-httpoison-0.8.3...
cannot build derivation '/nix/store/cw6l4r6qhrbfgwwsz14h0ask513njmlm-airbrakify-0.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/cw6l4r6qhrbfgwwsz14h0ask513njmlm-airbrakify-0.0.1.drv' failed

matthewbauer

Looks good. Anyone have objections?

GrahamcOfBorg · 2018-04-21T21:09:04Z

Failure on aarch64-linux (full log)

Attempted: beam.packages.erlang.airbrakify

Partial log (click to expand)

  lib/httpoison.ex:66

Generated httpoison app
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/yghf8vy3zqlmz7yfx0gcvghr7g4b8bfl-httpoison-0.8.3
patching script interpreter paths in /nix/store/yghf8vy3zqlmz7yfx0gcvghr7g4b8bfl-httpoison-0.8.3
checking for references to /build in /nix/store/yghf8vy3zqlmz7yfx0gcvghr7g4b8bfl-httpoison-0.8.3...
cannot build derivation '/nix/store/pzbx3zjzb9pm0k8fjh96zayz6cbla7g4-airbrakify-0.0.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/pzbx3zjzb9pm0k8fjh96zayz6cbla7g4-airbrakify-0.0.1.drv' failed

veprbl · 2019-01-12T22:17:49Z

@GrahamcOfBorg eval

ryantm · 2019-02-23T23:24:05Z

@GrahamcOfBorg build beam.packages.erlang.airbrakify

yurrriq · 2019-02-25T16:21:58Z

Anyone have objections?

None from me.

yorickvP · 2019-07-02T11:06:45Z

Removing --no-deps-check here broke a lot of builds, since it now runs the deps check. Is running the deps check the intended effect?

cc: @yegortimoshenko

yorickvP · 2019-07-02T11:08:24Z

Example of a build that doesn't work anymore:

{
  jason = buildMix rec {
    name = "jason";
    version = "1.1.2";

    src = fetchzip {
      url = "https://github.com/michalmuskala/jason/archive/v${version}.tar.gz";
      sha256 = "0fh87vrfqsyiaazsangsg992i1azad8cmzyzvg7fdm9z6b3v7lm0";
    };
  };
}

lukateras · 2019-07-02T12:14:08Z

Can confirm, --no-deps-check should stay.

lukateras · 2019-07-02T12:25:55Z

@elitak rebar3 doesn't want to give up dependency management in any way shape or form. I have had conversations with the maintainers and they are not interested in fixing that problem. We had to do quite a number of hacky things to get it working. You are running into all that stuff. Mix was much simpler.

You can use fake-hex-registry, it was created specifically for that purpose. (I'm going to try to upstream mix-to-nix to Nixpkgs once it doesn't require eval-time builds and fetches).

elitak force-pushed the beam branch from 5980192 to 8089faf Compare May 10, 2017 17:03

LnL7 added the 6.topic: erlang label May 11, 2017

buildMix: properly ignore deps

a6e35f2

elitak force-pushed the beam branch from 8089faf to a6e35f2 Compare August 18, 2017 07:33

matthewbauer reviewed Apr 21, 2018

View reviewed changes

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jan 12, 2019

ryantm merged commit fcc2834 into NixOS:master Feb 25, 2019

yorickvP mentioned this pull request Jul 16, 2019

build-mix: re-add --no-deps-check flag #64884

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildRebar3 and buildMix depend less on hex registry #25655

buildRebar3 and buildMix depend less on hex registry #25655

elitak commented May 9, 2017

mention-bot commented May 9, 2017

yurrriq commented May 9, 2017

elitak commented May 9, 2017

ericbmerritt commented May 9, 2017

elitak commented May 9, 2017

elitak commented May 10, 2017

ericbmerritt commented May 10, 2017

elitak commented May 10, 2017

matthewbauer commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

matthewbauer commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

matthewbauer left a comment •

edited

GrahamcOfBorg commented Apr 21, 2018

veprbl commented Jan 12, 2019

ryantm commented Feb 23, 2019

yurrriq commented Feb 25, 2019 •

edited

yorickvP commented Jul 2, 2019 •

edited

yorickvP commented Jul 2, 2019

lukateras commented Jul 2, 2019

lukateras commented Jul 2, 2019

buildRebar3 and buildMix depend less on hex registry #25655

buildRebar3 and buildMix depend less on hex registry #25655

Conversation

elitak commented May 9, 2017

Motivation for this change

Things done

mention-bot commented May 9, 2017

yurrriq commented May 9, 2017

elitak commented May 9, 2017

ericbmerritt commented May 9, 2017

elitak commented May 9, 2017

elitak commented May 10, 2017

ericbmerritt commented May 10, 2017

elitak commented May 10, 2017

matthewbauer commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

matthewbauer commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

GrahamcOfBorg commented Apr 21, 2018

matthewbauer left a comment • edited

Choose a reason for hiding this comment

GrahamcOfBorg commented Apr 21, 2018

veprbl commented Jan 12, 2019

ryantm commented Feb 23, 2019

yurrriq commented Feb 25, 2019 • edited

yorickvP commented Jul 2, 2019 • edited

yorickvP commented Jul 2, 2019

lukateras commented Jul 2, 2019

lukateras commented Jul 2, 2019

matthewbauer left a comment •

edited

yurrriq commented Feb 25, 2019 •

edited

yorickvP commented Jul 2, 2019 •

edited