resholve: init at 0.4.0 #85827

abathur · 2020-04-23T00:09:01Z

Motivation for this change

Give the Nix ecosystem a new superpower: resolving external dependencies in shell scripts.

Shell scripts/libraries with external dependencies are an odd duck in the Nix ecosystem. Packaging them with Nix doesn't ensure dependencies are available at runtime or address the possibility that the script could run executables provided by packages other than those specified as dependencies, for example.

More information

This PR packages a project of mine, https://github.com/abathur/resholve, and provides a Nix build function. resholve is built atop the parser from the Oil shell project. A fair portion of the Nix code in this PR is just packaging/building a development version of Oil.

The PR itself isn't terribly useful for seeing what it does; it'll help to look at some additional resources:

bashup-events: init at it's-complicated #107182 shows how I've used it to package a new shell library.
This commit shows (roughly, I'm synthesizing this) how you'd resholve the existing shunit2, and how it would change the script that ends up in the store.
The repo includes two demos (shell API, Nix API), which you can read about (including invocation instructions) at https://github.com/abathur/resholve/blob/master/demos.md.

FWIW, I dogfood resholve in my own darwin-configuration, where it is responsible for building a chain of packages structured like:

my bashrc
- a bash history-management module I'm developing
  - an event-based shell-profile "orchestration" framework I'm developing to support both of the above
    - a third-party bash event/promise library (now PRed in bashup-events: init at it's-complicated #107182)

A question came up during review about whether some of the packages in `deps.nix` should be split out into standalone packages. I've resolved the thread, but others may have the same question so I'm keeping the answer somewhere easier to find:

BTW, having oilshell and maybe oildev available in all-packages is probably a good idea, since those packages are kinda interesting. You could, for example, open a PR with only oildshell, oildev and maybe py-yajl, making it easier to review only those dependencies. Afterwards you could make this PR depends on those dependencies.

I have reservations about this.

I'm happy to break these out if there's a clear imperative, but they are all pretty idiomatic (and all ultimately subservient to the oildev package itself). I don't anticipate them adding standalone value to nixpkgs in the short run.

The Oil shell already has a binary user package included here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/shells/oil/default.nix

oildev isn't (for now) a real thing that the Oil project intentionally publishes--it's a fiction created by forcibly patching-and-packaging Oil's python code in order to build resholve

Because these Python APIs aren't intentionally published for external consumption and Oil is still a fast-changing young project, they aren't stable. I'd be very surprised if more than one consumer could share a single version of it for long.

AFAIK resholve is the only consumer of this. No one's asked me about using it.

resholve still needs ~140 lines of code just to monkeypatch all of oil's packages into a unified oil.* namespace. If anyone used it without this shim, all of its subpackages would leak into the root Python package namespace.

I can't guarantee any Oil code paths that resholve doesn't exercise actually work.

Because resholve is very tightly coupled to Oil's internal Python APIs, there's a fair chance any update of this would break resholve.

py-yajl is Oil's fork of https://github.com/rtyler/py-yajl. They've modified it, dropped tests they don't need, and cut code they don't use. I have no idea how suitable it is or isn't for general use. Oil has made no attempt to promote their py-yajl fork, it has no stars, and I have zero reason to expect that Oil would support it in any way.

cc @grahamc @worldofpeace

Known issues

The current design of the Nix API assumes all of the shell files can be resolved with the same settings, and it won't be terribly obvious what to do (aside from break it up into multiple derivations, or override to manually invoke resholve) in these cases.

I'm not sure what the best approach here is. For reference, here's one possible approach from @grahamc: https://gsc.io/content-addressed/92d2781347e6e0d9b761ff2d4512d6885f48d8fb6a97db15f0ba5f4863ec91bd.nix

I have pushed a ~~tentative~~ fix for this; see #85827 (comment) and d36bb7d

Things done

abathur · 2020-05-17T22:26:40Z

I marked this as draft for a few days while I refactored how it packages and consumes its Oil-shell dependency. Not super relevant, but the main changes are:

It previously built from a fork of Oil where I fixed a few issues and added a setup.py, but I collapsed these changes into a set of patches maintained in the resholved repo. It now builds against a commit in the official repo.
Oil isn't structured with being turned into a Python package in mind, so I also hacked through forcing an oil namespace on its subpackages, which were previously just leaking into the Python package namespace. Some of them have very generic names (like core, misc, frontend, tools, etc.) so I've been a little concerned about them causing namespace clashes. To make this work (without patching all of the oil source) resholved now rewrites oil's imports.

pkgs/development/misc/resholved/mk-resholved-derivation.nix

abathur · 2020-12-15T05:01:55Z

@grahamc I have finally released version ~~0.1.0~~ 0.1.1 of resholve, updated this PR accordingly, and marked it ready for review.

resholve attempts to resolve executables in shell scripts. Includes Nix builder for resolving dependencies in Nix-built shell projects.

abathur · 2020-12-15T15:22:04Z

Edit: #84062 has been merged, so my maintainer entry is now in master.

I rewrote the Nix API to support multiple resholve invocations with different settings.

abathur · 2020-12-18T02:42:02Z

I've pushed an update which reworks the Nix API so that it can specify more than one resholve invocation.

Here's a (very abbreviated) example of how I was handling a package with one sourceable profile script, and one executable utility script:

resholve.resholvePackage rec {
  # ...
  scripts = [ "bin/hag.bash" "bin/hag_import_history.bash" ];
  interpreter = "${bashInteractive_5}/bin/bash";
  # ...
}

Here's how it looks with the API update:

resholve.resholvePackage rec {
  # ...
  solutions = {
    profile = {
      scripts = [ "bin/hag.bash" ];
      interpreter = "none";
    };
    # profile/cmd are arbitrary names; could be a list
    # but I gather set is preferred for overridability
    cmd = {
      scripts = [ "bin/hag_import_history.bash" ];
      interpreter = "${bashInteractive_5}/bin/bash";
    };
  };
  #...
}

nixos-discourse · 2020-12-24T05:24:35Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/412

thiagokokada

Code wise LGTM, I can also build with with nixpkgs-review but it conflicts with nixpkgs PATH so I can't post the result.

Anyway, approving, but I didn't test it very much (did run resholve --help and it works). Probably someone with more context of how the package is supposed to work can check the functionality.

Don't forget to squash the commits before merging.

pkgs/development/misc/resholve/resholve-package.nix

abathur · 2021-01-03T04:16:32Z

@SuperSandro2000 I haven't had any luck fishing for feedback on how to document this in the short run, so I pushed a commit updating the doc/comments where I previously had TODOs.

That's the last thing I wanted to do here, so I think it is ready for a squash-and-merge (if you think they're okay?)

SuperSandro2000

Package should be fine after my suggestions. I am not to sure if the build support could be made easier or we have inbuild functions for some of the ones but it should be fine for now.

pkgs/development/misc/resholve/deps.nix

pkgs/development/misc/resholve/default.nix

pkgs/development/misc/resholve/resholve.nix

pkgs/development/misc/resholve/resholve-package.nix

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

abathur · 2021-01-04T06:53:20Z

@SuperSandro2000 I've pushed commits that:

mostly address the review comments here
add a README.md as informal documentation, and a step towards eventually documenting it in the nixpkgs manual
bump/release resholve 0.4.0 to track the breaking namespace change (the Nix API is included there as well; no other functionality change over the 0.3.0)

SuperSandro2000 · 2021-01-05T01:29:55Z

Didn't read the entire doc but if we find mistake/areas to improve we can always do that later.

grahamc

I am exceedingly excited to use this!

worldofpeace · 2021-01-05T17:02:58Z

It's a miracle ✨ @abathur I'm excited that you got this into nixpkgs ❤️

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Apr 23, 2020

abathur force-pushed the add_resholved branch 2 times, most recently from e29da15 to b6b79b1 Compare April 23, 2020 00:36

abathur changed the title ~~resholved: init at hopes and dreams~~ [WIP] resholved: init at hopes and dreams May 10, 2020

abathur marked this pull request as draft May 10, 2020 15:28

abathur changed the title ~~[WIP] resholved: init at hopes and dreams~~ resholved: init at hopes and dreams May 10, 2020

abathur force-pushed the add_resholved branch 2 times, most recently from 55c6945 to 71d5ae4 Compare May 17, 2020 19:41

abathur marked this pull request as ready for review May 17, 2020 19:53

abathur force-pushed the add_resholved branch from 71d5ae4 to 54c53ba Compare May 22, 2020 01:27

abathur commented May 22, 2020

View reviewed changes

pkgs/development/misc/resholved/mk-resholved-derivation.nix Outdated Show resolved Hide resolved

abathur force-pushed the add_resholved branch from 54c53ba to 83f77de Compare June 28, 2020 19:32

SuperSandro2000 marked this pull request as draft November 29, 2020 00:35

abathur force-pushed the add_resholved branch from 83f77de to 05cb20e Compare December 15, 2020 04:18

abathur marked this pull request as ready for review December 15, 2020 04:59

abathur changed the title ~~resholved: init at hopes and dreams~~ resholve: init at 0.1.0 Dec 15, 2020

resholve: init

6937d9e

resholve attempts to resolve executables in shell scripts. Includes Nix builder for resolving dependencies in Nix-built shell projects.

abathur force-pushed the add_resholved branch from 05cb20e to 6937d9e Compare December 15, 2020 06:23

abathur changed the title ~~resholve: init at 0.1.0~~ resholve: init at 0.1.1 Dec 15, 2020

resholve: 0.1.1 -> 0.2.0-1

d36bb7d

I rewrote the Nix API to support multiple resholve invocations with different settings.

abathur mentioned this pull request Dec 19, 2020

bashup-events: init at it's-complicated #107182

Merged

10 tasks

abathur added 2 commits December 27, 2020 10:34

resholve: 0.2.0-1 -> 0.2.0

6bbf251

resholve: 0.2.0 -> 0.2.1

e69d508

abathur changed the title ~~resholve: init at 0.1.1~~ resholve: init at 0.2.1 Dec 27, 2020

ofborg bot added 10.rebuild-linux: 1 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jan 1, 2021

thiagokokada approved these changes Jan 1, 2021

View reviewed changes

doc/comment on functions

0e150a0

abathur commented Jan 3, 2021

View reviewed changes

pkgs/development/misc/resholve/resholve-package.nix Outdated Show resolved Hide resolved

abathur requested a review from SuperSandro2000 January 3, 2021 04:12

SuperSandro2000 reviewed Jan 4, 2021

View reviewed changes

abathur and others added 12 commits January 3, 2021 19:20

cut commented lines

9eb3b8a

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

revert previous review addition

cf482d3

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

adopt review suggestion

772dacf

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

revert previous review suggestion

b741728

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

brevity is the soul of wit

0228da3

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

de-materialize resolveTimeDeps

cb37f5b

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

drop unnecessary dontStrip

1321e2a

move makeWrapper to nativeBuildInputs

714c0b9

explain commented loglevel

b934618

fixup issues with review suggestions

b590d7a

flatten namespace

e9868c1

add resholve README

30b58c9

abathur force-pushed the add_resholved branch from d7c0779 to 30b58c9 Compare January 4, 2021 06:07

resholve: 0.3.0 -> 0.4.0

acdd2ce

abathur changed the title ~~resholve: init at 0.3.0~~ resholve: init at 0.4.0 Jan 4, 2021

SuperSandro2000 approved these changes Jan 5, 2021

View reviewed changes

grahamc approved these changes Jan 5, 2021

View reviewed changes

grahamc merged commit 6fd9283 into NixOS:master Jan 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

resholve: init at 0.4.0 #85827

resholve: init at 0.4.0 #85827

abathur commented Apr 23, 2020 •

edited

abathur commented May 17, 2020 •

edited

abathur commented Dec 15, 2020 •

edited

abathur commented Dec 15, 2020 •

edited

abathur commented Dec 18, 2020 •

edited

nixos-discourse commented Dec 24, 2020

thiagokokada left a comment •

edited

abathur commented Jan 3, 2021 •

edited

SuperSandro2000 left a comment

abathur commented Jan 4, 2021

SuperSandro2000 commented Jan 5, 2021

grahamc left a comment

worldofpeace commented Jan 5, 2021

resholve: init at 0.4.0 #85827

resholve: init at 0.4.0 #85827

Conversation

abathur commented Apr 23, 2020 • edited

Motivation for this change

More information

Known issues

Things done

abathur commented May 17, 2020 • edited

abathur commented Dec 15, 2020 • edited

abathur commented Dec 15, 2020 • edited

abathur commented Dec 18, 2020 • edited

nixos-discourse commented Dec 24, 2020

thiagokokada left a comment • edited

Choose a reason for hiding this comment

abathur commented Jan 3, 2021 • edited

SuperSandro2000 left a comment

Choose a reason for hiding this comment

abathur commented Jan 4, 2021

SuperSandro2000 commented Jan 5, 2021

grahamc left a comment

Choose a reason for hiding this comment

worldofpeace commented Jan 5, 2021

abathur commented Apr 23, 2020 •

edited

abathur commented May 17, 2020 •

edited

abathur commented Dec 15, 2020 •

edited

abathur commented Dec 15, 2020 •

edited

abathur commented Dec 18, 2020 •

edited

thiagokokada left a comment •

edited

abathur commented Jan 3, 2021 •

edited