Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

buildLazyBinaries: init build support helper #71951

Closed
wants to merge 1 commit into from
Closed

buildLazyBinaries: init build support helper #71951

wants to merge 1 commit into from

Conversation

bendlas
Copy link
Contributor

@bendlas bendlas commented Oct 24, 2019

This helps create shell binaries, which build their corresponding
packages, only when run.

By default, it uses package names identical to command
names. Binaries, contained by packages with different name, can be
listed in a catalog.

Motivation for this change

I often want to update my system, but I also want to have access to many packages on my system path. This creates a tension: installing many packages means lots of download volume and / or build time (especially if you're on master).

Considering the pervasive laziness of nix, wouldn't it be nice, if there was a sort of lazy nix-shell, that would build packages only at the last second, before calling into binaries from them?

Things done

This is a proof of concept, mainly for personal use and to gauge interest.
It might already be a workable prototype, please have a look at the code comments, to get a sense of the direction, I can see this going.

I'm open to suggestions for the interface, especially on collision-detection/prioritization/runtime-dispatch and renames.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

not sure who to cc this to, in particular. I need feedback about having the catalog built on hydra and other integrations. I think, right now, this should already be of interest to everybody doing nixos development, but a nixos module for it could also benefit infrequent users of heavy packages, replacing nix-shell -p.

@bendlas
buildLazyBinaries: init build support helper
9c00937 
This helps create shell binaries, which build their corresponding
packages, only when run.

By default, it uses package names identical to command
names. Binaries, contained by packages with different name, can be
listed in a catalog.
@matthewbauer
Copy link
Member

I like the idea! I had a similar project with lazy binding things a while back:

https://github.com/matthewbauer/omnix/blob/master/omnix.sh

I'm not sure how useful it is on its own though. Most commands and apps names are not going to be known ahead of time. You just want them lazily available. Something like nix-index or the command-not-found db would be necessary to get this working for all commands.

@bendlas
Copy link
Contributor Author

bendlas commented Nov 3, 2019 •
edited

I like the idea! I had a similar project with lazy binding things a while back:

looks awesome! would you recommend any idea or approach in particular, from your experience with that?

Something like nix-index or the command-not-found db would be necessary to get this working for all commands.

Oh wow, I didn't know, that the cache already offered *.ls urls (or nix-index for that matter). Are those built by hydra, or on demand?

To be fair, even nix-index will only work for packages built by hydra either. But many more packages will hopefully be caught by the command = package default, and it might nudge people towards naming packages that way, as well.

Before pushing towards getting a catalog built, though, I'd like to focus on user experience:

  • symlinking packages into ~/.local/share/nix-lazy-binaries, to protect them from GC
  • Something like
$ foo-cmd
Building `foo-cmd-1.3.3.7 (eb3fbc)`
|=====>                        12.5%                                                   |
A previous version of `foo-cmd-1.3.3.7 (f35a43)` is available.
To continue with that version (while downloading the update in the background),
press `Y`

@bendlas bendlas requested a review from bennofs November 3, 2019 18:46
@bennofs
Copy link
Contributor

bennofs commented Nov 3, 2019

.ls files are built by hydra as part of building the derivation.

@bendlas
Copy link
Contributor Author

bendlas commented Apr 10, 2020

I'm closing this PR, since I don't want to work on it any more.

The main problem with this is, that I want to protect lazy binaries from gc, until they have been updated. At this point, I'm basically reimplementing nix-env, and I think I'm better served by automating that.

Therefore I'm dropping this patch from my queue.

@bendlas bendlas closed this Apr 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bennofs bennofs Awaiting requested review from bennofs

Assignees
No one assigned
Labels
0.kind: enhancement 2.status: work-in-progress 6.topic: developer experience 6.topic: user experience 8.has: package (new) 9.needs: changelog 9.needs: community feedback 9.needs: documentation 9.needs: module (new) 10.rebuild-darwin: 0 10.rebuild-linux: 0 11.by: nixpkgs-member
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@bendlas @matthewbauer @bennofs