linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a list of derivations #70154

cdepillabout · 2019-10-01T06:49:43Z

Motivation for this change

This PR adds a trivial builder called linkFarmFromDrvs. It is similar to linkFarm, but it is a easier to use if you just want directory with a bunch of links to derivations, but you're fine with links just being the name of the derivation.

We use this in a couple places in our nix code-base at work, and we thought it could be helpful to other nix users.

Here's an example of using it:

$ nix-build -E 'with (import ./. {}); linkFarmFromDrvs "example" [ pkgs.hello pkgs.gcc pkgs.xterm ]'
...
/nix/store/hsi98apxc80l8r3lpi13hzc3lk8i8623-example
$ ls -l result/
lrwxrwxrwx 1 root root 61 Jan  1  1970 gcc-wrapper-8.3.0 -> /nix/store/7hzmz83nv8khpwsk858yaac7y3idh0a7-gcc-wrapper-8.3.0
lrwxrwxrwx 1 root root 54 Jan  1  1970 hello-2.10 -> /nix/store/rr3y0c6zyk7kjjl8y19s4lsrhn4aiq1z-hello-2.10
lrwxrwxrwx 1 root root 53 Jan  1  1970 xterm-348 -> /nix/store/y0cz7w39v4sb743mix5qps3kc2ra9b72-xterm-348

Things done

Notify maintainers

cc @

edolstra · 2019-10-01T07:29:04Z

What is a "link farm"?

cdepillabout · 2019-10-01T07:42:13Z

@edolstra Thanks for taking a look at this.

linkFarm is the name of one of the trivial builders currently in nixpkgs:

nixpkgs/pkgs/build-support/trivial-builders.nix

Lines 294 to 312 in 6b11080

    
             /* 
        
              * Quickly create a set of symlinks to derivations. 
        
              * entries is a list of attribute sets like 
        
              * { name = "name" ; path = "/nix/store/..."; } 
        
              * 
        
              * Example: 
        
              * 
        
              * # Symlinks hello path in store to current $out/hello 
        
              * linkFarm "hello" [ { name = "hello"; path = pkgs.hello; } ]; 
        
              * 
        
              */ 
        
             linkFarm = name: entries: runCommand name { preferLocalBuild = true; allowSubstitutes = false; } 
        
               ''mkdir -p $out 
        
                 cd $out 
        
                 ${lib.concatMapStrings (x: '' 
        
                     mkdir -p "$(dirname ${lib.escapeShellArg x.name})" 
        
                     ln -s ${lib.escapeShellArg x.path} ${lib.escapeShellArg x.name} 
        
                 '') entries} 
        
               '';

I'm using linkFarm in the title and description of this PR to describe the output of a derivation created with the linkFarm function. Although I agree that this is not a widely-used term and I'd be happy to change it if it is confusing.

This PR adds an additional helper function called linkFarmFromDrvs that wraps linkFarm and makes it a little easier to use in the common case.

Profpatsch · 2020-01-27T18:08:02Z

What’s the difference to symlinkJoin "name" { paths = [ drv1 drv2 … ]; }?

cdepillabout · 2020-01-28T01:19:37Z

@Profpatsch linkFarmFromDrvs (and the similar linkFarm function), just create a bunch of symlinks in a directory:

$ nix-build -E 'with import ./. {}; linkFarmFromDrvs "example" [ pkgs.hello pkgs.gcc pkgs.xterm ]'
...
/nix/store/hsi98apxc80l8r3lpi13hzc3lk8i8623-example
$ tree /nix/store/hsi98apxc80l8r3lpi13hzc3lk8i8623-example
/nix/store/hsi98apxc80l8r3lpi13hzc3lk8i8623-example
├── gcc-wrapper-8.3.0 -> /nix/store/7hzmz83nv8khpwsk858yaac7y3idh0a7-gcc-wrapper-8.3.0
├── hello-2.10 -> /nix/store/rr3y0c6zyk7kjjl8y19s4lsrhn4aiq1z-hello-2.10
└── xterm-348 -> /nix/store/y0cz7w39v4sb743mix5qps3kc2ra9b72-xterm-348

You can see that the directory /nix/store/hsi98apxc80l8r3lpi13hzc3lk8i8623-example just contains a bunch of symlinks to other derivations.

However, symlinkJoin actually replicates the directory structure from the original derivations:

$ nix-build -E 'with import <nixpkgs> {}; symlinkJoin { name = "example"; paths = [ pkgs.hello pkgs.stack ]; }'
/nix/store/wh0nkyg2qklrlq74z17fpy6yqfbdxc4m-example
$ tree /nix/store/wh0nkyg2qklrlq74z17fpy6yqfbdxc4m-example
/nix/store/wh0nkyg2qklrlq74z17fpy6yqfbdxc4m-example
├── bin
│   ├── hello -> /nix/store/qy93dp4a3rqyn2mz63fbxjg228hffwyw-hello-2.10/bin/hello
│   └── stack -> /nix/store/6lzdpxshx78281vy056lbk553ijsdr44-stack-2.1.3.1/bin/stack
└── share
    ├── bash-completion
    │   └── completions
    │       └── stack -> /nix/store/6lzdpxshx78281vy056lbk553ijsdr44-stack-2.1.3.1/share/bash-completion/completions/stack
    ├── fish
    │   └── vendor_completions.d
    │       └── stack.fish -> /nix/store/6lzdpxshx78281vy056lbk553ijsdr44-stack-2.1.3.1/share/fish/vendor_completions.d/stack.fish
    ├── info
    │   └── hello.info -> /nix/store/qy93dp4a3rqyn2mz63fbxjg228hffwyw-hello-2.10/share/info/hello.info
...

You can see here that symlinkJoin tries to replicate the directory structure from the input derivations, while linkFarmFromDrvs just creates a link directory with symlinks.

We use linkFarmFromDrvs a bunch of places in our code base at work. Normally in CI when we want to test that a bunch of loosely-related derivations (normally tests) are able to be built, but we rarely access the actual output of the derivations.

Profpatsch · 2020-01-28T16:14:46Z

Then +1 to that change, but let’s move these three functions together and have an overview of their differences.

…ist of derivations

cdepillabout · 2020-02-14T04:52:35Z

@Profpatsch Reorganizing the functions is a good idea.

I just added a new commit that reorders the functions in this file, adds an example output for symlinkJoin and linkFarm, and adds some documentation that explains the difference between symlinkJoin and linkFarm.

I think this should be ready for a final review and hopefully a merge-in.

nixos-discourse · 2020-02-16T01:55:37Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review-may-2019/3032/117

Profpatsch

Fantastic! LGTM

nixos-discourse · 2020-03-19T15:26:46Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/comment-doc-praise-thread/6325/2

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 1, 2019

linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a l…

6f52dd4

…ist of derivations

cdepillabout force-pushed the add-linkFarmFromDrvs branch from 348084a to a4c1c05 Compare February 14, 2020 04:47

symlinkJoin: add more documentation

678a01a

cdepillabout force-pushed the add-linkFarmFromDrvs branch from a4c1c05 to 678a01a Compare February 14, 2020 07:09

Profpatsch reviewed Feb 18, 2020

View reviewed changes

Profpatsch merged commit 55e1529 into NixOS:master Feb 18, 2020

cdepillabout deleted the add-linkFarmFromDrvs branch February 19, 2020 05:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a list of derivations #70154

linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a list of derivations #70154

cdepillabout commented Oct 1, 2019

edolstra commented Oct 1, 2019

cdepillabout commented Oct 1, 2019

Profpatsch commented Jan 27, 2020

cdepillabout commented Jan 28, 2020

Profpatsch commented Jan 28, 2020

cdepillabout commented Feb 14, 2020

nixos-discourse commented Feb 16, 2020

Profpatsch left a comment

nixos-discourse commented Mar 19, 2020

linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a list of derivations #70154

linkFarmFromDrvs: Add a trivial builder to create a linkFarm from a list of derivations #70154

Conversation

cdepillabout commented Oct 1, 2019

Motivation for this change

Things done

Notify maintainers

edolstra commented Oct 1, 2019

cdepillabout commented Oct 1, 2019

Profpatsch commented Jan 27, 2020

cdepillabout commented Jan 28, 2020

Profpatsch commented Jan 28, 2020

cdepillabout commented Feb 14, 2020

nixos-discourse commented Feb 16, 2020

Profpatsch left a comment

Choose a reason for hiding this comment

nixos-discourse commented Mar 19, 2020