add a filter function argument to builtins.fetchGit #3171
Conversation
When working with monorepos containing projects in sub directories each sub project typically has a derivation where the `src` attribute is set to the path of the sub directory. That setup is not ideal since all files below that sub directory, including build artefacts, temporary files or even secrets, will be copied to the store. Ideally only files added to the git repository should be copied to the store to ensure local nix builds are reproducible on CI. Note that `builtins.fetchGit` can be used to copy a local git repository to the store. This commit extends `builtins.fetchGit` with a filter function argument which can be used to only include a sub directory. The filter function has the same type as expected by `builtins.filterSource` and has the same semantics except that the path is relative to the git checkout.
|
I marked this as stale due to inactivity. → More info |
|
I closed this issue due to inactivity. → More info |
|
@Ericson2314 I see you've reopened this PR, but we should consider the other implementation strategy, which is to compose a filtering lazy tree onto the git lazy tree. This solution has better composability and separation of concerns, with the only "downside" being that lazy trees are still in development. However, I am confident that lazy trees are a good idea, so considering the scarcity of maintainer time, I think we should work towards a better lazy trees implementation. |
fricklerhandwerk
added
feature
|
Notes from the maintainers team meeting
Otherwise, we'd rather go the lazy-trees (#6530) route as it would provide a more generic solution to this problem. |
I recently worked on a very compositional API that's easy and safe to use for this purpose! It's implemented as a draft PR to Nixpkgs and the interface needs some feedback to figure out whether it satisfies all necessary use cases. Please take a look, try it out and give feedback! Here's a Discourse post giving a brief tour 😃 https://discourse.nixos.org/t/easy-source-filtering-with-file-sets/29117 |
Not if you're looking for a fetchGit that doesn't but certain files in the store ever. With the Nix language, you can currently only achieve this by either manually parsing the files of a local repository, or by fetching through a derivation (IFD / Read From Realisation (is that what we're going for now?)). Anyway, most repositories don't have a performance or confidentiality problem, so do check out the link to learn how to filter better and help validate the design! Let's go |
|
This PR is very much out of date, and the file set library already solves part of the issue, therefore closing it. Current state of discussion on the other part, lazy fetching: #2944 (comment) |
Fixes: #2944.
When working with monorepos containing projects in sub directories
each sub project typically has a derivation where the
srcattributeis set to the path of the sub directory.
That setup is not ideal since all files below that sub directory,
including build artefacts, temporary files or even secrets, will be
copied to the store. Ideally only files added to the git repository
should be copied to the store to ensure local nix builds are
reproducible on CI.
Note that
builtins.fetchGitcan be used to copy a local gitrepository to the store. This commit extends
builtins.fetchGitwitha filter function argument which can be used to only include a sub
directory.
The filter function has the same type as expected by
builtins.filterSourceand has the same semantics except that thepath is relative to the git checkout.
@edolstra at NixCon you mentioned we should possibly think about a more compositional API since most builtin fetch functions now take a filter argument. However this addition provides in a common need and prevents users from IFD. Let me know what you think.