log all ifd #3494

cleverca22 · 2020-04-14T18:51:50Z

No description provided.

grahamc · 2020-04-14T18:57:55Z

src/libexpr/primops.cc

@@ -56,6 +56,9 @@ void EvalState::realiseContext(const PathSet & context)
            throw InvalidPathError(store->printStorePath(ctx));
        if (!decoded.second.empty() && ctx.isDerivation()) {
            drvs.push_back(StorePathWithOutputs{ctx.clone(), {decoded.second}});
+            if (evalSettings.logAllIFD) {
+              printInfo(format("%1% depends on %2% via %3%") % pos % decoded.first % reason);


maybe something a bit more explicit, mentioning "importing from derivation"?

Also in the help text please.

src/libexpr/eval.hh

grahamc · 2020-04-14T19:05:08Z

One last nit, and looks great to me

doc/manual/command-ref/conf-file.xml

edolstra · 2020-04-14T20:05:51Z

What's the use case for this? Why does it need a separate option rather than a debug(...) message?

grahamc · 2020-04-14T20:25:51Z

debug sounds like a fine option, the main thing is being able to easily see the origin of the IFD

cleverca22 · 2020-04-15T01:37:42Z

if it was done via debug(...) then you would need enough -vvvv (which can harm performance?) and know the magic string to grep for, and grep on stderr is more complex

having a dedicated option eliminates the need to know the magic string to search for

i can also see value in just turning this option on globally, so you have more feedback about why something just started a build when it was doing an eval

src/libexpr/eval.hh

src/libexpr/primops.cc

Ericson2314 · 2020-04-15T15:22:02Z

What's the use case for this?

I just discovered this. this would be very useful to us making sure we have all build-time and eval-time deps, when trying to export caches for clients. Definitely do want!

matthewbauer · 2020-04-16T19:55:13Z

Debugging this stuff seems useful, but I wonder if we could have a flag that gives a machine readable output? A common reason to want to know about all of these .drv files is so that you can keep a cache, not just of the dependencies of your final drv, but also of the dependencies of the imported drv.

For this, I would propose having an option to nix-instantiate like this:

$ nix-instantiate --include-ifd my-imported-drv.nix
/nix/store/00000000000000000000000000000000-my-drv.drv
/nix/store/00000000000000000000000000000000-my-imported-drv.drv

This way you could pipe it to something like cachix with:

$ nix-store -qR --include-outputs $(nix-instantiate --include-ifd my-imported-drv.nix) | cachix push my-cache

and have all dependencies cached. If this sounds like a good idea, I'd be happy to take a stab at implementing it.

/cc @domenkozar

domenkozar · 2020-04-16T20:22:57Z

Hell yes, that would really help with syncing to binary caches!

disassembler · 2020-05-22T19:33:47Z

@edolstra what's needed to get this done? It seems like a machine readable option is a bonus unrelated to just getting the logging. Do you think this could be merged as-is and @matthewbauer could put his new functionality in a follow-up?

deepfire · 2021-01-13T16:57:20Z

@grahamc, what do you think needs to be done (besides a rebase)?

bqv · 2021-02-04T11:26:31Z

Tumbleweed?

src/libexpr/primops.cc

stale · 2021-10-30T17:19:06Z

I marked this as stale due to inactivity. → More info

cleverca22 · 2021-11-01T04:17:53Z

not stale, i do intend to finish this at some point

deepfire · 2022-02-03T20:22:39Z

This works -- for example, in recent cardano-node:

[user@machine:~/cardano-node]$ nix flake show --option log-all-ifd true --option allow-import-from-derivation true -v
warning: unknown experimental feature 'ca-references'
warning: Git tree '/home/user/cardano-node' is dirty
evaluating ''...
git+file:///home/user/cardano-node
evaluating 'apps'...
├───apps
evaluating 'apps.x86_64-darwin'...
│   ├───x86_64-darwin
trace: WARNING: 8.10.5 is out of date, consider using 8.10.7.
trace: haskell-nix.haskellLib.cleanGit: /nix/store/q9lj3lzf4khxhl2w0mzhni2w190gv47w-source does not seem to be a git repository,
assuming it is a clean checkout.
trace: To make project.plan-nix for cardano-node a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
trace: To materialize project.plan-nix for cardano-node entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
trace: WARNING: 8.10.5 is out of date, consider using 8.10.7.
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/import-and-filter-project.nix:21:13 importing from derivation /nix/store/vnkwjg5aq1jkqy3cfjf6xlmz1ba8fp32-nix-tools-plan-to-nix-pkgs.drv via scopedImport
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/spdx/licenses.nix:6:8 importing from derivation /nix/store/dscrvrxsk3sc6q86vix26sdd4k7w3l09-spdx-license-list-data-3.12.drv via readFile
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/import-and-filter-project.nix:21:13 importing from derivation /nix/store/l6zrcg2d26n7gjyzsz5dqlp6xhdrl0ji-alex-plan-to-nix-pkgs.drv via scopedImport
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/spdx/licenses.nix:6:8 importing from derivation /nix/store/dscrvrxsk3sc6q86vix26sdd4k7w3l09-spdx-license-list-data-3.12.drv via readFile
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/import-and-filter-project.nix:21:13 importing from derivation /nix/store/2k63z16v1kphs60w763i9n4326sl34n9-happy-plan-to-nix-pkgs.drv via scopedImport
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/spdx/licenses.nix:6:8 importing from derivation /nix/store/dscrvrxsk3sc6q86vix26sdd4k7w3l09-spdx-license-list-data-3.12.drv via readFile
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/import-and-filter-project.nix:21:13 importing from derivation /nix/store/5dm6ijxsv3b5mga4bjqx2vzgdv4k2sfa-hscolour-plan-to-nix-pkgs.drv via scopedImport
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/spdx/licenses.nix:6:8 importing from derivation /nix/store/dscrvrxsk3sc6q86vix26sdd4k7w3l09-spdx-license-list-data-3.12.drv via readFile
/nix/store/jr8mn1grmhlp707j02hnm8pllr89pbsi-source/lib/import-and-filter-project.nix:21:13 importing from derivation /nix/store/hkvscbssj0cmkl5gl15xn84zrr5l3xmm-happy-plan-to-nix-pkgs.drv via scopedImport

src/libexpr/eval.hh

src/libexpr/primops.cc

cidkidnix · 2022-10-12T15:36:48Z

I've fixed/updated this PR/patch to work on Nix 2.9+ if anyone is interested
https://github.com/cidkidnix/nix/tree/log-ifd

fricklerhandwerk · 2023-03-22T10:23:36Z

Triaged in the Nix team meeting 2023-03-17:

The team was concerned it be too verbose, but since it's hidden behind a flag, should be OK. The idea is approved and we will shepherd the implementation to completion.

@cleverca22 are you still interested in finishing the PR?
In case not, @cidkidnix since you've rebased this somewhat recently (on Nix development time scales), would you be able to take it over?

nixos-discourse · 2023-03-22T20:18:34Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-03-17-nix-team-meeting-minutes-41/26614/1

Ericson2314 · 2023-05-13T13:17:26Z

I am going to mark this draft because it is old with conflicts, and also because I suspect the rebase in #8094 will get there first.

This will probably break it, but we can cherry-pick to rebase. Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

cleverca22 force-pushed the log-all-ifd branch from 76ed429 to 8069033 Compare April 14, 2020 18:53

grahamc reviewed Apr 14, 2020

View reviewed changes

cleverca22 force-pushed the log-all-ifd branch from 8069033 to 595971e Compare April 14, 2020 19:03

grahamc reviewed Apr 14, 2020

View reviewed changes

src/libexpr/eval.hh Outdated Show resolved Hide resolved

cleverca22 force-pushed the log-all-ifd branch from 595971e to 3afd739 Compare April 14, 2020 19:07

grahamc approved these changes Apr 14, 2020

View reviewed changes

grahamc reviewed Apr 14, 2020

View reviewed changes

doc/manual/command-ref/conf-file.xml Outdated Show resolved Hide resolved

johnalotoski force-pushed the log-all-ifd branch 2 times, most recently from d43b06d to 2c36d22 Compare April 14, 2020 19:39

edolstra reviewed Apr 15, 2020

View reviewed changes

src/libexpr/eval.hh Outdated Show resolved Hide resolved

edolstra reviewed Apr 15, 2020

View reviewed changes

src/libexpr/primops.cc Outdated Show resolved Hide resolved

matthewbauer mentioned this pull request Apr 16, 2020

Add --include-ifd option to nix-instantiate #3506

Closed

cleverca22 force-pushed the log-all-ifd branch from 2c36d22 to 76f441f Compare May 19, 2020 00:48

deliciouslytyped mentioned this pull request Apr 25, 2021

Nix GC collects derivations used for IFD #719

Open

cleverca22 force-pushed the log-all-ifd branch 2 times, most recently from da9b4f5 to 5cbb618 Compare April 25, 2021 20:25

bqv reviewed Apr 25, 2021

View reviewed changes

src/libexpr/primops.cc Outdated Show resolved Hide resolved

cleverca22 force-pushed the log-all-ifd branch from 5cbb618 to a81c0fa Compare April 25, 2021 20:53

bqv added a commit to bqv/rc that referenced this pull request Apr 25, 2021

pkgs(nixFlakes): optimistically use NixOS/nix#3494

3913593

stale bot added the stale label Oct 30, 2021

stale bot removed the stale label Nov 1, 2021

cleverca22 force-pushed the log-all-ifd branch from a81c0fa to de58e97 Compare February 3, 2022 18:38

cleverca22 force-pushed the log-all-ifd branch from de58e97 to 9a6b441 Compare February 4, 2022 01:24

log all ifd

be43082

cleverca22 force-pushed the log-all-ifd branch from 9a6b441 to be43082 Compare February 4, 2022 01:34

edolstra approved these changes Feb 21, 2022

View reviewed changes

src/libexpr/eval.hh Outdated Show resolved Hide resolved

src/libexpr/eval.hh Outdated Show resolved Hide resolved

src/libexpr/primops.cc Outdated Show resolved Hide resolved

src/libexpr/primops.cc Outdated Show resolved Hide resolved

src/libexpr/primops.cc Outdated Show resolved Hide resolved

madeline-os mentioned this pull request May 12, 2022

Evaluating attribute exe on project ./. {...} require more than 4.5 GiB free space obsidiansystems/obelisk#858

Open

stale bot added the stale label Sep 21, 2022

stale bot removed the stale label Oct 12, 2022

fricklerhandwerk added error-messages Confusing messages and better diagnostics UX The way in which users interact with Nix. Higher level than UI. labels Mar 3, 2023

thufschmitt added the idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. label Mar 17, 2023

cidkidnix mentioned this pull request Mar 22, 2023

Log all IFD #8094

Open

7 tasks

Ericson2314 marked this pull request as draft May 13, 2023 13:17

Apply suggestions from code review

4b3afdb

This will probably break it, but we can cherry-pick to rebase. Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log all ifd #3494

log all ifd #3494

cleverca22 commented Apr 14, 2020

grahamc Apr 14, 2020

bjornfor Apr 14, 2020

grahamc commented Apr 14, 2020

edolstra commented Apr 14, 2020

grahamc commented Apr 14, 2020

cleverca22 commented Apr 15, 2020 •

edited

Ericson2314 commented Apr 15, 2020 •

edited

matthewbauer commented Apr 16, 2020 •

edited

domenkozar commented Apr 16, 2020

disassembler commented May 22, 2020

deepfire commented Jan 13, 2021

bqv commented Feb 4, 2021

stale bot commented Oct 30, 2021

cleverca22 commented Nov 1, 2021

deepfire commented Feb 3, 2022 •

edited

cidkidnix commented Oct 12, 2022

fricklerhandwerk commented Mar 22, 2023 •

edited

nixos-discourse commented Mar 22, 2023

Ericson2314 commented May 13, 2023

log all ifd #3494

Are you sure you want to change the base?

log all ifd #3494

Conversation

cleverca22 commented Apr 14, 2020

grahamc Apr 14, 2020

Choose a reason for hiding this comment

bjornfor Apr 14, 2020

Choose a reason for hiding this comment

grahamc commented Apr 14, 2020

edolstra commented Apr 14, 2020

grahamc commented Apr 14, 2020

cleverca22 commented Apr 15, 2020 • edited

Ericson2314 commented Apr 15, 2020 • edited

matthewbauer commented Apr 16, 2020 • edited

domenkozar commented Apr 16, 2020

disassembler commented May 22, 2020

deepfire commented Jan 13, 2021

bqv commented Feb 4, 2021

stale bot commented Oct 30, 2021

cleverca22 commented Nov 1, 2021

deepfire commented Feb 3, 2022 • edited

cidkidnix commented Oct 12, 2022

fricklerhandwerk commented Mar 22, 2023 • edited

nixos-discourse commented Mar 22, 2023

Ericson2314 commented May 13, 2023

cleverca22 commented Apr 15, 2020 •

edited

Ericson2314 commented Apr 15, 2020 •

edited

matthewbauer commented Apr 16, 2020 •

edited

deepfire commented Feb 3, 2022 •

edited

fricklerhandwerk commented Mar 22, 2023 •

edited