Substitutions from different store dirs #3689

matthewbauer · 2020-06-12T14:52:30Z

Substituters can substitute from one store dir to another with a
little bit of help. The store api just needs to have a CA so it can
recompute the store path based on the new store dir. We can only do
this for fixed output derivations with no references, though.

Substituters can substitute from one store dir to another with a little bit of help. The store api just needs to have a CA so it can recompute the store path based on the new store dir. We can only do this for fixed output derivations with no references, though.

this needs to only continue if the path replacement fails.

src/libstore/local-store.cc

src/libstore/store-api.cc

This can be used to add flat hashes to the nix store.

these rewrites should be transparent, but they are important to know about when debugging

We can’t use custom name here because different names will have different store paths. This is a limitation of the Store API’s reliance on store paths. We might be able to get around the above in the future by using a dummy name for certain fixed output paths.

This puts what we are already doing into a shared method. It just needs a path name and a ca and produces a store path.

Originally, the test was only checking for different “real” storeDir. That’s an easy case to handle, but the much harder one is if different virtual store dirs are used. To do this, we need the SubstitutionGoal to know about the ca, so it can recalculate the path to copy it over. An important note here is that the store path passed to copyStorePath needs to be one for srcStore - so that queryPathInfo works properly. This also adds an error message when the store path from queryPathInfo is different from the one we requested.

Ericson2314 · 2020-06-13T17:39:39Z

@edolstra We really like this as a new future. One downside of replacing hashed mirrors with it, however, is that we still use the same "name" when rewriting the path even though it is immaterial for fixed output derivations. This means one has to be quite cautious about renaming fixed output derivations, which isn't really the case today.

I might still consider the old #3673 in addition to this as a stop-gap until we add another mitigation, such as making a <ca>.cainfo file to allow looking up content addresses rather than store paths directly. That also gives people using hashed mirrors some time to migrate.

matthewbauer · 2020-06-14T00:50:52Z

There are some legitimate issues with #3673 I think though. We don’t have any good disk caching because we don’t have narinfo and we also will end up requesting a lot more things from tarballs.nixos.org than we currently do. It might make sense to keep hashed-mirrors for people that still use them.

Ericson2314 · 2020-06-14T03:16:19Z

I am fine with either transition plan (this + #3673 hash mirror store or this without removing hashed mirrors), but for the record I do think we could fix the issues with #3673 with enough elbow grease (for example, perhaps CURLOPT_NOBODY to just see whether something exists).

src/libstore/build.cc

src/nix/add-to-store.cc

src/libstore/store-api.hh

src/libstore/store-api.cc

…oredir

src/libstore/build.cc

edolstra · 2020-07-02T09:12:28Z

src/libstore/build.cc

@@ -4526,8 +4545,15 @@ void SubstitutionGoal::tryToRun()
            Activity act(*logger, actSubstitute, Logger::Fields{worker.store.printStorePath(storePath), sub->getUri()});
            PushActivity pact(act.id);

+            auto subPath = storePath;
+            if (ca) {
+                subPath = sub->makeFixedOutputPathFromCA(storePath.name(), *ca);


Since we already computed subPath in SubstitutionGoal::tryNext(), we could avoid computing it again by storing subPath in SubstitutionGoal.

…oredir

Ericson2314 · 2020-07-07T19:19:05Z

I think error in CI has been fixed on master since.

…oredir

nixos-discourse · 2020-07-26T20:05:17Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/67

Mic92 · 2020-07-27T22:00:44Z

~~Wow this actually really cool. This could solve installing nix without root into the user's home right?~~. No this won't work as noted in the initial comment.

Ericson2314 · 2020-07-27T22:59:59Z

@Mic92 So it is my goal to handle more obscure situations like:

rewriting content addresses with store paths with different store dirs if the store dir lengths align.
rewriting names if the lengths align
opting in to rewriting if the lengths don't align, but the destination is shorter so we can pad with ////

This basic approach scales to those sorts of things, and combined with CA derivations will get us decently close to making arbitrary store dirs work in practice. I think that is the right foundation on the Nix sides to then pursue the various tricks discussed in NixOS/rfcs#17 on the Nixpkgs side and get all the way there for practical purposes.

Mic92 · 2020-07-27T23:15:36Z

@Mic92 So it is my goal to handle more obscure situations like:
* rewriting content addresses with store paths with different store dirs if the store dir lengths align.

* rewriting names if the lengths align

* _opting in_ to rewriting if the lengths don't align, but the destination is shorter so we can pad with `////`
This basic approach scales to those sorts of things, and combined with CA derivations will get us decently close to making arbitrary store dirs work in practice. I think that is the right foundation on the Nix sides to then pursue the various tricks discussed in NixOS/rfcs#17 on the Nixpkgs side and get all the way there for practical purposes.

I thought a bit about changing the length of the store path i.e. exporting to different directories. The hard part are string references elf binaries, but it may be possible to solve this as well: https://github.com/yalue/elf32_string_replace There might be other binary formats that could also break but I think with plain text substitutions + elf binaries most programs would already run.

Ericson2314 · 2020-07-27T23:29:15Z

@Mic92 Cool. Basically it's my view that we need the basic Nix support to get us motivated, but once we are motivated we can persue all the tricks in the book to beat back mass rebuilds in Nixpkgs / user code:

Rewrite strings tables, or even delay linking of data sections (c.f. how I think gettext internationalization works)
Can't actually stat SOs or dylibs during build (trying to observe file or even whether it is mounted at location would fail build), and just get dummy path and symbols via TAPI file or linker script (cc @eternaleye)
Using LTO and other tricks to share work between optimization levels---it's great to get a quick smoke test with global -O0 and if all is well wait for bigger builds
CA drv mass rebuild in parallel #3819
Moving stuff from bash to eval time

And further generic Nix mechanisms to help:

Recursive nix (hopefully not the nix-build in nix-build variety!)

I think we can really pull all stops and increase our productivity by orders of magnitude. Just need to get these initial steps out of the way so the "crowd" is motivated enough that the rest can be crowdsourced.

edit: clarified should should happen in external code vs Nix.

edolstra · 2020-07-28T09:44:37Z

Sorry, but that's crazy talk. There is no way we're going to teach Nix how to rewrite ELF binaries, or any file format. Nix is a generic package manager so it doesn't (and shouldn't) have any knowledge of particular file formats.

And I don't think it's even possible for ELF binaries. A tool like patchelf can rewrite some things because we know the relevant structure. But .data and .text (code) sections don't really have any defined structure, so if we move a string like /nix/store/abc in a data section, we wouldn't know how to update the references in the text section. (ELF binaries don't have to be relocatable after all, so a machine instruction in the .text section can refer directly to a .data address like 0x58a000.)

Also, how would you distinguish between /nix/store occurring as part of a reference, versus (say) in an HTML doc that shouldn't be rewritten (e.g. a string like "Nix by default uses /nix/store")?

Mic92 · 2020-07-28T10:04:03Z

Sorry, but that's crazy talk. There is no way we're going to teach Nix how to rewrite ELF binaries, or any file format. Nix is a generic package manager so it doesn't (and shouldn't) have any knowledge of particular file formats.

And I don't think it's even possible for ELF binaries. A tool like patchelf can rewrite some things because we know the relevant structure. But .data and .text (code) sections don't really have any defined structure, so if we move a string like /nix/store/abc in a data section, we wouldn't know how to update the references in the text section. (ELF binaries don't have to be relocatable after all, so a machine instruction in the .text section can refer directly to a .data address like 0x58a000.)

Also, how would you distinguish between /nix/store occurring as part of a reference, versus (say) in an HTML doc that shouldn't be rewritten (e.g. a string like "Nix by default uses /nix/store")?

That would be external tools, not part of nix, like patchelf is not part of nix itself. It's possible to compile binaries to be relocatable.

Ericson2314 · 2020-07-29T02:16:20Z

Sorry, but that's crazy talk. There is no way we're going to teach Nix how to rewrite ELF binaries, or any file format. Nix is a generic package manager so it doesn't (and shouldn't) have any knowledge of particular file formats.
And I don't think it's even possible for ELF binaries. A tool like patchelf can rewrite some things because we know the relevant structure. But .data and .text (code) sections don't really have any defined structure, so if we move a string like /nix/store/abc in a data section, we wouldn't know how to update the references in the text section. (ELF binaries don't have to be relocatable after all, so a machine instruction in the .text section can refer directly to a .data address like 0x58a000.)
Also, how would you distinguish between /nix/store occurring as part of a reference, versus (say) in an HTML doc that shouldn't be rewritten (e.g. a string like "Nix by default uses /nix/store")?

That would be external tools, not part of nix, like patchelf is not part of nix itself. It's possible to compile binaries to be relocatable.

Sorry!! Yes I absolutely did mean for any "tricks" to be in Nixpkgs/user code and not Nix, which I agree should stay a generic tool (c.f. CMake vs Ninja). Edited my post to make that clear----keep the kitchen sink out of Nix :)!

nixos-discourse · 2020-07-30T14:38:52Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-var-nix-opt-nix-usr-local-nix/7101/69

…oredir

bhipple · 2020-08-01T01:06:57Z

It looks like you're deleting hashed mirrors, but is there a way to have a dead simple FTP server serving source tarballs for mirroring into enterprise networks? We rely really heavily on the hashed mirrors in our production setup. Is it possible to make a simple nix binary cache that can only hold fixed-output derivations?

Ericson2314 · 2020-08-01T01:25:06Z

Is it possible to make a simple nix binary cache that can only hold fixed-output derivations?

@bhipple A previous version of this made a substituter out of the hashed mirrors code, so existing hashed mirrors still work. @edolstra didn't want that since regular binaries caches subsume it, but I think it would be great to resurrect that as a store plugin or something.

Ericson2314 · 2020-08-01T01:33:41Z

@bhipple https://github.com/NixOS/nix/pull/3673/files is the earlier version with the HashedMirrorStore.

Some users have their own hashed-mirrors setup, that is used to mirror things in addition to what’s available on tarballs.nixos.org. Although this should be feasable to do with a Binary Cache, it’s not always easy, since you have to remember what "name" each of the tarballs has. Continuing to support hashed-mirrors is cheap, so it’s best to leave support in Nix. Note that NIX_HASHED_MIRRORS is also supported in Nixpkgs through fetchurl.nix. Note that this excludes tarballs.nixos.org from the default, as in \NixOS#3689. All of these are available on cache.nixos.org.

When deploying a Hydra instance with current Nix master, most builds would not run because of errors like this: queue monitor: error: --- Error --- hydra-queue-runner error: --- UsageError --- nix-daemon not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv The last error message is from parseContentAddress, which expects a colon-separated string, however what we got here is a store path. Looking at the worker protocol, the following message sent to the Nix daemon caused the error above: 0x1E -> wopQuerySubstitutablePathInfos 0x01 -> Number of paths 0x16 -> Length of string "/nix/store/...-somedrv" 0x00 -> Length of string "" Looking at writeStorePathCAMap, the store path is indeed the first field that's transmitted. However, readStorePathCAMap expects it to be the *second* field *on my machine*, since expression evaluation order is a classic form of unspecified behaviour[1] in C++. This has been introduced in #3689, specifically in commit 66a62b3. [1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions Signed-off-by: aszlig <aszlig@nix.build>

matthewbauer added 2 commits June 12, 2020 09:49

Remove hashed-mirrors

3e3eaa9

matthewbauer mentioned this pull request Jun 12, 2020

WIP: Add Store API-based HashedMirrors #3673

Closed

Fix storeDir != storeDir condition

11c9707

this needs to only continue if the path replacement fails.