Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: NixOS/nixos-homepage
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 1d6422b54834
Choose a base ref
...
head repository: NixOS/nixos-homepage
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: ed3897f521f8
Choose a head ref
  • 1 commit
  • 1 file changed
  • 1 contributor

Commits on Sep 11, 2020

  1. Update flake.lock and blogs.xml [ci skip]

    GitHub Actions committed Sep 11, 2020
    Copy the full SHA
    ed3897f View commit details
Showing with 15 additions and 14 deletions.
  1. +15 −14 blogs.xml
29 changes: 15 additions & 14 deletions blogs.xml
Original file line number Diff line number Diff line change
@@ -70,29 +70,30 @@ colleague at the other side of the world type <code class="language-text
same content in your tree. Git commits are content addressed, therefore the name
<code class="language-text">7cc16bb8cd38ff5806e40b32978ae64d54023ce0</code> refers to that exact
tree.</p>
<p>Yet another content-addressed storage is <a href="https://ipfs.io/">IPFS</a>. In IPFS storage
files can be stored in any number of computers, and even move from
computer to computer. The content-derived name is used a way to give
<p>Yet another example of content-addressed storage is <a href="https://ipfs.io/">IPFS</a>. In IPFS storage
files can be stored in any number of computers, and even moved from
computer to computer. The content-derived name is used as a way to give
an intrinsic name to a file, regardless of where it is stored.</p>
<p>In fact, even the particular use case that we are discussing here:
avoid recompilation when a rebuilt dependency hasn’t changed, can be
found in various build systems such as
<p>In fact, even the particular use case that we are discussing here -
avoiding recompilation when a rebuilt dependency hasn’t changed -
can be found in various build systems such as
<a href="https://bazel.build/">Bazel</a>. In build systems, such recompilation
avoidance is sometimes known as the <em>early cutoff optimization</em> − see the <a href="https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf">build systems a la carte paper</a> for example).</p>
avoidance is sometimes known as the <em>early cutoff optimization</em>
see the <a href="https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf">build systems a la carte paper</a>
for example).</p>
<p>So all we need to do is to move the Nix store from an input-addressed
model, to a content-addressed model, as used by many tools
already, and we will be able to save a lot of storage space and cpu
usage, by rebuilding much fewer packages. Nixpkgs contributors will
see their CI time improved. It could also help <a href="https://github.com/NixOS/nix/issues/859">serving a binary cache
model to a content-addressed model, as used by many tools
already, and we will be able to save a lot of storage space and CPU
usage, by rebuilding many fewer packages. Nixpkgs contributors will
see their CI time improved. It could also allow <a href="https://github.com/NixOS/nix/issues/859">serving a binary cache
over IPFS</a>.</p>
<p>Well, like many things with computers,
this is actually way harder than it sounds (which explains why this
isn’t done despite being already discussed nearly 15 years ago in the
hasn’t already been done despite being discussed nearly 15 years ago in the
<a href="https://github.com/edolstra/edolstra.github.io/raw/49a78323f6b319da6e078b4f5f6b3112a30e8db9/pubs/phd-thesis.pdf">original paper</a>), but we now believe that there’s <a href="https://github.com/NixOS/rfcs/pull/62">a way forward</a>… more on that in a later post.</p>
<h2>Conclusion</h2>
<p>A content-addressed store for Nix would help reduce the insane load
that <a href="https://hydra.nixos.org">Hydra</a> has to sustain. But, despite the fact that
content-addressing is a common technique both in distributed systems
that <a href="https://hydra.nixos.org">Hydra</a> has to sustain. While content-addressing is a common technique both in distributed systems
and build systems (Nix is both!), getting to the point where it was
feasible to integrate content-addressing in Nix has been a long journey.</p>
<p>In a future post, I’ll explain why it was so hard, and how we finally