Comparing changes
Open a pull request
base repository: NixOS/nixpkgs-channels
base: 4c95508641fe
head repository: NixOS/nixpkgs-channels
compare: 89b618771ad4
- 14 commits
- 12 files changed
- 9 contributors
Commits on Sep 26, 2018
-
Configuration menu - View commit details
-
Copy full SHA for 2bf0ee3 - Browse repository at this point
Copy the full SHA 2bf0ee3View commit details -
referencesByPopularity: init to sort packages by a cachability heuristic
Using a simple algorithm, convert the references to a path in to a sorted list of dependent paths based on how often they're referenced and how deep in the tree they live. Equally-"popular" paths are then sorted by name. The existing writeReferencesToFile prints the paths in a simple ascii-based sorting of the paths. Sorting the paths by graph improves the chances that the difference between two builds appear near the end of the list, instead of near the beginning. This makes a difference for Nix builds which export a closure for another program to consume, if that program implements its own level of binary diffing. For an example, Docker Images. If each store path is a separate layer then Docker Images can be very efficiently transfered between systems, and we get very good cache reuse between images built with the same version of Nixpkgs. However, since Docker only reliably supports a small number of layers (42) it is important to pick the individual layers carefully. By storing very popular store paths in the first 40 layers, we improve the chances that the next Docker image will share many of those layers.* Given the dependency tree: A - B - C - D -\ \ \ \ \ \ \ \ \ \ \ - E ---- F \- G Nodes which have multiple references are duplicated: A - B - C - D - F \ \ \ \ \ \- E - F \ \ \ \- E - F \ \- G Each leaf node is now replaced by a counter defaulted to 1: A - B - C - D - (F:1) \ \ \ \ \ \- E - (F:1) \ \ \ \- E - (F:1) \ \- (G:1) Then each leaf counter is merged with its parent node, replacing the parent node with a counter of 1, and each existing counter being incremented by 1. That is to say `- D - (F:1)` becomes `- (D:1, F:2)`: A - B - C - (D:1, F:2) \ \ \ \ \ \- (E:1, F:2) \ \ \ \- (E:1, F:2) \ \- (G:1) Then each leaf counter is merged with its parent node again, merging any counters, then incrementing each: A - B - (C:1, D:2, E:2, F:5) \ \ \ \- (E:1, F:2) \ \- (G:1) And again: A - (B:1, C:2, D:3, E:4, F:8) \ \- (G:1) And again: (A:1, B:2, C:3, D:4, E:5, F:9, G:2) and then paths have the following "popularity": A 1 B 2 C 3 D 4 E 5 F 9 G 2 and the popularity contest would result in the paths being printed as: F E D C B G A * Note: People who have used a Dockerfile before assume Docker's Layers are inherently ordered. However, this is not true -- Docker layers are content-addressable and are not explicitly layered until they are composed in to an Image.
Configuration menu - View commit details
-
Copy full SHA for fd04517 - Browse repository at this point
Copy the full SHA fd04517View commit details -
dockerTools.buildLayeredImage: init
Create a many-layered Docker Image. Implements much less than buildImage: - Doesn't support specific uids/gids - Doesn't support runninng commands after building - Doesn't require qemu - Doesn't create mutable copies of the files in the path - Doesn't support parent images If you want those feature, I recommend using buildLayeredImage as an input to buildImage. Notably, it does support: - Caching low level, common paths based on a graph traversial algorithm, see referencesByPopularity in 0a80233487993256e811f566b1c80a40394c03d6 - Configurable number of layers. If you're not using AUFS or not extending the image, you can specify a larger number of layers at build time: pkgs.dockerTools.buildLayeredImage { name = "hello"; maxLayers = 128; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; - Parallelized creation of the layers, improving build speed. - The contents of the image includes the closure of the configuration, so you don't have to specify paths in contents and config. With buildImage, paths referred to by the config were not included automatically in the image. Thus, if you wanted to call Git, you had to specify it twice: pkgs.dockerTools.buildImage { name = "hello"; contents = [ pkgs.gitFull ]; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; buildLayeredImage on the other hand includes the runtime closure of the config when calculating the contents of the image: pkgs.dockerTools.buildImage { name = "hello"; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; Minor Problems - If any of the store paths change, every layer will be rebuilt in the nix-build. However, beacuse the layers are bit-for-bit reproducable, when these images are loaded in to Docker they will match existing layers and not be imported or uploaded twice. Common Questions - Aren't Docker layers ordered? No. People who have used a Dockerfile before assume Docker's Layers are inherently ordered. However, this is not true -- Docker layers are content-addressable and are not explicitly layered until they are composed in to an Image. - What happens if I have more than maxLayers of store paths? The first (maxLayers-2) most "popular" paths will have their own individual layers, then layer #(maxLayers-1) will contain all the remaining "unpopular" paths, and finally layer #(maxLayers) will contain the Image configuration.
Configuration menu - View commit details
-
Copy full SHA for 4fe9006 - Browse repository at this point
Copy the full SHA 4fe9006View commit details
Commits on Sep 27, 2018
-
Configuration menu - View commit details
-
Copy full SHA for d1e46df - Browse repository at this point
Copy the full SHA d1e46dfView commit details -
Configuration menu - View commit details
-
Copy full SHA for fb2d153 - Browse repository at this point
Copy the full SHA fb2d153View commit details
Commits on Sep 30, 2018
-
Configuration menu - View commit details
-
Copy full SHA for bf46094 - Browse repository at this point
Copy the full SHA bf46094View commit details
Commits on Oct 1, 2018
-
Configuration menu - View commit details
-
Copy full SHA for 396320c - Browse repository at this point
Copy the full SHA 396320cView commit details -
Merge pull request #47569 from NeQuissimus/kernel_4_19_rc6
linux: 4.19-rc5 -> 4.19-rc6
Configuration menu - View commit details
-
Copy full SHA for 97920e9 - Browse repository at this point
Copy the full SHA 97920e9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3ac175e - Browse repository at this point
Copy the full SHA 3ac175eView commit details -
Configuration menu - View commit details
-
Copy full SHA for c417b26 - Browse repository at this point
Copy the full SHA c417b26View commit details -
Merge pull request #47411 from graham-at-target/multi-layered-images-…
…crafted Multi-Layered Docker Images
Configuration menu - View commit details
-
Copy full SHA for 56b4db9 - Browse repository at this point
Copy the full SHA 56b4db9View commit details -
Configuration menu - View commit details
-
Copy full SHA for b256df4 - Browse repository at this point
Copy the full SHA b256df4View commit details -
Merge pull request #47582 from srhb/dockerTools-nix-stable
dockerTools: Use nix instead of nixUnstable
Configuration menu - View commit details
-
Copy full SHA for 976edf1 - Browse repository at this point
Copy the full SHA 976edf1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 89b6187 - Browse repository at this point
Copy the full SHA 89b6187View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff 4c95508641fe...89b618771ad4