@kuznero @grahamc @puffnfresh @purefn @aneeshusa
Docker image manipulation is a little bit tricky... so if you can try this patch on your image to be sure it doesn't break any existing stuffs!

@nlewo Just tested it with nixpkgs pointing to pr/docker-build-image-registry-format of your nixpkgs fork:

nix-build \                                                                                                                                     100 ↵ 
 -I nixpkgs=/home/kuznero/Projects/GitHub/nixpkgs \                                                                                                                          
-E 'with import <nixpkgs> {}; pkgs.dockerTools.buildImage { fromImage = pkgs.dockerTools.pullImage { imageName = "alpine"; imageTag = "3.6"; sha256 = "120lvxx285f0xmrsq4v3p$
ny6ln32nm35n78rz21p3ybqr454grg";}; name = "test"; }'

This produces following output:

these derivations will be built:                                                                                                                                             
  /nix/store/dnbqmj42yyprsk1vjbfnhbahjlhdk3hq-docker-image-test.tar.gz.drv                                                                                                   
building path(s) ‘/nix/store/05a3fhvydyy9naqv13q10lcgbvv5z4a9-docker-image-test.tar.gz’                                                                                      
Unpacking base image...                                                                                                                                                      
Adding layer...                                                                                                                                                              
tar: Removing leading `/' from member names                                                                                                                                  
Generating image configuration and manifest...                                                                                                                               
Cooking the image...                                                                                                                                                         
Finished.                                                                                                                                                                    
/nix/store/05a3fhvydyy9naqv13q10lcgbvv5z4a9-docker-image-test.tar.gz

Then I also tried to load it into docker just to double check that it really worked with docker load < ./result && docker images, it also looked ok:

1333e580ee66: Loading layer [==================================================>]     148B/148B                                                                              
5bef08742407: Loading layer [==================================================>]   1.99MB/1.99MB                                                                            
Loaded image: test:latest

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
test                latest              82233a67702d        47 years ago        3.96MB

And as a final test I ran docker run --rm -it test echo test and as expected it gave test output.

So, from my perspective it works. I do also plan to check if this works in the scenario of my projects. Will come back with an update a bit later.

Just tested it with my monitor project I was referring to a couple of times in other issues related to dockerTools. And it all works for me now! Thanks for a great job, @nlewo !!!

Ran some tests, too and seems to work
cc @lo1tuma, @offlinehacker and @matejc

This change broke dockerTools for me. I’m using a tool that can work with docker image spec v1.2 compliant images (which is AFAIK what skopeo also supports).
According to the docker image spec the layer changeset tar files should be tar files, but this PR changed this to tar.gz files, which doesn’t make sense to me. I also didn’t found any reason for that in the skopeo documentation.

@nlewo can you please explain why layers are now .tar.gz files.

Apart from that I think dockerTools buildImage should produce spec-compliant images. While it is ok to use other formats internally in order to support base images the final output should be compliant to the docker image spec.

@lo1tuma I was not able to find a clear documentation/spec regarding format image. I proposed this change (WIP) by doing some kind of retroengineering on image produced by Skopeo... (by supposing Skopeo knows what it does).
This image format is the format of image generated by Skopeo and it is also well supported by Docker. Moreover, it is really close from what Docker registry v2 uses since you can directly send tar.gz layers to the registry by using their name (which is the identifier on the registry).
So, I don't know what is this format... :/

What is the tool you are using?

But it would be really nice to use a tool to pack and unpack image since it is really fragile to manually do this kind of things.

I’m using a tool which I implemented myself. The only thing it does is to push a docker image in the Combined Image JSON + Filesystem Changeset Format (spec v1.2) to a given docker registry.
I can’t find anything in the docker image spec that says something about .tar.gz support. However I could imagine that the docker cli tools support more than what the spec says. But the question is do we really want to produce something that is not spec compliant?

@lo1tuma containers/skopeo#425

@lo1tuma So, I think you could continue with the current implementation which seems to be the shema-v2 + blobs version :/ If you find a clear spec regarding this, I'm really interested!

Regarding your script to push images, I use Skopeo to push them https://github.com/nlewo/nixpkgs-cloudwatt/blob/master/lib/image.nix#L35.

@nlewo schema-v2 is AFAIK only a format for the docker registry which is unrelated to the docker image spec itself. There might be other use-cases without any docker registry involved.

The nixpkgs manual says that dockerTools support the docker image spec which is no longer the case now. Even the latest version of the docker image spec is fully backwards compatible, by containing the legacy files. This backwards compatibility was also removed in this PR.

BTW: I agree that is quite hard to find precise information in all those specs and APIs that exist around docker, IMHO the source of truth is still the docker image spec, although there are a lot of tools that support different variations of that format.

This also breaks mesos loading the docker images.. I'll revert this for now