Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dockerTools: Add buildNixShellImage #78967

Closed
wants to merge 2 commits into from

Conversation

infinisil
Copy link
Member

@infinisil infinisil commented Jan 31, 2020

Motivation for this change

Edit: An updated and improved version of this is being continued in #172736

This adds buildNixShellImage, which can be used to create a Docker image with an environment to build a derivation, similar to a nix-shell, but much purer.

with import <nixpkgs> {};
dockerTools.buildNixShellImage {
  name = "hello";
  drv = hello;
}
$ nix-build hello.nix
these derivations will be built:
[...]
Finished.
/nix/store/2irs3yciz5pvi8r63bd0w16wnynx4l7b-docker-image-hello-env.tar.gz
$ docker load -i result
506c9779afde: Loading layer  10.24kB/10.24kB
[...]
96557dcba6be: Loading layer  10.24kB/10.24kB
Loaded image: hello-env:2irs3yciz5pvi8r63bd0w16wnynx4l7b
$ docker run -it hello-env:2irs3yciz5pvi8r63bd0w16wnynx4l7b
bash-4.4$ genericBuild
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
[...]
patching script interpreter paths in /home/outputs/out
checking for references to /tmp/ in /home/outputs/out...
bash-4.4$ $out/bin/hello
Hello, world!

This PR was sponsored by Niteo

Ping @grahamc @nlewo @danieldk

Things done
  • Made sure it can build pkgs.hello, pkgs.openssl and pkgs.id3v2
  • Wrote docs for how to use it
  • Built the docs successfully

Copy link
Contributor

@tomberek tomberek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builds and functions as expected on NixOS. Thanks for the documentation!

@danbst
Copy link
Contributor

danbst commented Feb 3, 2020

is it possible to mount source as a volume?

...
src = ./some_directory;

should make $src to point to /source, which can be then binded to hosts ./some_directory. Obviously, when src is specified, unpackPhase and patchPhase should be run manually.

Copy link
Member

@nlewo nlewo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to add a test in nixos/tests/docker-tools.nix. I think building and executing hello would be perfect ;)

pkgs/build-support/docker/default.nix Show resolved Hide resolved
, drv
, tag ? null
# Extra packages available inside the image
, extraPackages ? []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about calling this contents? It seems to me it behaves like the contents attribute of our other docker tools.

let

finalDrv = drv.overrideAttrs (old: {
nativeBuildInputs = old.nativeBuildInputs or [] ++ extraPackages;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just adding extraPackages in the contents list?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm it's not the same thing: contents are added to the / path, but extraPackages are things that are added to PATH (and other env vars) later by stdenv's setup-hook. So by using nativeBuildInputs all kinds of dependencies should work, though now I'm debating whether that's even needed. Maybe it's needed for man support or so though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this is to allow a user to add extra tooling in its container. I think, in other images, this use case is covered by the contents attribute, even if it is less convenient than what you are doing (because enviroment variables are not set). In order to have a coherent dockerTools API, I would prefer to expose the dockerTools.buildLayeredImage contents and env attributes: if the user needs extra tooling, it would be still possible to add packages with contents and set some environment variables with env. WDYT?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this again, I feel like extraPackages should be removed, because people can just use .overrideAttrs on the derivation they pass in instead.

Similarly, I don't think an env is needed, because this can also be achieved by setting that in the derivation in multiple ways already:

{
  VAR = "VAL";
  shellHook = ''
    export VAR=VAL
  '';
}

I think the idea of passing through contents is a good one though, I'll change it as such

@stale
Copy link

stale bot commented Sep 12, 2020

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 12, 2020
@tomberek
Copy link
Contributor

still important to me

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 12, 2020
@zupo
Copy link
Contributor

zupo commented Sep 13, 2020

Me too.

@yipengsun
Copy link
Contributor

This tool looks really nice for preserving development environment for teams that don't universally use nix! However, given that nix is introducing flakes and nix-shell seems to be replaced by nix develop, how much tweak is needed to make this work with the flake workflow?

Thanks.

@3noch
Copy link
Contributor

3noch commented Jan 25, 2021

This would be amazing!


# Pass --login such that /etc/profile gets loaded
# Use bashInteractive for tab completion support and such
config.Entrypoint = [ "${bashInteractive}/bin/bash" "--login" ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is different from the bash image, which has this entrypoint https://github.com/tianon/docker-bash/blob/master/5.1/docker-entrypoint.sh

When I try this in a normal session I get

$ bash --login echo foo
/run/current-system/sw/bin/echo: /run/current-system/sw/bin/echo: cannot execute binary file

Which seems to be a cryptic way of telling us that it's refusing to interpret an ELF executable as a script.

Passing a script does work

$ bash --login <(echo "echo hello") asdf
hello

So I think it should be like the official bash entrypoint.

The Cmd should probably be ["bash"].

We'll need a test for this. You can add cases to nixos/tests/docker-tools.nix, based on a new example you can add to pkgs/build-support/docker/examples.nix.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I get the motivation. nix-shell is bash-based, but it's not an image for the purpose of executing bash. In this case, we need some way to source stdenv and execute shellHook when it starts, which I'm now doing by passing --login and writing the shell init to /etc/profile. Currently this allows you to either do docker run -it <image>, then run genericBuild, or you can also do

docker run -it <image> -c genericBuild

And this works, which I don't think would be the case with your suggestion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we seem to have different use cases in mind, but we can probably support both.

  • use the nix-shell image to build something
  • use the nix-shell image as a developer environment

For building, we should invoke the builder with args and not wrongly assume bashInteractive.
For interactive use, I do think we should emulate the official bash image behavior.

Can't we support both?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now changed it to have no entrypoint at all, however there's bash, all of coreutils and a special buildDerivation binary in the PATH. In addition, the Cmd is set to [ "bash" ]. I managed to get around the --login requirement by using ~/.bashrc instead. This means by default you'll get a nix-shell environment, but you can also do any of

$ docker run -it <image> # nix-shell
$ docker run -it <image> bash # nix-shell
$ docker run -it <image> buildDerivation # build the derivation directly
$ docker run -it <image> bash -c 'buildDerivation && ls $out/bin' # builds derivation and lists the binaries it produced

pkgs/build-support/docker/default.nix Outdated Show resolved Hide resolved
Comment on lines 942 to 940
# We can't just use `toString` on all derivation attributes because that
# would not put path literals in the closure. So we explicitly copy
# those into the store here
stringValue = value: if builtins.typeOf value == "path" then "${value}"
else if builtins.isList value then toString (map stringValue value)
else toString value;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we're basically emulating the nix sandbox env setup here. Structured attrs and passAsFile are missing, but that's probably ok for a start.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, didn't want to bother with structuredAttrs, but I guess I can issue a warning when it's enabled

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An assertion now triggers when __structuredAttrs is set. The current code can't handle __structuredAttrs at all really.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about passAsFile though?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that didn't work before, but I implemented support for it now :)

pkgs/build-support/docker/default.nix Outdated Show resolved Hide resolved
Comment on lines 916 to 917
cd /home
'' + extraInit;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extraInit seems like a rather arbitrary term that doesn't have a well-understood purpose. Can't we remove it and rely on the user extending shellHook beforehand, or setting an appropriate Cmd?

cd /home breaks the WorkingDir config item.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm removing extraInit in the next iteration. Can also be done with just shellHook.

WorkingDir can't be configured in the current version anyways

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With docker you can always configure any config attribute afterwards.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now using WorkingDir directly to specify this, instead of cd ing

pkgs/build-support/docker/default.nix Outdated Show resolved Hide resolved
This also readds support for those arguments to buildLayeredImage, which
was removed in 560201d
@infinisil
Copy link
Member Author

In addition to above things, I also changed the derivation output variables to point directly at the correct /nix/store path, which allows placeholder to work correctly (tested with bluez-alsa). I also set a default name for the image.

@infinisil
Copy link
Member Author

I'll update/add docs/tests/examples once I get confirmation by @nlewo and/or @roberth regarding the current implementation.

'';

# A mapping from output name to the nix store path where they should end up
outputPaths = lib.genAttrs drv.outputs (output: builtins.unsafeDiscardStringContext drv.${output}.outPath);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these paths writable in the store?
We should have a test case for this as well.

It would be nice to have symlinks to these paths in a fixed location for derived images to use.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are indeed writable. Perhaps I could create /home/user/outputs/* again, symlinking into the store

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now created symlinks from /home/user/outputs/* into the /nix/store paths

@tomberek
Copy link
Contributor

Result of nixpkgs-review pr 78967 run on x86_64-linux 1

@3noch
Copy link
Contributor

3noch commented Feb 2, 2021

@infinisil What else needs to be done for this?

@roberth
Copy link
Member

roberth commented Feb 2, 2021

@3noch @infinisil mostly just tests by the looks of it.

@yipengsun
Copy link
Contributor

For the documentation, it is using nix-build as an example and for drv, I think I can just import my shell.nix. However, I'm confused about how to do this for a flake-based workflow.

Naively I'd just use the backward-compatibility tool to "expose" my devShell to a shell.nix and then build from that. Can I do something similar without any compatibility tool?

@roberth
Copy link
Member

roberth commented Mar 15, 2021

@yipengsun You could move the buildNixShellImage call under your flake's packages and reference the shell derivation via self.
I believe there may be a function for invoking flakes but that's at least undocumented so ymmv.

exec ${lib.escapeShellArg (stringValue drv.drvAttrs.builder)} ${lib.escapeShellArgs (map stringValue drv.drvAttrs.args)}
'';

in buildLayeredImage {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to override some settings in this config. I also want to use streamLayeredImage. So really it makes sense to just return the attrset directly and let the user decide where the image is actually realized.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use (buildLayeredImage args).stream to solve that part of the problem.
I feel the increasing number of interfaces for building docker images is starting to become a problem, so I'm not excited about having a function that returns arguments for another function. I'd like to experiment with a more powerful interface during Ocean Sprint, so I'll know more in a month.

inherit uid gid;

# By default just starts a shell
config.Cmd = [ "bash" ];
Copy link
Contributor

@3noch 3noch May 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be easily overridable. I want to override it in my use case as I don't want to only be able to run containers interactively. Using shellHook to run a task means it only works when .bashrc is loaded which means it must be interactive.

@stale
Copy link

stale bot commented Nov 9, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 9, 2021
@3noch
Copy link
Contributor

3noch commented Nov 14, 2021

This is important to get in.

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 14, 2021
@roberth
Copy link
Member

roberth commented Nov 14, 2021

This needs tests.
It needs to test some examples and compare its behavior against regular nix-shell.

@@ -496,4 +496,123 @@ buildImage {
Creating base files like <literal>/etc/passwd</literal> or <literal>/etc/login.defs</literal> is necessary for shadow-utils to manipulate users and groups.
</para>
</section>
<section xml:id="ssec-pkgs-dockerTools-buildNixShellImage">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that file not yet converted to markdown?

@infinisil
Copy link
Member Author

I don't have any interest in this anymore. I think this would be really nice to get in, but I'll leave it up to someone that actually wants to use it ;). Anybody feel free to use the code from this PR

@infinisil infinisil closed this Nov 29, 2021
@infinisil infinisil deleted the docker-nix-shell branch March 9, 2022 19:50
@infinisil
Copy link
Member Author

Check out the new #172736, where I'm continuing work on this function

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants