Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dockerTools: fix export #80068

Merged
merged 6 commits into from Sep 29, 2021
Merged

dockerTools: fix export #80068

merged 6 commits into from Sep 29, 2021

Conversation

jbedo
Copy link
Contributor

@jbedo jbedo commented Feb 13, 2020

Motivation for this change

exportImage is currently broken due to a change in runWithOverlay. This fixes exportImage by modifying runWithOverlay slightly. An alternative would be to patch only exportImage, but this would require an extra copy and increase garbage which is undesirable with large docker images.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Test expression used:

{ pkgs ? import <nixpkgs> { } }:

with pkgs.dockerTools;

let
  img = pullImage {
    imageName = "alpine";
    imageDigest =
      "sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d";
    sha256 = "sha256-x8gTbpAxU0wQXxwDUed1Z+fqN/ETZL8dwpQSehf6UU0=";
    finalImageName = "alpine";
    finalImageTag = "latest";
  };

in exportImage { fromImage = img; }

@stale
Copy link

stale bot commented Aug 11, 2020

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 11, 2020
@SuperSandro2000
Copy link
Member

@jbedo is this still relevant?

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Dec 29, 2020
@jbedo
Copy link
Contributor Author

jbedo commented Dec 31, 2020

Yes, the test expression I provided still failed on master and the patch resolves it. I've updated the test expression hash since alpine's latest image has changed since opening the PR.

@roberth
Copy link
Member

roberth commented May 4, 2021

Please add a test in https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/docker-tools.nix so this won't break again in the future.

@jbedo
Copy link
Contributor Author

jbedo commented May 8, 2021

Please add a test in https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/docker-tools.nix so this won't break again in the future.

I've added an example (exportBash) that exercises the function, but adding a test along the lines of

docker.succeed("tar -tf ${examples.exportBash} > /dev/null")

fails with

would reformat /nix/store/2frmry0c8ni7bmza4kjika3ry4k12hwk-nixos-test-driver-docker-tools/test-script
Oh no! 💥 💔 💥
1 file would be reformatted.

and I have no idea why. Any tips?

@roberth
Copy link
Member

roberth commented May 8, 2021

Please add a test in https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/docker-tools.nix so this won't break again in the future.

I've added an example (exportBash) that exercises the function, but adding a test along the lines of

docker.succeed("tar -tf ${examples.exportBash} > /dev/null")

fails with

would reformat /nix/store/2frmry0c8ni7bmza4kjika3ry4k12hwk-nixos-test-driver-docker-tools/test-script
Oh no! 💥 💔 💥
1 file would be reformatted.

and I have no idea why. Any tips?

I've disabled that "check" on master, so if you rebase, it won't happen again. It wasn't an actual linter, but just a code formatter that doesn't understand Nix.

@jbedo
Copy link
Contributor Author

jbedo commented May 18, 2021

The testing all seems good now, we just need to decide how to best resolve the issue. Wrapping is perhaps the cleanest, but I'm not keen on doubling disk usage.

@jbedo
Copy link
Contributor Author

jbedo commented Sep 29, 2021

Sorry for the spam, I'm trying to rebase against current master and it's pulling in other people's commits for some reason.

@jbedo
Copy link
Contributor Author

jbedo commented Sep 29, 2021

OK got there by cherry picking in a new branch, sorry again for the noise.

master is currently broken for exporting images so I still think it's worth resolving this. The test suits (extended with the export image test) passes.

@roberth
Copy link
Member

roberth commented Sep 29, 2021

The test suits (extended with the export image test) passes.

Before 1a0edf1, the new test for exportImage didn't pass because it still wrote a directory. I've fixed it in order to run the tests for my other changes. Do you agree with the fix?

Copy link
Member

@roberth roberth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the test checks the tarball contents, this lgtm.

nixos/tests/docker-tools.nix Outdated Show resolved Hide resolved
@jbedo
Copy link
Contributor Author

jbedo commented Sep 29, 2021

That's a much cleaner solution, thanks! The fix looks good to me.

Copy link
Member

@roberth roberth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jbedo

@roberth
Copy link
Member

roberth commented Sep 29, 2021

x86_64-linux failure on ofborg does not reproduce locally and seems unrelated.

@roberth roberth merged commit ae03fb8 into NixOS:master Sep 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants