Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-lib.nix: initialize pkgs for the currentSystem #71791

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

basvandijk
Copy link
Member

Previously calling release-lib for only x86_64-darwin like:

  import ./release-lib.nix {
    supportedSystems = [ "x86_64-darwin" ];
  }

would still cause the packageSet to be evaluated for
x86_64-linux. This is annoying if you are evaluating this on
x86_64-darwin and your packageSet uses Import From Derivation to
produce a jobset. In that case you need to have a x86_64-linux build
machine configured in order to finish the evaluation.

This commit initializes pkgs for the currentSystem such that all IFD
derivations are build for the current system.

This shouldnt change anything for hydra.nixos.org since there
builtins.currentSystem == "x86-64-linux".

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

@grahamc grahamc requested a review from edolstra October 23, 2019 09:24
@grahamc
Copy link
Member

grahamc commented Oct 23, 2019

Ideally @edolstra will take a look at this one.

@FRidh
Copy link
Member

FRidh commented Oct 23, 2019

You know you can use e.g. the following?

import ./release-lib.nix {
  supportedSystems = [ "x86_64-darwin" ];
  nixpkgsArgs = { system = "x86_64-darwin"; };
}

Along with whatever Nixpkgs config you want in nixpkgsArgs.

Generally I think we should avoid impurities like builtins.currentSystem, however, here it seems to make sense to me, as that function seems to be intended to be used when testing for the current system.

@edolstra
Copy link
Member

I've been eliminating the use of currentSystem because it doesn't work in pure evaluation mode (da5fc5c, 7eb332a). Note that the nix command will default to pure mode in the future. So reintroducing currentSystem is not a good idea IMHO.

@matthewbauer
Copy link
Member

I've been eliminating the use of currentSystem because it doesn't work in pure evaluation mode (da5fc5c, 7eb332a). Note that the nix command will default to pure mode in the future. So reintroducing currentSystem is not a good idea IMHO.

builtins.currentSystem still seems like a better thing than hardcoding "x86_64-linux" everywhere. I think a compromise might be to add a "currentSystem" arg here that Nix could some day provide automatically. I think pure-eval mode can handle this?

@basvandijk
Copy link
Member Author

@FRidh thanks for pointing out the work around of passing system via nixpkgsArgs.

That solves my problem and it doesn't introduce any impurities that @edolstra pointed out. So closing this.

@basvandijk basvandijk closed this Oct 25, 2019
@basvandijk
Copy link
Member Author

basvandijk commented Oct 31, 2019

@FRidh unfortunately I discovered today that the nixpkgsArgs = { system = "x86_64-darwin"; }; workaround doesn't work because it overrides the system argument passed to packageSet' thereby causing release-lib to only create jobs for x86_64-darwin.

@edolstra what about introducing an additional system argument to release-lib which defaults to builtins.currentSystem? That will solve the problem in the OP and allow users who use pure mode to override it to x86_64-linux.

I've forced-pushed the aforementioned approach to this PR.

@basvandijk basvandijk reopened this Oct 31, 2019
@basvandijk basvandijk force-pushed the use-currentSystem-in-release-lib branch from 76034ed to 3fece9c Compare October 31, 2019 21:23
Previously calling release-lib for only x86_64-darwin like:

  import ./release-lib.nix {
    supportedSystems = [ "x86_64-darwin" ];
  }

would still cause the packageSet to be evaluated for
x86_64-linux. This is annoying if you are evaluating this on
x86_64-darwin and your packageSet uses Import From Derivation to
produce a jobset. In that case you need to have a x86_64-linux build
machine configured in order to finish the evaluation.

This commit initializes `pkgs` for the currentSystem such that all IFD
derivations are build for the current system.
@Ericson2314
Copy link
Member

Ericson2314 commented Nov 1, 2019

Why does release-lib need pkgs at all? If it is just for my assertTrue then I am sure we can do something better.

@basvandijk
Copy link
Member Author

@Ericson2314

Why does release-lib need pkgs at all?

pkgs is primarily used at call-sites of release-lib.

In nixpkgs itself there are plenty of examples. For example pkgs/top-level/release.nix brings pkgs into scope here and uses pkgs in lots of places.

At work I use the following for example:

      release-lib = import (pkgs.path + "/pkgs/top-level/release-lib.nix") { 
        # Here it would be great if I could specify:
        #   system = builtins.currentSystem;
        # Such that `release-lib.pkgs` is always evaluated for the current system 
        # which enables darwin users to evaluate this file 
        # without them needing a x86_64-linux builder.
      };

      jobs = release-lib.mapTestOn
        (lib.mapAttrsRecursiveCond
          (as: !lib.isDerivation as)
          (_path: drv: release-lib.supportedMatches (drv.meta.platforms or supportedSystems))
          release-lib.pkgs);

@Ericson2314
Copy link
Member

Ericson2314 commented Nov 5, 2019

@basvandijk if it is just used by consumers of release-lib.nix and not release-lib.nix itself, then let's definitely get rid of it. I agree that the underlying issue of getting rid of this Linux assumption is worth fixing, and my change doesn't directly help with that, but at least the Linux assumptions are not obfuscated with tech debt with my change.

@infinisil
Copy link
Member

What's the status of this?

@stale
Copy link

stale bot commented Sep 8, 2020

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 8, 2020
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 3, 2020
@stale
Copy link

stale bot commented Jun 4, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 4, 2021
@wegank wegank marked this pull request as draft March 20, 2024 15:10
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants