Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos-rebuild: init as package #48122

Merged
merged 2 commits into from Oct 16, 2018
Merged

Conversation

zimbatm
Copy link
Member

@zimbatm zimbatm commented Oct 9, 2018

Motivation for this change

Move the nixos-rebuild script from the nixos distribution into the
package set.

This allows non-nixos users to run the script as well. For example,
deploying a remote machine with:

nixos-rebuild --target-host root@hostname --build-host root@hostname

If that idea is interesting, potentially move all the other nixos
commands to packages.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nixos-rebuild

Partial log (click to expand)

copying path '/nix/store/9kipmsxjyjgj5ddlxzqqrs6l9wlmii57-busybox-1.29.3' from 'https://cache.nixos.org'...
copying path '/nix/store/hik4v1b8a8h3mga9g0ghx481kysjmqp5-nix-2.1.3-man' from 'https://cache.nixos.org'...
copying path '/nix/store/ag088hzpkdlzd9lgxvi094bxicvcn32k-libatomic_ops-7.6.6' from 'https://cache.nixos.org'...
copying path '/nix/store/5fvlfkwzfl3a89apvvyaz7652clrcf76-aws-sdk-cpp-1.5.17' from 'https://cache.nixos.org'...
copying path '/nix/store/bwlaq8bvbpyqsfw9a5rgz7wr09rdpikm-curl-7.59.0' from 'https://cache.nixos.org'...
copying path '/nix/store/4ri26b7qgh53s65iqyj50sj4pvknl1qv-libsodium-1.0.16' from 'https://cache.nixos.org'...
copying path '/nix/store/846brfjv2rw3gk5yq744316qpkmk84jl-boehm-gc-7.6.8' from 'https://cache.nixos.org'...
copying path '/nix/store/crg1xmgsyqfcxk0wp5pycrzn32sbwwjh-nix-2.1.3' from 'https://cache.nixos.org'...
building '/nix/store/73qcwchc0aa4f9y35ywdrvy2l13lhhpx-nixos-rebuild.drv'...
/nix/store/q0asgrckwqgk4zl8y2z5v9dl2ws87isy-nixos-rebuild

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: nixos-rebuild

Partial log (click to expand)

these derivations will be built:
  /nix/store/39vrdzsyw5wax59x6rmdk7hk1n0zlxcz-nixos-rebuild.drv
building '/nix/store/39vrdzsyw5wax59x6rmdk7hk1n0zlxcz-nixos-rebuild.drv'...
/nix/store/qfnl0zskb5sj1br41ndvav1pi0xs718p-nixos-rebuild

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: nixos-rebuild

Partial log (click to expand)

  /nix/store/6rf5riyf2yd6db8rcbpl3iq34753na0p-aws-sdk-cpp-1.5.17
  /nix/store/7ncbv7a0z43bh0bhp4bl3r0qy79aq8k9-libsodium-1.0.16
  /nix/store/plwp97zfj51hgr3rv80jkqmwa8mdxv79-nixos-rebuild
  /nix/store/s02dcjkyf7f6584k03cv1v6pman9m1vl-curl-7.59.0
copying path '/nix/store/6rf5riyf2yd6db8rcbpl3iq34753na0p-aws-sdk-cpp-1.5.17' from 'https://cache.nixos.org'...
copying path '/nix/store/s02dcjkyf7f6584k03cv1v6pman9m1vl-curl-7.59.0' from 'https://cache.nixos.org'...
copying path '/nix/store/7ncbv7a0z43bh0bhp4bl3r0qy79aq8k9-libsodium-1.0.16' from 'https://cache.nixos.org'...
copying path '/nix/store/1m6749jbggi50mn6jh0k9893377mhyqi-nix-2.1.3' from 'https://cache.nixos.org'...
copying path '/nix/store/plwp97zfj51hgr3rv80jkqmwa8mdxv79-nixos-rebuild' from 'https://cache.nixos.org'...
/nix/store/plwp97zfj51hgr3rv80jkqmwa8mdxv79-nixos-rebuild

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: nixos-rebuild

Partial log (click to expand)

these derivations will be built:
  /nix/store/39vrdzsyw5wax59x6rmdk7hk1n0zlxcz-nixos-rebuild.drv
these paths will be fetched (1.39 MiB download, 7.12 MiB unpacked):
  /nix/store/lyhsl93f2w3vh7rf6plfvc6agvwzfdn5-nix-2.1.3
  /nix/store/xv9i3861lzjrbmg035465y6y86i33sql-nix-2.1.3-man
copying path '/nix/store/xv9i3861lzjrbmg035465y6y86i33sql-nix-2.1.3-man' from 'https://cache.nixos.org'...
copying path '/nix/store/lyhsl93f2w3vh7rf6plfvc6agvwzfdn5-nix-2.1.3' from 'https://cache.nixos.org'...
building '/nix/store/39vrdzsyw5wax59x6rmdk7hk1n0zlxcz-nixos-rebuild.drv'...
/nix/store/qfnl0zskb5sj1br41ndvav1pi0xs718p-nixos-rebuild

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nixos-rebuild

Partial log (click to expand)

copying path '/nix/store/hik4v1b8a8h3mga9g0ghx481kysjmqp5-nix-2.1.3-man' from 'https://cache.nixos.org'...
copying path '/nix/store/r7j7s6rakammqxrlpcrbyv5x8fdb8p17-brotli-1.0.6-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/ag088hzpkdlzd9lgxvi094bxicvcn32k-libatomic_ops-7.6.6' from 'https://cache.nixos.org'...
copying path '/nix/store/5fvlfkwzfl3a89apvvyaz7652clrcf76-aws-sdk-cpp-1.5.17' from 'https://cache.nixos.org'...
copying path '/nix/store/bwlaq8bvbpyqsfw9a5rgz7wr09rdpikm-curl-7.59.0' from 'https://cache.nixos.org'...
copying path '/nix/store/4ri26b7qgh53s65iqyj50sj4pvknl1qv-libsodium-1.0.16' from 'https://cache.nixos.org'...
copying path '/nix/store/846brfjv2rw3gk5yq744316qpkmk84jl-boehm-gc-7.6.8' from 'https://cache.nixos.org'...
copying path '/nix/store/crg1xmgsyqfcxk0wp5pycrzn32sbwwjh-nix-2.1.3' from 'https://cache.nixos.org'...
building '/nix/store/73qcwchc0aa4f9y35ywdrvy2l13lhhpx-nixos-rebuild.drv'...
/nix/store/q0asgrckwqgk4zl8y2z5v9dl2ws87isy-nixos-rebuild

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: nixos-rebuild

Partial log (click to expand)

/nix/store/plwp97zfj51hgr3rv80jkqmwa8mdxv79-nixos-rebuild

@zimbatm
Copy link
Member Author

zimbatm commented Oct 11, 2018

Seems like it's popular so I moved all the other tools.

There is a new <nixpkgs/.codeName> file so it's accessible from the pkgs side. I think we should introduce a <nixpkgs/version.nix> that holds the structured data but wanted to avoid breaking backward-compatibility.

nixos-rebuild-vms depends on <nixpkgs/nixos/modules/installer/tools/nixos-build-vms/build-vms.nix> which is not ideal in terms of data locality but I wanted to avoid breaking backward-compatibility here.

@zimbatm
Copy link
Member Author

zimbatm commented Oct 11, 2018

@GrahamcOfBorg build nixos-build-vms nixos-enter nixos-generate-config nixos-install nixos-option nixos-rebuild nixos-version

@Mic92
Copy link
Member

Mic92 commented Oct 15, 2018

It seems that one could replace nixops with a shell script for the most part.

@zimbatm
Copy link
Member Author

zimbatm commented Oct 16, 2018

I want to back-port this to 18.09. Did a bunch of tests and it seems to be working well.

Any objections?

Make the codeName globally accessible in the repo. The release is not
only for NixOS anymore.
Move all the nixos-* scripts from the nixos distribution as real
packages in the pkgs/ package set.

This allows non-nixos users to run the script as well. For example,
deploying a remote machine with:

    nixos-rebuild --target-host root@hostname --build-host root@hostname
@zimbatm
Copy link
Member Author

zimbatm commented Oct 16, 2018

@GrahamcOfBorg build nixos-build-vms nixos-enter nixos-generate-config nixos-install nixos-option nixos-rebuild nixos-version

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: nixos-build-vms, nixos-enter, nixos-install, nixos-option, nixos-rebuild, nixos-version

The following builds were skipped because they don't evaluate on x86_64-darwin: nixos-generate-config

Partial log (click to expand)

copying path '/nix/store/m96ck5z05cwfrvnk12f7xj5rxbg6jadl-boehm-gc-7.6.8' from 'https://cache.nixos.org'...
copying path '/nix/store/wy3m25r5b75ihfpyvs48zjkvjdc3yf71-nix-2.1.3' from 'https://cache.nixos.org'...
building '/nix/store/dh65939vxnyq0zgy449w70i4k8lqyakb-nixos-install.drv'...
building '/nix/store/q924pq801rwwvdwnni4n2yiyf1q9mr16-nixos-rebuild.drv'...
/nix/store/q1qqyrvsliw15ddzvlyahm5w2y1qa8ix-nixos-build-vms
/nix/store/ps5wjqagp06s8bcn414lzmk9kdcr61ya-nixos-enter
/nix/store/gzqi4wlhyhlqjv7859fhrrapgpxfd0ws-nixos-install
/nix/store/gkg34xn55817kid22amlwsqmrxwn65d1-nixos-option
/nix/store/h4sdfj4cnvb42qslqpz7wkk4dr849q9x-nixos-rebuild
/nix/store/jcc18iqz600x82z5y03fk1qyqi0ckbfh-nixos-version

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: nixos-build-vms, nixos-enter, nixos-generate-config, nixos-install, nixos-option, nixos-rebuild, nixos-version

Partial log (click to expand)

building '/nix/store/nlv0dfs6f4an6g6ljjafh5rrzg309dbd-nixos-generate-config.drv'...
building '/nix/store/jis1s6j0igj0i6sdjmqrf298c5aah8g8-nixos-install.drv'...
building '/nix/store/7ybs65wkbdd7qp4sx4vihfgzw20zbwkn-nixos-rebuild.drv'...
/nix/store/sj35sb30zwyqc94bsq6l2n0s42r8wzn4-nixos-build-vms
/nix/store/90lv72207g0jiwjvixxgcylrx5f9pzda-nixos-enter
/nix/store/ayrry8s855r04528swjf8f7c8zhlkfdd-nixos-generate-config
/nix/store/g4za42hdpkh8kq2w5v2chfz7g95h99wh-nixos-install
/nix/store/1niica3bk9l0ikn3qj1wmxp0c7ng0zvd-nixos-option
/nix/store/n9s3cawslna0nxv5i4my561r761px38v-nixos-rebuild
/nix/store/7ddj7pg17bjzkgrl7vkvsxz1kljwr4jd-nixos-version

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: nixos-build-vms, nixos-enter, nixos-generate-config, nixos-install, nixos-option, nixos-rebuild, nixos-version

Partial log (click to expand)

copying path '/nix/store/lvl6zx25lhdmbc5sc9kcxq90z3apsm5h-nixos-rebuild' from 'https://cache.nixos.org'...
building '/nix/store/xxs5816zaa8jrl02jrsyqwhbj3qki7g3-nixos-build-vms.drv'...
building '/nix/store/zk27gl8hrryw2wwv22h20644nx3zz13l-nixos-version.drv'...
/nix/store/3pxa66im0zyrw64k2xxjmi6q6krbkqk4-nixos-build-vms
/nix/store/qd6a5wwl4370k60ci2wdqrsi0r0pajm5-nixos-enter
/nix/store/487ch5shz6bypi0saz6vxcwc5qgdfs1n-nixos-generate-config
/nix/store/x0qggva1a37iyi06hd3cp0zmvam3xrxb-nixos-install
/nix/store/0qnz2i2453znqjnd1hqf00nnzk5c2i3c-nixos-option
/nix/store/lvl6zx25lhdmbc5sc9kcxq90z3apsm5h-nixos-rebuild
/nix/store/fqb4m47abinn3qf88kl6hf3k4rssfgqx-nixos-version

@zimbatm
Copy link
Member Author

zimbatm commented Oct 16, 2018

I'll just go ahead and merge this in master. If everything goes well, I will cherry-pick this in 18.09 next Monday.

@zimbatm zimbatm merged commit 10addad into NixOS:master Oct 16, 2018
@zimbatm zimbatm deleted the pkg-nixos-rebuild branch October 16, 2018 17:39
@@ -0,0 +1 @@
Koi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't use camelCase filenames. Also, does this really have to be in the root directory of the repository?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about something like this #48588 ?

@edolstra
Copy link
Member

I'm strongly against this. These are not packages - they should be in the NixOS source tree, not in pkgs.

edolstra added a commit that referenced this pull request Oct 16, 2018
This reverts commit 10addad, reversing
changes made to 7786575.

NixOS scripts should be kept in the NixOS source tree, not in
pkgs. Moving them around is just confusing and creates unnecessary
code/history churn.
@zimbatm zimbatm mentioned this pull request Oct 16, 2018
9 tasks
anpryl added a commit to anpryl/nixpkgs that referenced this pull request Oct 17, 2018
* 'master' of https://github.com/NixOS/nixpkgs: (104 commits)
  python3.pkgs.pyls: 0.2.1 -> 0.3.0
  chromium: fix aarch64 build (NixOS#48586)
  pythonPackages.cligj: 0.4.0 -> 0.5.0 (NixOS#48584)
  Flexget: 2.15.1 -> 2.16.2 (NixOS#48583)
  gitolite 3.6.7 -> 3.6.10 (NixOS#48562)
  oh-my-zsh: 2018-09-03 -> 2018-09-14
  julia_10: update to 1.0.1
  signal-desktop: 1.16.2 -> 1.16.3
  Revert "Merge pull request NixOS#48122 from zimbatm/pkg-nixos-rebuild"
  terraform-providers.libvirt: fix plugin output (NixOS#48577)
  libssh: 0.7.5 -> 0.7.6
  googler: 3.7 -> 3.7.1 (NixOS#48508)
  chromium plugins: add meta.platforms (NixOS#48495)
  varnish6: 6.0.1 -> 6.1.0 (NixOS#48487)
  erlangR21: 21.0 -> 21.1.1 (NixOS#47371)
  varnish4: 4.1.9 -> 4.1.10 (NixOS#48488)
  tmux: 2.7 -> 2.8
  nixpart0.blivet: fix reference to selinux python modules
  blivet: fix reference to selinux python modules
  magit-filenotify: Fix missing dependency on git
  ...
@zimbatm
Copy link
Member Author

zimbatm commented Oct 17, 2018

@edolstra is it a question of data locality, so that the script can evolve at the same pace as NixOS?

Personally I don't tend to make a big distinction between both as they are in the same repo and have the same release schedule. NixOS is just another type of (albeit complex) derivation output.

@Mic92
Copy link
Member

Mic92 commented Oct 19, 2018

I would even say it is a complex derivation.

@Ekleog
Copy link
Member

Ekleog commented Oct 19, 2018

@zimbatm At first I agreed with the idea that it'd be nice to have nixos-rebuild be a regular package, but it's true it makes sense only in the NixOS context, and thus has no meaning on all other systems.

So it'd require, to make sense, to either add a platforms.nixos platform, which sounds overkill… or to live in the nixos/ tree.

Now, maybe a NixOS overlay to nixpkgs could be done, containing these? That'd solve both issues, with both the scripts as actual packages and packages in nixos/, explicitly being marked for NixOS.

@Mic92
Copy link
Member

Mic92 commented Oct 19, 2018

A few use case for non-NixOS users might be:

  • building NixOS VMs with nixos-build-vms
  • Using nixos-enter for nixos containers or existing installations on the same machine
  • Remotely to deploy a nixos system with nixos-rebuild

@zimbatm
Copy link
Member Author

zimbatm commented Oct 19, 2018

I talked with Eelco a bit and his main objection was to move the scripts outside of the nixos folder. It's a data locality problem.

After NixCon I'll do another attempt where the packages are in pkgs/ but leaving the scripts in the nixos/ folder. And only port the ones that have an actual use-case.

@Ekleog
Copy link
Member

Ekleog commented Oct 19, 2018 via email

@Mic92
Copy link
Member

Mic92 commented Oct 19, 2018

I talked with Eelco a bit and his main objection was to move the scripts outside of the nixos folder. It's a data locality problem. After NixCon I'll do another attempt where the packages are in pkgs/ but leaving the scripts in the nixos/ folder. And only port the ones that have an actual use-case.
Hmm… why not a nixos overlay put in the nixos/ folder and enabled by default on NixOS but pick-able on-demand on other systems? That would sound like it'd also solve all the problems caused by nixos-* tools not being packaged, to me

I think that can be achieved by using (nixos{}) already.

@Mic92
Copy link
Member

Mic92 commented Nov 4, 2018

  • easily boostrap nixos from a linux distribution + nix.

@marler8997
Copy link
Contributor

I'm currently installing nix onto my second hard drive from Ubuntu. I got the drive partitioned and the filesystems mounted, but then I was surprised to find that nixos-generate-config was not available as a package...? Since this was removed...what's the recommended way forward for me here to complete the install? Note that I don't have a CD drive.

@Mic92
Copy link
Member

Mic92 commented Mar 25, 2019

If you have USB port/stick, use dd the image to that. You can also use an kexec image that can drop you into a installer from your existing Linux: https://github.com/nix-community/nixos-generators

You can still get nixos-generate-config using:

$(nix-build -E 'with import <nixpkgs> {}; (nixos {}).nixos-generate-config')/bin/nixos-generate-config

@marler8997
Copy link
Contributor

Thanks for the tips, I didn't know about kexec I'll have to learn about that one.

So nixos-generate-config is still in nixpkgs, it's just not a top-level entry anymore. Seems odd to a new Nix user like me. I would think the reason not to put a tool in the top-level attribute set would be if it had a name that could clash...but nixos-generate-config is pretty explicit.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-is-nixos-rebuild-an-output-attribute-of-a-nixos-system/9904/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants