Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ec2/create-amis.sh: register root device as /dev/xvda #67349

Merged
merged 1 commit into from Nov 21, 2019

Conversation

thefloweringash
Copy link
Member

@thefloweringash thefloweringash commented Aug 23, 2019

For the case of blkfront drives, there appears to be no difference
between /dev/sda1 and /dev/xvda: the drive always appears as the
kernel device /dev/xvda.

For the case of nvme drives, the root device typically appears as
/dev/nvme0n1. Amazon provides the 'ec2-utils' package for their first
party linux ("Amazon Linux"), which configures udev to create symlinks
from the provided name to the nvme device name. This name is
communicated through nvme "Identify Controller" response, which can be
inspected with:

nvme id-ctrl --raw-binary /dev/nvme0n1 | cut -c3073-3104 | hexdump -C

On Amazon Linux, where the device is attached as "/dev/xvda", this
creates:

  • /dev/xvda -> nvme0n1
  • /dev/xvda1 -> nvme0n1p1

On NixOS where the device is attach as "/dev/sda1", this creates:

  • /dev/sda1 -> nvme0n1
  • /dev/sda11 -> nvme0n1p1

This is odd, but not inherently a problem.

NixOS unconditionally configures grub to install to /dev/xvda, which
fails on an instance using nvme storage. With the root device name set
to xvda, both blkfront and nvme drives are accessible as /dev/xvda,
either directly or by symlink.

Motivation for this change

Bootloader installation fails on ec2 instances using nvme drives.

$ sudo env NIXOS_INSTALL_BOOTLOADER=1 nixos-rebuild boot
building Nix...
building the system configuration...
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
/nix/store/dw6gx658kl755nfpwhf1fpy8pqmg6n8w-grub-2.02/sbin/grub-install: error: cannot find a GRUB drive for /dev/xvda.  Check your device.map.
/nix/store/b1nwca04xfngbb8w7swn8g4x6g90qxa5-install-grub.pl: installation of GRUB on /dev/xvda failed
warning: error(s) occurred while switching to the new configuration

The root drive on this istance is /dev/nvme0n1.

Depends on #67347 to provide the /dev/xvda symlink.

Results:

[root@ip-172-31-18-196:~]# ls -lh /dev/nvme0* /dev/xvda*
crw------- 1 root root 249, 0 Nov  1 20:47 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Nov  1 20:47 /dev/nvme0n1
brw-rw---- 1 root disk 259, 2 Nov  1 20:47 /dev/nvme0n1p1
lrwxrwxrwx 1 root root      7 Nov  1 20:47 /dev/xvda -> nvme0n1
lrwxrwxrwx 1 root root      9 Nov  1 20:47 /dev/xvda1 -> nvme0n1p1

[root@ip-172-31-18-196:~]# NIXOS_INSTALL_BOOTLOADER=1 /run/current-system/bin/switch-to-configuration boot
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
Installation finished. No error reported.
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@tomberek
Copy link
Contributor

What is needed to test or make progress for this issue?

@thefloweringash
Copy link
Member Author

What is needed to test or make progress for this issue?

Since this depends on #67347, we need that to progress first.

For the case of blkfront drives, there appears to be no difference
between /dev/sda1 and /dev/xvda: the drive always appears as the
kernel device /dev/xvda.

For the case of nvme drives, the root device typically appears as
/dev/nvme0n1.  Amazon provides the 'ec2-utils' package for their first
party linux ("Amazon Linux"), which configures udev to create symlinks
from the provided name to the nvme device name. This name is
communicated through nvme "Identify Controller" response, which can be
inspected with:

  nvme id-ctrl --raw-binary /dev/nvme0n1 | cut -c3073-3104 | hexdump -C

On Amazon Linux, where the device is attached as "/dev/xvda", this
creates:

- /dev/xvda  -> nvme0n1
- /dev/xvda1 -> nvme0n1p1

On NixOS where the device is attach as "/dev/sda1", this creates:

- /dev/sda1  -> nvme0n1
- /dev/sda11 -> nvme0n1p1

This is odd, but not inherently a problem.

NixOS unconditionally configures grub to install to `/dev/xvda`, which
fails on an instance using nvme storage. With the root device name set
to xvda, both blkfront and nvme drives are accessible as /dev/xvda,
either directly or by symlink.
@thefloweringash thefloweringash marked this pull request as ready for review November 1, 2019 20:59
@mkaito
Copy link
Contributor

mkaito commented Nov 21, 2019

Hi, any news?

@domenkozar domenkozar merged commit a47e801 into NixOS:master Nov 21, 2019
@thefloweringash thefloweringash deleted the amazon-device-names branch November 21, 2019 17:23
@mkaito
Copy link
Contributor

mkaito commented Nov 22, 2019

@domenkozar thanks! does this push new AMIs automatically?

@domenkozar
Copy link
Member

No, (I think) only @edolstra can do that.

@mkaito
Copy link
Contributor

mkaito commented Nov 27, 2019

Well, from master I suppose it'll end up in the 20.03 stable image, so it might be a while.

thefloweringash added a commit to thefloweringash/nixos-factorio-server that referenced this pull request May 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants