Amazon Images for Aarch64 #62042

thefloweringash · 2019-05-25T10:00:24Z

Motivation for this change

AMIs for Amazon's A1 instance type. Following advice from #nixos-dev, this adds a hydra job to build the images, and a script that will upload and register the prebuilt images given the store path. The uploader does not require a configured aarch64 builder.

While rewriting the create-amis.sh script I only used the awscli package. I haven't seen it written anywhere, but it seems like the ec2-* tools do not support current AWS features like session tokens and assume-role based access to s3. I find this version more compatible with my authentication scheme (short lived tokens), but recognize this is a large change from the existing behavior.

Configuration details adapted from #52779

Tested booting both "arm64" and "x86_64" images produced from nix-build nixos/release.nix -A amazon_image.

Things done

thefloweringash · 2019-05-25T10:01:21Z

nixos/maintainers/scripts/ec2/amazon-image.nix

@@ -51,7 +51,9 @@ in {
    inherit lib config;
    inherit (cfg) contents format name;
    pkgs = import ../../../.. { inherit (pkgs) system; }; # ensure we use the regular qemu-kvm package
-    partitionTableType = if config.ec2.hvm then "legacy" else "none";
+    partitionTableType = if config.ec2.efi then "efi"
+                         else if config.ec2.hvm then "legacy"


ec2.hvm is blocked by an assertion, but the implementation still seems to support it. I preserved the existing behavior.

thefloweringash · 2019-05-25T10:04:18Z

nixos/maintainers/scripts/ec2/amazon-image.nix

@@ -61,6 +63,9 @@ in {
          ${optionalString config.ec2.hvm ''
            ec2.hvm = true;
          ''}
+          ${optionalString config.ec2.efi ''
+            ec2.efi = true;


Implemented in the same pattern as ec2.hvm, but I don't know the reason for this.

thefloweringash · 2019-05-25T10:08:38Z

nixos/modules/virtualisation/amazon-image.nix

@@ -25,6 +25,9 @@ in
      { assertion = cfg.hvm;
        message = "Paravirtualized EC2 instances are no longer supported.";
      }
+      { assertion = cfg.efi -> cfg.hvm;


I don't know if this is a strict requirement, but it reduces the possible configurations to support.

memberbetty · 2019-06-01T08:37:12Z

If you can add some documentation too, which would make it possible to use this in 10 minutes, this is the kind of thing that would make me want to run master.

Great to see arm support coming, regardless.

thefloweringash · 2019-06-04T15:14:06Z

nixos/maintainers/scripts/ec2/create-amis.sh

+         us-east-1 us-east-2 us-west-1 us-west-2
+         ca-central-1
+         ap-southeast-1 ap-southeast-2 ap-northeast-1 ap-northeast-2
+         ap-south-1 ap-east-1


ap-east-1 is new here, and may required additional configuration to enable this region

I don't think we have to do any region-specific configuration.

Some regions can be individually disabled and enabled, see the Amazon docs on Managing AWS Regions. I think ap-east-1 is the only region that this applies to, and is disabled by default.

Yeah, looks like ap-east-1 and me-south-1 are currently disabled.

thefloweringash · 2019-06-04T15:14:31Z

nixos/maintainers/scripts/ec2/create-amis.sh

+
+# configuration
+state_dir=/home/deploy/amis/ec2-images
+home_region=us-west-2


I can't tell the home region of the nixos-amis bucket, this is probably wrong.

It's in eu-west-1 (or EU if it needs the old S3 location constraint).

edolstra · 2019-06-04T16:25:02Z

nixos/release.nix

@@ -196,6 +196,22 @@ in rec {
  );


+  # A disk image that can be imported to Amazon EC2 and registered as an AMI
+  amazon_image = forMatchingSystems [ "x86_64-linux" "aarch64-linux" ] (system:


amazon_image -> amazonImage.

edolstra · 2019-06-04T16:27:27Z

nixos/maintainers/scripts/ec2/amazon-image.nix

@@ -42,7 +42,7 @@ in {

    format = mkOption {
      type = types.enum [ "raw" "qcow2" "vpc" ];
-      default = "qcow2";
+      default = "vpc";


Is vpc a sparse / compressed disk format? For Hydra it's important not to increase the size of the images a lot.

I believe it's sparse but not compressed, comparisons:

└── [ 1193577472] nixos-amazon-image-19.09pre130979.gfedcba-aarch64-linux.vhd └── [ 1177550848] nixos-amazon-image-19.09pre130979.gfedcba-aarch64-linux.qcow2 └── [ 1132745216] nixos-amazon-image-19.09pre130979.gfedcba-x86_64-linux.vhd └── [ 1116078080] nixos-amazon-image-19.09pre130979.gfedcba-x86_64-linux.qcow2

edolstra · 2019-09-04T15:33:31Z

@thefloweringash Can you resolve the merge conflict? Then I'll merge this. Thanks!

These can be imported without converison.

thefloweringash · 2019-09-04T15:55:11Z

Rebased on current master. I haven't tested it since the rebase, but the rebase itself was straightforward.

thefloweringash · 2019-09-04T17:06:10Z

I've now successfully tested booting both amd64 and aarch64 AMIs on current master (after fixing the ceph entry in the release notes).

kirelagin · 2019-10-25T14:11:02Z

I’m trying to use this image and I am importing it manually (since there seem to be no images on the nixos.org website currently), however the import-image task fails with

"StatusMessage": "ClientError: EFI partition detected. UEFI booting is not supported in EC2.",

🤔

kirelagin · 2019-10-25T14:56:58Z

Hm, ok, apparently EFI is specifically for aarch64, and one has to pass --architecture arm64 to import-image, however there is probably something wrong on their end:

An error occurred (InvalidParameter) when calling the ImportImage operation: Parameter architecture=arm64 has an invalid format.

thefloweringash · 2019-11-01T21:50:25Z

I am importing it manually

one has to pass --architecture arm64 to import-image,

Are you using nixos/maintainers/scripts/ec2/create-amis.sh? It should handle this for you. It's not the best interface, but you can modify the top of script to configure your bucket and regions in order to run it locally.

I've created an aarch64 ami with create-amis.sh and successfully booted an a1.medium on it. If this is still failing for you, can you share more details so I can try to reproduce it?

thefloweringash commented May 25, 2019

View reviewed changes

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels May 25, 2019

thefloweringash force-pushed the aarch64-ami branch from 6174864 to 83b54f9 Compare May 25, 2019 11:40

samueldr requested a review from edolstra May 31, 2019 15:59

thefloweringash force-pushed the aarch64-ami branch from 83b54f9 to fb44e07 Compare May 31, 2019 17:03

thefloweringash changed the title ~~amazon-image.nix: add EFI support, enable by default for aarch64~~ Amazon Images for Aarch64 May 31, 2019

thefloweringash commented Jun 4, 2019

View reviewed changes

thefloweringash force-pushed the aarch64-ami branch 2 times, most recently from fe34388 to 65e0249 Compare June 4, 2019 15:49

edolstra reviewed Jun 4, 2019

View reviewed changes

edolstra approved these changes Jun 4, 2019

View reviewed changes

thefloweringash force-pushed the aarch64-ami branch 2 times, most recently from d82bec8 to ec7d1db Compare June 4, 2019 16:47

thefloweringash mentioned this pull request Aug 20, 2019

ec2: update create-amis script #67088

Closed

10 tasks

thefloweringash added 5 commits September 5, 2019 00:52

amazon-image.nix: add EFI support, enable by default for aarch64

5501274

amazon-image.nix: default to vpc formatted images

027e556

These can be imported without converison.

nixos/release.nix: add "amazonImage" for building Amazon EC2 Images

9898fd0

amazon-image.nix: add hydra-build-products and improve metadata

25bee97

amazon-image.nix: upload prebuilt images

84742e2

thefloweringash force-pushed the aarch64-ami branch from ec7d1db to 84742e2 Compare September 4, 2019 15:53

edolstra merged commit 68801b4 into NixOS:master Sep 4, 2019

thefloweringash deleted the aarch64-ami branch September 4, 2019 17:01

thefloweringash mentioned this pull request Jan 28, 2020

nixos/tests/ec2: return to passing state #77570

Merged

10 tasks

thefloweringash mentioned this pull request Nov 4, 2020

AWS ARM A1 instances #52779

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Amazon Images for Aarch64 #62042

Amazon Images for Aarch64 #62042

thefloweringash commented May 25, 2019 •

edited

thefloweringash May 25, 2019

thefloweringash May 25, 2019

thefloweringash May 25, 2019

memberbetty commented Jun 1, 2019

thefloweringash Jun 4, 2019

edolstra Jun 4, 2019

thefloweringash Jun 4, 2019

edolstra Sep 4, 2019

thefloweringash Jun 4, 2019

edolstra Jun 4, 2019

edolstra Jun 4, 2019

edolstra Jun 4, 2019

thefloweringash Jun 4, 2019

edolstra commented Sep 4, 2019

thefloweringash commented Sep 4, 2019

thefloweringash commented Sep 4, 2019 •

edited

kirelagin commented Oct 25, 2019

kirelagin commented Oct 25, 2019

thefloweringash commented Nov 1, 2019

Amazon Images for Aarch64 #62042

Amazon Images for Aarch64 #62042

Conversation

thefloweringash commented May 25, 2019 • edited

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

memberbetty commented Jun 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edolstra commented Sep 4, 2019

thefloweringash commented Sep 4, 2019

thefloweringash commented Sep 4, 2019 • edited

kirelagin commented Oct 25, 2019

kirelagin commented Oct 25, 2019

thefloweringash commented Nov 1, 2019

thefloweringash commented May 25, 2019 •

edited

thefloweringash commented Sep 4, 2019 •

edited