Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ec2-metadata-fetcher: use IMDSv2, fetch public-ipv4 as well #96593

Closed
wants to merge 4 commits into from
Closed

ec2-metadata-fetcher: use IMDSv2, fetch public-ipv4 as well #96593

wants to merge 4 commits into from

Conversation

endgame
Copy link
Contributor

@endgame endgame commented Aug 29, 2020 •
edited

I'd love to have this change make it to the 20.09 NixOS AMIs.

Motivation for this change
  • Version 2 of the Instance MetaData Service (IMDS) is the new recommended way of fetching instance metadata on AWS: https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/ and means NixOS AMIs can work in environments that are migrating away from permitting IMDSv1.
  • I have a server in a private config that wants to know its external IPv4 address, otherwise it doesn't work properly. I intend to use a readFile /etc/ec2-metadata/public-ipv4 in the configuration I push as instance userdata, so that I can launch EC2 instances running this server.
  • Instances can change user data, public IPv4 address &c. over time, so it's correct to repopulate /etc/ec2-metadata on each boot.
Things done
  • Tested that the changes produce a working image, by:

@endgame
Copy link
Contributor Author

endgame commented Aug 29, 2020 •
edited

Couple of things that need to happen:

  • https://docs.openstack.org/nova/ussuri/user/metadata.html#metadata-ec2-format The openstack module uses the metadata fetcher. We can back up to version 2009-04-04 of IMDS and probably remain compatible.
  • We need to see how openstack reacts to the fetcher trying to get an IMDSv2 token. Hopefully it just ignores the PUT. I'll need someone who has access to an openstack setup to help me test. EDIT: Let's just make the "get a V2 token" optional instead.

@endgame endgame requested a review from nlewo August 29, 2020 11:35
@endgame
Copy link
Contributor Author

endgame commented Aug 29, 2020

https://docs.openstack.org/nova/ussuri/user/metadata.html#metadata-ec2-format note for my reference

@endgame endgame marked this pull request as draft August 30, 2020 13:20
@endgame endgame marked this pull request as ready for review August 31, 2020 01:15
@endgame
Copy link
Contributor Author

endgame commented Aug 31, 2020

Okay, this is ready now. I have tested the images by building amazon-image.nix both with and without the useImdsToken parameter, and importing the resulting images using AWS VM Import/Export to use a custom AMI.

Haven't done any OpenStack testing, but I reworked the metadata fetcher to use an IMDS API version that's documented as compatible with OpenStack compute. (The original version in this file — 1.0 — didn't have the public-ipv4 option.)

@copumpkin FYI since I've seen your name on a bunch of cloud stuff in this part of nixpkgs.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-in-distress/3604/34

@jonringer
Copy link
Contributor

@garbas I know you had an interest in supporting ec2 instances

@endgame
Copy link
Contributor Author

endgame commented Oct 16, 2020 •
edited

I have found it completely impossible to test the openstack side of this: I can't find a provider that will give me openstack access at an affordable price, asking in #nixos or the PR-in-distress thread never found someone who had the resources to help, and the openstack "devstack" script never gave me a working stack to test with. On this front, I really feel that I've done my due diligence, and that nobody seems to be using this feature anyway.

I have done the next best thing I could think of: I built EC2 images with useImdsToken = false to generate the IMDS fetcher script as it will run when baked into an openstack image, and made sure that the metadata fetcher script continues to work. The IMDS version the script uses is explicitly documented as compatible in the openstack docs.

I am curious to see what will happen if this script runs on an EC2 instance with no public IPv4 address (i.e., isolated VPC), but my AWS-fu is not good enough to set that up yet.

@garbas
Copy link
Member

garbas commented Oct 19, 2020

@jonringer I'm interested in automated the upload of the images :) NixOS/nixos-homepage#447

@endgame
ec2-metadata-fetcher: use IMDSv2, fetch public-ipv4 as well
832df8f
@endgame
ec2-metadata-fetcher: conditionally use IMDSv2 token
c3bd027 
This allows us to retain OpenStack compatiblity.
@endgame
ec2-metadata-fetcher: Unconditionally download from IMDS
d725a83 
Busybox wget will overwrite the files if they exist, and re-fetching
user-data and meta-data avoids surprising behaviour when an instance
references stale data on reboot.

Example: Without this commit, it's possible to:

1. Create an EC2 image with configuration in user data;
2. Stop that image;
3. Change the user data in the EC2 Management Console;
4. Restart the image;
5. Note that the original user data is re-applied, not the new user data.
@endgame
ec2-metadata-fetcher: call wget using shell function
c733138 
This dodges a mysterious quoting problem that stops requests that use
the IMDSv2 token from parsing properly.
@endgame
Copy link
Contributor Author

endgame commented Nov 16, 2020

Okay, I have learned enough about VPCs to do the test I wanted. I built a nixos AMI with the changes in this PR, booted that AMI into a private subnet. The fetch of http://169.254.169.254/meta-data/public-ipv4 404s, and doesn't create /etc/ec2-metadata/public-ipv4. Which seems fine.

Other things from the metadata fetcher work (e.g., I can SSH in with the keypair I tell AWS to use).

Given that the openstack side of things is impossible to test (see above), I'm now calling this ready-to-merge. What else do I need to do?

@jonringer
Copy link
Contributor

@AmineChikhaoui do you mind taking a look at this?

endgame
endgame commented Nov 17, 2020
View reviewed changes
nixos/modules/virtualisation/ec2-metadata-fetcher.nix
''
imds=http://169.254.169.254/2009-04-04
metaDir=${targetRoot}etc/ec2-metadata
mkdir -m 0755 -p "$metaDir"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I be doing a rm -f $metaDir/* here? Otherwise, files that might exist in certain conditions only (e.g. /etc/ec2-metadata/public-ipv4) might persist when they should be removed (if the public ipv4 goes away for some reason).

@AmineChikhaoui AmineChikhaoui mentioned this pull request Nov 19, 2020
Merged
10 tasks
@endgame
Copy link
Contributor Author

endgame commented Nov 20, 2020

Closing as this was resolved by #104193.

@endgame endgame closed this Nov 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nlewo nlewo Awaiting requested review from nlewo

Assignees
No one assigned
Labels
6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@endgame @nixos-discourse @jonringer @garbas