AWS EC2 Windows 10 can't access metadata

Question

What could explain why an EC2 instance running Windows 10 does not consistently have access to its own metadata or userdata?

I know the userdata is set correctly because the exact same script was used for about thirty launches of t2.nano and c4.xlarge instances: the t2.nano never encountered any issue reading the metadata, but out of three attempts with a c4.xlarge only one had access to it. The script only differed by the name of instance (as per git history at least).

I followed the instructions below, and even from a Powershell, the Uri fails to load (cf. Figure 2).

Any hint is appreciated.

Does it work if you try to retrieve a normal web page from the Internet? — John Rotenstein, Jul 14 '17 at 05:43
You're on Windows 10? Did you bring your own image across from a PC? Do you have SSM installed? I've been told there's some strangeness with Windows 2016 if you change subnets -- try running the `InitializeInstance.ps1` script to fix. — John Rotenstein, Jul 14 '17 at 08:34
@JohnRotenstein thanks, that works! Seems like you can also run the script ProgramData/Amazon/EC2-Windows/Launch/Module/Scripts/Add-Routes.ps1 in order to only start the routes. The full script will also reset the password. It you post your comment as an answer, I can accept it as the correct answer. Cheers. — ChrisR, Jul 14 '17 at 16:33

score 19 · Accepted Answer · answered Jul 15 '17 at 08:27

19

There is a script call InitializeInstance.ps1 that resets some configuration information.

For example, if the instance has changed subnets it might not work correctly due to cached routing rules. The InitializeInstance.ps1 can correct this.

answered Jul 15 '17 at 08:27

John Rotenstein

241,921
22
380
470

5

In Windows Instance, this script is located here: C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 – probitaille Mar 14 '19 at 17:03
1

Does this script have any side effects? – Erik Maldonado Jul 24 '20 at 20:15
no side effects, it just continually corrects the metadata, and it also tries to upsize your disk part, for example if you have increased the vol size in aws panel, to more than the part is, otherwise it will do nothing there. so no side effects unless your vol isnt sized all the way. so normally it does nothing there, and just fixes the metadata routes – blamb Jul 14 '23 at 19:10

score 7 · Answer 2 · answered Jun 28 '18 at 11:51

We experienced the same issue on a Windows 2016 server on EC2. We noticed that the default gateway on the 169 IPs routes (persistent) where pointing at a non-existing (old?) gateway IP.

We changed the routes to the default gateway of the primary adapter, after that Instance Metadata started to work and AmazonSSMAgent service is running again.

Old situation:

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     172.16.3.129     172.16.3.152     15
     ....
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
  169.254.169.254  255.255.255.255     172.16.0.129      15
  169.254.169.250  255.255.255.255     172.16.0.129      15
  169.254.169.251  255.255.255.255     172.16.0.129      15

Notice the gateway on the persistent routes for 169 to point at an IP, which is not the default for 0.0.0.0. This 172.16.0.129 is also not pingable.

After changing the routes using route CHANGE:

route CHANGE 169.254.169.254 MASK 255.255.255.255  172.16.3.129 METRIC 15 IF 4 /P
route CHANGE 169.254.169.250 MASK 255.255.255.255  172.16.3.129 METRIC 15 IF 4 /P
route CHANGE 169.254.169.251 MASK 255.255.255.255  172.16.2.129 METRIC 15 IF 4 /P

Where:

172.16.3.129 is the default gateway on the primary network interface. This will be different on each instance.
And 4 at then end METRIC 15 IF 4 is the interface ID of primary adapter, listed in interface list on route PRINT, this could also be different on each instance.

We now have:

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     172.16.3.129     172.16.3.152     15
....
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
  169.254.169.254  255.255.255.255     172.16.3.129      15
  169.254.169.250  255.255.255.255     172.16.3.129      15
  169.254.169.251  255.255.255.255     172.16.3.129      15
===========================================================================

This is basically what the for mentioned ProgramData/Amazon/EC2-Windows/Launch/Module/Scripts/Add-Routes.ps1 script does.

We just had a similar issue. We believe it was because we took an AMI of an instance in the default VPC and launched it in a new one. The assumption is that the routes were not recreated on start up. — jackofallcode, May 30 '19 at 10:04

score 4 · Answer 3 · answered Feb 11 '21 at 11:49

Can't edit John Rotenstein so I'll add it here,

This issue was solved at my end by initialise the instance again, it happen when you created an image in one subnet or vpc and launch it in a different one.

Warning this would change the admin password, make sure you have access to the required key to get the new password at console.

In order to initialise the instance only once run:

C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule

in order to initialise it every boot run

C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -SchedulePerBoot

the docs: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html

after running it, reboot the instance, get a new password from the console, rdp inside and you can get metadata and use aws s3 cli / powershell with the attached instance IAM role

blamb · Answer 4 · 2023-07-14T19:22:06.330

A better way to do this would be to run AWS's Add-Routes function. But in order to do that, you need to import it first.

You can import it and run it on a one liner like this from powershell on your instance that is not getting the metadata accessible;

Import-Module c:\ProgramData\Amazon\EC2-Windows\Launch\Module\Ec2Launch.psm1 ; Add-Routes

This Add-Routes function is what the InitializeInstance.ps1 script runs to fix the metadata.

FYI, I experience this issue all the time on Windows 2019 Server, and feel like its an issue on AWS side where sometimes instances spin up and don't complete the cloud init always. This is a much used command for ASG groups, etc..

The benefit of this over the @John Rotenstein answer, is that it will run only the routes, will not run the Initialize disks command, so will finish quicker.

One further note that may be good to add to this Post, is that there is two ways to get the instance metadata, the IMDSv1 method mentioned above may not work if your using IMDSv2. The proper way to get the metadata for IMDSv2 is

PS C:\> [string]$token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} -Method PUT -Uri http://169.254.169.254/latest/api/token

PS C:\> Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} -Method GET -Uri http://169.254.169.254/latest/meta-data/

Docs on that are here

AWS EC2 Windows 10 can't access metadata

4 Answers4

Linked