0

The UserData section of my EC2Instance resource has the following:

#!/bin/bash
apt-get update -y
apt-get -y install python3-pip
mkdir -p /opt/aws/
pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz
ln -s /usr/local/init/ubuntu/cfn-hup /etc/init.d/cfn-hup
/usr/local/bin/cfn-init -v --stack ${AWS::StackId} --resource EC2Instance --configsets server_install --region ${AWS::Region}
/usr/local/bin/cfn-signal -e $? --stack ${AWS::StackId} --resource EC2Instance --region ${AWS::Region}

While running the stack creation process, I noticed that the cfn-init and the cfn-signal commands failed to run because my EC2 didn't have any of the packages I installed through AWS::Cloudformation:Init.

I logged on to the server to check things out (NOTE: logging on to the server after a failed stack build is only possible if you don't revert stack upon failure or don't signal for failures) and I found the following errors:

/var/log/cloud-init.log:

2023-04-12 21:17:17,121 - subp.py[DEBUG]: Running command 
['/var/lib/cloud/instance/scripts/part-001'] with allowed return codes [0] (shell=False, capture=False)
2023-04-12 21:28:34,625 - subp.py[DEBUG]: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/part-001']
Exit code: 1
Reason: -
Stdout: -
Stderr: -
2023-04-12 21:28:34,629 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)

/var/lib/cloud/instance/scripts/part-001: Same UserData section from CF template

/var/log/cloud-init-output.log:

2023-04-12 21:18:06,101 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com
2023-04-12 21:18:06,102 [DEBUG] Describing resource EC2Instance in stack arn:aws:cloudformation:us-east-1:<accountId>:stack/<app-name>/<uuid>
2023-04-12 21:19:06,102 [WARNING] Timeout of 60 seconds breached
2023-04-12 21:19:06,103 [ERROR] Client-side timeout
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/cfnbootstrap/util.py", line 193, in _retry
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/cfnbootstrap/util.py", line 266, in _timeout
    raise TimeoutError(
cfnbootstrap.util.TimeoutError


2023-04-12 21:18:06,101 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com
2023-04-12 21:18:06,102 [DEBUG] Signlaing resource EC2Instance in stack arn:aws:cloudformation:us-east-1:<accountId>:stack/<app-name>/<uuid>
2023-04-12 21:19:06,102 [WARNING] Timeout of 60 seconds breached
2023-04-12 21:19:06,103 [ERROR] Client-side timeout
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/cfnbootstrap/util.py", line 193, in _retry
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/cfnbootstrap/util.py", line 266, in _timeout
    raise TimeoutError(
cfnbootstrap.util.TimeoutError
neuquen
  • 3,991
  • 15
  • 58
  • 78

1 Answers1

1

I saw a ton of other answers (on Stackoverflow and elsewhere) that stated that the EC2 instance needed use a VPC with a public subnet. It needs to be public because the cfn-init script reaches out to the Cloudformation API through the public internet.

I validated that my VPC subnet was actually public by following these steps. I also validated that my instance was connected to the internet by following these steps.

However, I noticed that when I tried to curl the cloudformation API from the EC2 instance that it was timing out:

curl -v https://cloudformation.us-east-1.amazonaws.com

But, I then noticed that if I curled a different region it worked fine. So this worked:

curl -v https://cloudformation.us-east-2.amazonaws.com

Curling different services in the same region also worked:

curl -v https://ec2.us-east-1.amazonaws.com

All of this was perplexing, and it brought me to this question and answer.

After that, I realized that when I curled the two Cloudformation APIs from the different regions, that the IP address on us-east-1 was private (192.168.x.x:443), whereas the IP address from us-east-2 was public (52.95.16.95:443).

I then dug into my VPC a little bit more, and found that in addition to the public subnet, there were VPC interface endpoints created, including one that was providing access to cloudformation.us-east-1.amazonaws.com. There was a security group attached to that VPC interface endpoint, and I realized that it was too restrictive. Once I updated the inbound rules on that SG, my calls from the EC2 instance to the Cloudformation API started working. Problem solved.

Endnote: I noticed in the troubleshooting steps from https://repost.aws that it said the following:

If you're using an interface VPC endpoint, then the endpoint must be located in the same AWS Region as the instance. Also, the security group that's attached to the interface endpoint must allow incoming connections on port 443 from the private subnet of the Amazon VPC.

But unfortunately, I didn't realize an interface endpoint was being used until later...

neuquen
  • 3,991
  • 15
  • 58
  • 78