0

Trying to run few steps of CI/CD in a EC2 instance. Please don't ask for reasons.

Need to: 1) Start an instance using AWS CLI. Set few environment variables. 2) Run few bash commands. 3) Stream the command from the above commands into the console of the caller script. 4) If any of the commands fail, need to fail the calling script as well. 5) Terminate the instance.

rubenhak
  • 830
  • 2
  • 10
  • 28
  • What exactly is the question here? Which AWS service to use? How the instance's user data should look like? Where to stream the logs, e.g. to CloudWatch or to the script which starts the EC2 instance? Do you look for a cli script which does all the stuff such as `aws ec2 run-instances`? – Martin Löper Jul 18 '19 at 21:31
  • I want my script to start with "aws ec2 run ..." and trigger few commands. I also want to know the success/fail status of those commands running in EC2 in my script. Also need to stream the stdout from EC2 into the script that starts the EC2 instance – rubenhak Jul 18 '19 at 22:28

2 Answers2

2

There is a SO thread which indicates that streaming the output is not as easy. [1] What I would do, if I had to implement this task:

  • Start the instance using the cli command aws ec2 run-instances and using an AMI which has the AWS SSM agent preinstalled. [2]

  • Run your commands using AWS SSM. [3] This has the benefit that you can run any number of commands you want - whenever you want (i.e. the commands must not be specified at instance launch, but can be chosen afterwards). You also get the status code of each command.[4]

  • Use the CloudWatch integration in SSM to stream your command output to CloudWatch logs. [5]

  • Stream the logs from CloudWatch to your own instance. [6]

Note: Instead of streaming the command output via CloudWatch, you could also periodically poll the SSM API by using aws ssm get-command-invocation. [7]

Reference

[1] How to check whether my user data passing to EC2 instance working or not?
[2] Working with SSM Agent - AWS Systems Manager
[3] Walkthrough: Use the AWS CLI with Run Command - AWS Systems Manager
[4] Understanding Command Statuses - AWS Systems Manager
[5] Streaming AWS Systems Manager Run Command output to Amazon CloudWatch Logs | AWS Management Tools Blog
[6] how to view aws log real time (like tail -f)
[7] get-command-invocation — AWS CLI 1.16.200 Command Reference

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Martin Löper
  • 6,471
  • 1
  • 16
  • 40
0

Approach 1.

Start an instance using AWS CLI.

 aws ec2 start-instances --instance-ids i-1234567890abcdef0

Set few environment variables.

Use user dat of ec2 to set env. & run commands

.. Run your other logic / scripts

To terminate the instance run below command in the same instance.

instanceid=`curl http://169.254.169.254/latest/meta-data/instance-id`
aws ec2 terminate-instances --instance-ids $instanceid

Approach 2.

Use python boto3 or kitchen chef ci.
mahendra rathod
  • 1,438
  • 2
  • 15
  • 23