How do I make cloud-init startup scripts run every time my EC2 instance boots?

Question

I have an EC2 instance running an AMI based on the Amazon Linux AMI. Like all such AMIs, it supports the cloud-init system for running startup scripts based on the User Data passed into every instance. In this particular case, my User Data input happens to be an Include file that sources several other startup scripts:

#include
http://s3.amazonaws.com/path/to/script/1
http://s3.amazonaws.com/path/to/script/2

The first time I boot my instance, the cloud-init startup script runs correctly. However, if I do a soft reboot of the instance (by running sudo shutdown -r now, for instance), the instance comes back up without running the startup script the second time around. If I go into the system logs, I can see:

Running cloud-init user-scripts
user-scripts already ran once-per-instance
[  OK  ]

This is not what I want -- I can see the utility of having startup scripts that only run once per instance lifetime, but in my case these should run every time the instance starts up, like normal startup scripts.

I realize that one possible solution is to manually have my scripts insert themselves into rc.local after running the first time. This seems burdensome, however, since the cloud-init and rc.d environments are subtly different and I would now have to debug scripts on first launch and all subsequent launches separately.

Does anyone know how I can tell cloud-init to always run my scripts? This certainly sounds like something the designers of cloud-init would have considered.

Answer 1

In 11.10, 12.04 and later, you can achieve this by making the 'scripts-user' run 'always'. In /etc/cloud/cloud.cfg you'll see something like:

cloud_final_modules:
 - rightscale_userdata
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - keys-to-console
 - phone-home
 - final-message

This can be modified after boot, or cloud-config data overriding this stanza can be inserted via user-data. Ie, in user-data you can provide:

#cloud-config
cloud_final_modules:
 - rightscale_userdata
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - [scripts-user, always]
 - keys-to-console
 - phone-home
 - final-message

That can also be '#included' as you've done in your description. Unfortunately, right now, you cannot modify the 'cloud_final_modules', but only override it. I hope to add the ability to modify config sections at some point.

There is a bit more information on this in the cloud-config doc at https://github.com/canonical/cloud-init/tree/master/doc/examples

Alternatively, you can put files in /var/lib/cloud/scripts/per-boot , and they'll be run by the 'scripts-per-boot' path.

Answer 2

In /etc/init.d/cloud-init-user-scripts, edit this line:

/usr/bin/cloud-init-run-module once-per-instance user-scripts execute run-parts ${SCRIPT_DIR} >/dev/null && success || failure

to

 /usr/bin/cloud-init-run-module always user-scripts execute run-parts ${SCRIPT_DIR} >/dev/null && success || failure

Good luck !

Answer 3

cloud-init supports this now natively, see runcmd vs bootcmd command descriptions in the documentation (http://cloudinit.readthedocs.io/en/latest/topics/examples.html#run-commands-on-first-boot):

"runcmd":

#cloud-config

# run commands
# default: none
# runcmd contains a list of either lists or a string
# each item will be executed in order at rc.local like level with
# output to the console
# - runcmd only runs during the first boot
# - if the item is a list, the items will be properly executed as if
#   passed to execve(3) (with the first arg as the command).
# - if the item is a string, it will be simply written to the file and
#   will be interpreted by 'sh'
#
# Note, that the list has to be proper yaml, so you have to quote
# any characters yaml would eat (':' can be problematic)
runcmd:
 - [ ls, -l, / ]
 - [ sh, -xc, "echo $(date) ': hello world!'" ]
 - [ sh, -c, echo "=========hello world'=========" ]
 - ls -l /root
 - [ wget, "http://slashdot.org", -O, /tmp/index.html ]

"bootcmd":

#cloud-config

# boot commands
# default: none
# this is very similar to runcmd, but commands run very early
# in the boot process, only slightly after a 'boothook' would run.
# bootcmd should really only be used for things that could not be
# done later in the boot process.  bootcmd is very much like
# boothook, but possibly with more friendly.
# - bootcmd will run on every boot
# - the INSTANCE_ID variable will be set to the current instance id.
# - you can use 'cloud-init-per' command to help only run once
bootcmd:
 - echo 192.168.1.130 us.archive.ubuntu.com >> /etc/hosts
 - [ cloud-init-per, once, mymkfs, mkfs, /dev/vdb ]

also note the "cloud-init-per" command example in bootcmd. From it's help:

Usage: cloud-init-per frequency name cmd [ arg1 [ arg2 [ ... ] ]
   run cmd with arguments provided.

   This utility can make it easier to use boothooks or bootcmd
   on a per "once" or "always" basis.

   If frequency is:
      * once: run only once (do not re-run for new instance-id)
      * instance: run only the first boot for a given instance-id
      * always: run every boot

Answer 4

One possibility, although somewhat hackish, is to delete the lock file that cloud-init uses to determine whether or not the user-script has already run. In my case (Amazon Linux AMI), this lock file is located in /var/lib/cloud/sem/ and is named user-scripts.i-7f3f1d11 (the hash part at the end changes every boot). Therefore, the following user-data script added to the end of the Include file will do the trick:

#!/bin/sh
rm /var/lib/cloud/sem/user-scripts.*

I'm not sure if this will have any adverse effects on anything else, but it has worked in my experiments.

Answer 5

I struggled with this issue for almost two days, tried all of the solutions I could find and finally, combining several approaches, came up with the following:

MyResource:
  Type: AWS::EC2::Instance
  Metadata:
    AWS::CloudFormation::Init:
      configSets:
        setup_process:
          - "prepare"
          - "run_for_instance"
      prepare:
        commands:
          01_apt_update:
            command: "apt-get update"
          02_clone_project:
            command: "mkdir -p /replication && rm -rf /replication/* && git clone https://github.com/awslabs/dynamodb-cross-region-library.git /replication/dynamodb-cross-region-library/"
          03_build_project:
            command: "mvn install -DskipTests=true"
            cwd: "/replication/dynamodb-cross-region-library"
          04_prepare_for_apac:
            command: "mkdir -p /replication/replication-west && rm -rf /replication/replication-west/* && cp /replication/dynamodb-cross-region-library/target/dynamodb-cross-region-replication-1.2.1.jar /replication/replication-west/replication-runner.jar"
      run_for_instance:
        commands:
          01_run:
            command: !Sub "java -jar replication-runner.jar --sourceRegion us-east-1 --sourceTable ${TableName} --destinationRegion ap-southeast-1 --destinationTable ${TableName} --taskName -us-ap >/dev/null 2>&1 &"
            cwd: "/replication/replication-west"
  Properties:
    UserData:
      Fn::Base64:
        !Sub |
          #cloud-config
          cloud_final_modules:
           - [scripts-user, always]
          runcmd:
           - /usr/local/bin/cfn-init -v -c setup_process --stack ${AWS::StackName} --resource MyResource --region ${AWS::Region}
           - /usr/local/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource MyResource --region ${AWS::Region}

This is the setup for DynamoDb cross-region replication process.

Answer 6

If someone wants to do this on CDK, here's a python example.

For Windows, user data has a special persist tag, but for Linux, you need to use MultiPart User data to setup cloud-init first. This Linux example worked with cloud-config (see ref blog) part type instead of cloud-boothook which requires a cloud-init-per (see also bootcmd) call I couldn't test out (eg: cloud-init-pre always).

Linux example:

    # Create some userdata commands
    instance_userdata = ec2.UserData.for_linux()
    instance_userdata.add_commands("apt update")
    # ...
    # Now create the first part to make cloud-init run it always
    cinit_conf = ec2.UserData.for_linux();
    cinit_conf .add_commands('#cloud-config');
    cinit_conf .add_commands('cloud_final_modules:');
    cinit_conf .add_commands('- [scripts-user, always]');
    multipart_ud = ec2.MultipartUserData()
    #### Setup to run every time instance starts
    multipart_ud.add_part(ec2.MultipartBody.from_user_data(cinit_conf , content_type='text/cloud-config'))
    #### Add the commands desired to run every time
    multipart_ud.add_part(ec2.MultipartBody.from_user_data(instance_userdata));

    ec2.Instance(
        self, "myec2",
        userdata=multipart_ud,
        #other required config...
    )

Windows example:

    instance_userdata = ec2.UserData.for_windows()
    # Bootstrap
    instance_userdata.add_commands("Write-Output 'Run some commands'")
    # ...
    # Making all the commands persistent - ie: running on each instance start
    data_script = instance_userdata.render()
    data_script += "<persist>true</persist>"
    ud = ec2.UserData.custom(data_script)
    ec2.Instance(
        self, "myWinec2",
        userdata=ud,
        #other required config...
    )

Answer 7

please use the below script above your bash script.

example: here m printing hello world to my file

stop instance before adding to userdata

script

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
/bin/echo "Hello World." >> /var/tmp/sdksdfjsdlf
--//

Answer 8

Another approach is to use #cloud-boothook in your user data script. From the docs:

Cloud Boothook

• Begins with #cloud-boothook or Content-Type: text/cloud-boothook.
• This content is boothook data. It is stored in a file under /var/lib/cloud and then executed immediately.
• This is the earliest "hook" available. There is no mechanism provided for running it only one time. The boothook must take care of this itself. It is provided with the instance ID in the environment variable INSTANCE_ID. Use this variable to provide a once-per-instance set of boothook data.

Answer 9

What worked for me on Amazon Linux 2 was removing semaphores under /var/lib/cloud/instance/sem/:

sudo rm /var/lib/cloud/instance/sem/config_write_files
sudo rm /var/lib/cloud/instance/sem/config_runcmd
sudo rm /var/lib/cloud/instance/sem/config_scripts_user

In theory one could create a cron to remove these files periodically