Custom Packer AMI does not execute user_data specified by Terraform

Question

I am currently struggling to get the user_data script to run when starting the EC2 instance using Terraform. I pre-configured my AMI using Packer, and referenced the custom AMI in my Terraform file. Since I need to now the RDS instance URL when starting the EC2 instance, I tried to read them inside the user_data script and set them as environment variables. My app tries to read these environment variables and can connect to the db. Everything works as expected locally, and on CI when running tests. Manually setting the variables and starting the app also works as expected. The only problem is the execution of the user_data script because it already ran when creating the AMI using Packer.

Notice how I read the current DB state inside Terraform, which is why I cannot use traditional approaches that would result in the user_data script getting executed again. I also tried deleting the cloud data as described in this question and this one without success.

This is my current Packer configuration:

build {
  name    = "spring-ubuntu"
  sources = [
    "source.amazon-ebs.ubuntu"
  ]

  provisioner "file" {
    source      = "build/libs/App.jar"
    destination = "~/App.jar"
  }

  provisioner "shell" {
    inline = [
      "sleep 30",
      "sudo apt update",
      "sudo apt -y install openjdk-17-jdk",
      "sudo rm -Rf /var/lib/cloud/data/scripts",
      "sudo rm -Rf /var/lib/cloud/scripts/per-instance",
      "sudo rm -Rf /var/lib/cloud/data/user-data*",
      "sudo rm -Rf /var/lib/cloud/instances/*",
      "sudo rm -Rf /var/lib/cloud/instance",
    ]
  }
}

This is my current Terraform configuration:

resource "aws_instance" "instance" {
  ami                    = "ami-123abc"
  instance_type          = "t2.micro"
  subnet_id              = tolist(data.aws_subnet_ids.all.ids)[0]
  vpc_security_group_ids = [aws_security_group.ec2.id]
  user_data              = <<EOF
                          #!/bin/bash
                          export DB_HOST=${data.terraform_remote_state.state.outputs.db_address}
                          export DB_PORT=${data.terraform_remote_state.state.outputs.db_port}
                          java -jar ~/App.jar
                          EOF

  lifecycle {
    create_before_destroy = true
  }
}

Starting an EC2 instance which was shut down previously or booting up a completely new EC2 instance? — Marko E, Feb 04 '22 at 13:18
Ah, ok, so you have used `user_data` in the pre-baked AMI and now when you want override the original `user_data` it does nothing? — Marko E, Feb 04 '22 at 13:28
exactly. the problem is, I need to set the RDS URL as environment variable before running the jar.. — Andre Thiele, Feb 04 '22 at 13:32
I know it should be used sparsely, but would `remote-exec` work better? — Marko E, Feb 04 '22 at 13:35

score 0 · Answer 1 · answered Feb 04 '22 at 14:16

I tried to run the commands using the remote-exec provisioner as proposed by Marko E. At first it failed with the following error, but after a second try, it worked.

This object does not have an attribute named "db_address".
This object does not have an attribute named "db_port".

This is the working configuration

resource "aws_instance" "instance" {
  ami                    = "ami-0ed33809ce5c950b9"
  instance_type          = "t2.micro"
  key_name               = "myKeys"
  subnet_id              = tolist(data.aws_subnet_ids.all.ids)[0]
  vpc_security_group_ids = [aws_security_group.ec2.id]

  lifecycle {
    create_before_destroy = true
  }

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/Downloads/myKeys.pem")
    host        = self.public_ip
  }

  provisioner "remote-exec" {
    inline = [
      "export DB_HOST=${data.terraform_remote_state.state.outputs.db_address}",
      "export DB_PORT=${data.terraform_remote_state.state.outputs.db_port}",
      "java -jar ~/App.jar &",
    ]
  }
}

The only thing I'd change would be to define local variables, e.g., `db_address` and `db_port` and assign those values instead of using `data` sources directly in the provisioner. :) Glad it helped. :) — Marko E, Feb 04 '22 at 14:32
I have another problem now, the script runs but cancels running applications after it is done. I am running with nohup and the & after the command: nohup java -jar ~/.App.jar & — Andre Thiele, Feb 04 '22 at 19:58

Custom Packer AMI does not execute user_data specified by Terraform

1 Answers1