16

I have a provisioning pipeline that incorporates Terraform Cloud, and our leadership is asking us to use Terragrunt to improve Terraform code quality.

Terragrunt is a great tool for this, but I haven't see any evidence that anyone has successfully used it on Terraform Cloud.

Can anyone address this? Please only answer if you either

  1. have done this yourself, or
  2. can provide documentation that this is feasible, or
  3. provide documentation that it is not a good idea or
  4. is not possible with Terraform Cloud.
BMW
  • 42,880
  • 12
  • 99
  • 116
codeblue0000ff
  • 225
  • 3
  • 9
  • 1
    [This article](https://blog.gruntwork.io/how-deploy-production-grade-infrastructure-using-gruntwork-with-terraform-cloud-aca919ca92c2) from gruntwork seems to answer your question. – lgylym Nov 17 '21 at 13:42

3 Answers3

28

Terragrunt expects you to run terragrunt commands, and under the hood, it runs terraform commands, passing along TF_VAR_* environment variables. TFC also runs terraform commands directly. Therefore, you cannot run Terragrunt within TFC - it won't execute the terragrunt binary, only the terraform binary.

However, you can use Terragrunt from the CLI to execute Terraform runs on TFC using a remote backend. In this mode, you run Terragrunt commands as usual, and when Terraform is called, it actually executes within TFC. You can review the runs in the TFC UI, the state is stored in TFC, etc. However, due to the limitation described above, you cannot actually run Terragrunt from the UI.

To set this up, first you need to get an API token and configure the CLI with a credentials block in .terraformrc.

Next, you'll need to generate a backend block:

generate "remote_state" {
  path      = "backend.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
terraform {
  backend "remote" {
    hostname = "app.terraform.io" # Change this to your hostname for TFE
    organization = "your-tfc-organization"
    workspaces {
      name = "your-workspace"
    }
  }
}
EOF
}

This code generates a file called backend.tf alongside your Terraform module. This instructs Terraform to use TFC as a remote backend. It will use workspace called your-workspace. If this workspace doesn't exist, TFC will create it automatically using implict workspace creation. You'll have one workspace for each module you're calling in Terragrunt.

TFC does not support the TF_VAR_* environment variables that "stock" Terraform supports. Therefore, the Terragrunt inputs block, which is the standard way that Terragrunt passes variables to Terraform, does not work.

Instead, you can create a *.auto.tfvars.json file. You can generate this file in Terragrunt as well:

generate "tfvars" {
  path      = "terragrunt.auto.tfvars.json"
  if_exists = "overwrite"
  disable_signature = true
  contents = jsonencode({name = "your-name"})
}

All the variables required for a module should be passed as JSON to the the contents attribute above. A more flexible pattern is to use a locals block to set up the variables, then just pass those in within the content block. JSON is preferred to avoid type issues.

A final wrinkle is that when the workspace is created automatically, it won't have the API credentials (e.g. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) that are needed to interface with the cloud provider. You can define this in the provider configuration by creating a provider.tf file, but then the credentials are static and in plain text. Not good.

Instead, you can either set the environment variables in each workspace manually, or you can use the tfe_workspace and tfe_variable resources to create them with Terraform in advance. The latter method is recommended since it's programmatic, making it much easier to update if you need to rotate your credentials.

In both cases you'll need to have a workspace for each module called by Terragrunt.

See also: this blog post on the topic and this content on integration with Terragrunt.

Ben Whaley
  • 32,811
  • 7
  • 87
  • 85
  • That's a great answer, thank you! Just another good case for us to make for Terraform Enterprise (we had some good reasons already, but this one makes it). – codeblue0000ff Feb 04 '20 at 23:08
  • In the short term do you see any reason why I can't do the same with terraform cloud? Run the terragrunt binary in the Docker container in my CI pipeline, and then tar up the output and send it to Terraform Cloud? Or is there no point in the execution where this could be accomplished? – codeblue0000ff Feb 04 '20 at 23:11
  • 1
    I don't see how it could work. TFC is an execution platform for Terraform, and it's designed to run Terraform on worker VMs. Terragrunt `plan-all` and `apply-all` commands invoke Terraform in a particular order, passing in variables and such. The TFC worker nodes need to be able to execute Terragrunt. If you did something like, for example, run terragrunt in your local CI first, it would kind of defeat the purpose of using TFC. – Ben Whaley Feb 05 '20 at 00:46
  • 2
    I suppose you could use TFC for state storage but local execution ([see here](https://www.terraform.io/docs/backends/types/remote.html)). But then you still wouldn't be leveraging some of the main benefits of TFC. – Ben Whaley Feb 05 '20 at 00:55
  • I've updated the answer with some working configuration for using Terragrunt locally and executing in Terraform Cloud. – Ben Whaley May 15 '20 at 04:00
  • 1
    Terraform version 1.1.3 with cloud block now supports `TF_VAR` and `-var-file` CLI flags – Rahul Patil Jan 11 '22 at 10:52
2

I have received a direct response from josh-padnick at at Gruntwork that as of now there are no concrete plans to make this work, and confirmed that it cannot currently work to the best of his knowledge. I appreciate all the answers!

codeblue0000ff
  • 225
  • 3
  • 9
1

Really the only thing that's stopping terragrunt from interfacing with the VCS workspaces in tfcloud fairly easily is that it stages all of its files to the terragrunt cache, and in there it runs the directory names through hashes to prevent conflict. What we did was put together a helper utility to help address this use case called terrastage. It uses the terragrunt libraries but instead of putting files into the terragrunt cache using hashed names it gives you control over where you want to put the files allowing it to interface with the VCS workspaces of terraform cloud fairly easily. You can use this same approach to make terragrunt work with any native terraform tool really.

I put it up on github under JasonPodgorny/terrastage if this helps anyone.