How can I move Terraform state in an old S3 bucket to a new S3 bucket?

Question

We have a few terraform configurations for which we use s3 as the backend. We have multiple AWS accounts, one for each of our environments.

In all the environments and across multiple region, we have different s3 bucket & dynamodb_table names used which as of now do not follow a valid convention and make it difficult to identify the purpose of buckets from its name. We now want to follow a convention based naming for all the s3 terraform state buckets. For this I will need to migrate the state of existing terraform resources to the new s3 buckets.

I am not sure what would be the best way to achieve this without having to run destroy on old state bucket and apply on the new one.

A few options that I could think of were as below but I am not sure if they are the right thing to do.

Rename the s3 bucket: I understand this is not possible (or is it?)
Move the s3 bucket: A solution to renaming is as mentioned here. Not sure if this approach will disturb the setup we have.
State migration: The terraform source code is Git controlled but .terraform is not. So, I can make changes to the source code with new bucket, commit it and create a new git tag from this commit. Now when actually doing the migration, I would checkout old git tag being used in my environment, run terraform init, checkout the new git tag and run terraform init again which would ideally ask me if I want to do a migration and do the needful. This process is something similar to what terraform suggests here, but what I am not sure is, will this approach work in the kind of movement that I am expecting to do.

P.S.: I assume renaming the DynamoDB table or just using a new one instead of old would work out of the box as I would be making sure that when doing the state migration, I do not have any live terraform runs in progress.

score 7 · Accepted Answer · answered Oct 27 '21 at 09:54

7

Did you try to copy the StateFile From old bucket to new bucket and then change the S3 bucket in terraform backend configuration

answered Oct 27 '21 at 09:54

Barak Haryati

116
1

That is what my 2nd option was, I would like to know if this will be ok, if someone can confirm it – Mukund Jalan Oct 27 '21 at 11:13
1

The remote State is the implementation of local state, and TF state files can be edited, copied and etc. We transfer in production several times and also restore state form backups with no issues. just verify no one else is running with the old state while you are transferring it. – Barak Haryati Oct 27 '21 at 22:22
That, and running terraform init -reconfigure after it. – M. Gleria Aug 31 '23 at 23:07

score 3 · Answer 2 · answered Mar 23 '23 at 18:57

Make a local copy of the Terraform state:

terraform state pull > terraform.tfstate

Remove the .terraform directory:

rm -rf .terraform/

Update the bucket in your backend definition

Initialize Terraform again and answer yes to the prompt about importing your existing state:

$ terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

See this answer for more details: https://stackoverflow.com/a/71054509/476159

score 2 · Answer 3 · answered Oct 27 '21 at 11:38

Move your Terraform state files to the new S3 buckets and then change the bucket parameter inside your S3 backend config to point to the bucket.

You do not need to do anything else if you just want to move the location of the state file - it's like changing the location of an application and then pointing the shortcut to the new location.

How can I move Terraform state in an old S3 bucket to a new S3 bucket?

3 Answers3