How to create a folder in an amazon S3 bucket using terraform

Question

I was able to create a bucket in an amazon S3 using this link.

I used the following code to create a bucket :

resource "aws_s3_bucket" "b" {
    bucket = "my_tf_test_bucket"
    acl    = "private"
}

Now I wanted to create folders inside the bucket, say Folder1.

I found the link for creating an S3 object. But this has a mandatory parameter source. I am not sure what this value have to , since my intent is to create a folder inside the S3 bucket.

score 95 · Accepted Answer · edited Feb 27 '19 at 14:13

95

For running terraform on Mac or Linux, the following will do what you want

resource "aws_s3_bucket_object" "folder1" {
    bucket = "${aws_s3_bucket.b.id}"
    acl    = "private"
    key    = "Folder1/"
    source = "/dev/null"
}

If you're on windows you can use an empty file.

While folks will be pedantic about s3 not having folders, there are a number of operations where having an object placeholder for a key prefix (otherwise called a folder) make life easier. Like s3 sync for example.

edited Feb 27 '19 at 14:13

GabLeRoux

16,715
16
63
81

answered Mar 10 '17 at 21:50

PaulR

1,074
1
8
3

2

Is it possible to give multiple key? – Murtuza Kolasavala Mar 27 '18 at 16:26
43

Thanks for providing a working answer instead of being pedantic about the nature of a folder. – Mark Ferree Apr 12 '18 at 21:00
2

On Windows I previously created an empty file which works but off the back of the /dev/null for *nix based stuff I thought I'd try the Windows equivalent which is just nul and it worked too! So in the above replace source = "/dev/null" with source = "nul" and it works a treat. – Andy Oct 05 '18 at 16:23
2

There seems to be an undocumented convention to use `application/x-directory` as the content type to designated a file as a directory. Hence, I've added `content_type = "application/x-directory"` to my TF file with success. Hat-tip to https://stackoverflow.com/a/44179929/1554386 – Alastair McCormack Jan 17 '19 at 16:43
5

I just tested it here, and I was able to create the folder using `content = ""` instead of `source = "/dev/null"`. I'm on Linux, but I guess it should work for Mac and Windows as well. I would say this might be a better solution since it is a non-OS-dependent one. – gabrielhof Sep 11 '20 at 13:22
It appears that v4 of the aws provider deprecated the aws_s3_bucket_object resource - https://registry.terraform.io/providers/hashicorp/aws/4.0.0/docs/resources/s3_bucket_object. See Iain's answer for an updated method - https://stackoverflow.com/a/71326287/2233241 – n8felton May 24 '22 at 16:34
this answer is out of date, see https://stackoverflow.com/a/71326287/7215135 – David Oct 19 '22 at 13:37

Ilhicas · Answer 2 · 2020-01-08T17:50:06.577

Actually, there is a canonical way to create it, without being OS dependent, by inspecting the Network on a UI put you see the content headers, as stated by : https://stackoverflow.com/users/1554386/alastair-mccormack ,

And S3 does support folders these days as visible from the UI.

So this is how you can achieve it:

resource "aws_s3_bucket_object" "base_folder" {
    bucket  = "${aws_s3_bucket.default.id}"
    acl     = "private"
    key     =  "${var.named_folder}/"
    content_type = "application/x-directory"
    kms_key_id = "key_arn_if_used"
}

Please notice the trailing slash otherwise it creates an empty file

Above has been used with a Windows OS to successfully create a folder using terraform s3_bucket_object.

Thanks for your answer, I found `content_type = "application/x-directory"` is a much better way of describing a S3 prefix/directory than the accepted answer which uses `source = "/dev/null"`. — jonashackt, Apr 29 '21 at 13:23

score 21 · Answer 3 · answered Mar 02 '22 at 16:45

The answers here are outdated, it's now definitely possible to create an empty folder in S3 via Terraform. Using the aws_s3_object resource, as follows:

resource "aws_s3_bucket" "this_bucket" {
  bucket = "demo_bucket"
}

resource "aws_s3_object" "object" {
  bucket = aws_s3_bucket.this_bucket.id
  key    = "demo/directory/"
}

If you don't supply a source for the object then terraform will create an empty directory.

IMPORTANT - Note the trailing slash this will ensure you get a directory and not an empty file

score 16 · Answer 4 · answered May 27 '16 at 21:14

16

S3 doesn't support folders. Objects can have prefix names with slashes that look like folders, but that's just part of the object name. So there's no way to create a folder in terraform or anything else, because there's no such thing as a folder in S3.

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html http://docs.aws.amazon.com/AWSImportExport/latest/DG/ManipulatingS3KeyNames.html

If you want to pretend, you could create a zero-byte object in the bucket named "Folder1/" but that's not required. You can just create objects with key names like "Folder1/File1" and it will work.

answered May 27 '16 at 21:14

Karen B

2,693
1
17
19

Thanks for the information. But my doubt is how to create an object something like a zero-byte object. What should the source be? – cmm user May 27 '16 at 21:41
You know your own use case, but the point is, S3 buckets are totally flat, there's no such thing as S3 folders, and if you want to put an object called Folder1/File1 in S3, you don't need to "create" Folder1 first. If you still really want to for some reason, just create a zero-byte file on your local filesystem and point terraform to that as the source. – Karen B May 27 '16 at 21:45
1

yes S3 doesn't support real folders but the question is how to create what S3 would consider an object that can be viewed as a folder. The answer for that which should be accepted is below. – Stephen Jun 25 '17 at 16:39
3

AWS has a create folder button. There should be an equivalent terraform op to do the same thing. If that is create a zero content file at a path that ends in a slash, terraform should allow it but I don't think it does. – Lo-Tan Oct 19 '18 at 21:32

score 3 · Answer 5 · edited Feb 27 '19 at 14:13

3

old answer but if you specify the key with the folder (that doesn't exist yet) terraform will create the folder automatically for you

terraform {
  backend "s3" {
    bucket  = "mysql-staging"
    key     = "rds-mysql-state/terraform.tfstate"
    region  = "us-west-2"
    encrypt = true
  }
}

edited Feb 27 '19 at 14:13

GabLeRoux

16,715
16
63
81

answered Aug 11 '17 at 15:10

Patrick

103
2
10

score 3 · Answer 6 · answered May 23 '22 at 17:37

I would like to add to this discussion that you can create a set of empty folders by providing the resource a set of strings:

resource "aws_s3_object" "default_s3_content" {
    for_each = var.default_s3_content
    bucket = aws_s3_bucket.bucket.id
    key = "${each.value}/"
}

where var.default_s3_content is a set of strings:

variable "default_s3_content" {
   description = "The default content of the s3 bucket upon creation of the bucket"
   type = set(string)
   default = ["folder1", "folder2", "folder3", "folder4", "folder5"]
}

Ashutosh Jindal · Answer 7 · 2019-09-05T12:06:22.040

v0.12.8 introduces a new fileset() function which can be used in combination with for_each to support this natively :

NEW FEATURES:

lang/funcs: New fileset function, for finding static local files that match a glob pattern. (#22523)

A sample usage of this function is as follows (from here):

# Given the file structure from the initial issue:
# my-dir
#    |- file_1
#    |- dir_a
#    |     |- file_a_1
#    |     |- file_a_2
#    |- dir_b
#    |     |- file_b_1
#    |- dir_c
# And given the expected behavior of the base_s3_key prefix in the initial issue

resource "aws_s3_bucket_object" "example" {
  for_each = fileset(path.module, "my-dir/**/file_*")

  bucket = aws_s3_bucket.example.id
  key    = replace(each.value, "my-dir", "base_s3_key")
  source = each.value
}

At the time of this writing, v0.12.8 is a day old (Released on 2019-09-04) so the documentation on https://www.terraform.io/docs/providers/aws/r/s3_bucket_object.html does not yet reference it. I am not certain if that's intentional.

As an aside, if you use the above, remember to update/create version.tf in your project like so:

terraform {
  required_version = ">= 0.12.8"
}

Worth noting that if you go this route it does NOT set the content-type according to your files, which is a problem if you intend to use this to host a website. If you use a data external or a null resource block (that calls aws s3 sync) you get a similar effect and it interprets the mime-type properly. — Ben Balentine, Nov 02 '19 at 20:02

How to create a folder in an amazon S3 bucket using terraform

7 Answers7

Linked