Terraform: Setting up logging from AWS LoadBalancer to S3 bucket

Question

I have an aws_lb that I want to log to an S3 bucket.

What I have unsuccessfully tried to do:

data "aws_elb_service_account" "main" {}

data "aws_iam_policy_document" "bucket_policy" {
  statement {
    sid       = ""
    actions   = ["s3:PutObject"]
    resources = ["arn:aws:s3:::my-bucket/*"]

    principals {
      type        = "AWS"
      identifiers = ["${data.aws_elb_service_account.main.arn}"]
    }
  }
}

I also tried this:

resource "aws_iam_role" "lb-logs-role" {
  name = "lb-logs-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "elasticloadbalancing.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF

  tags = {
    Name = "lb-logs-role"
    Environment  = terraform.workspace
    Management   = "Managed by Terraform"
  }
}

resource "aws_iam_role_policy" "s3-logs-access" {
  name = "s3-logs-access"
  role = aws_iam_role.lb-logs-role.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
EOF
}

This is the error I am seeing:

Error: Failure configuring LB attributes: InvalidConfigurationRequest: Access Denied for bucket: my-bucket. Please check S3bucket permission
        status code: 400, request id: 5b629210-9738-11e9-bcc6-6f3b4f22bf28

  on modules/tableau-linux/lb.tf line 1, in resource "aws_lb" "main":
   1: resource "aws_lb" "main" {

Any ideas?

Possible duplicate of [Terraform ELB S3 Permissions Issue](https://stackoverflow.com/questions/43366038/terraform-elb-s3-permissions-issue) — ydaetskcoR, Jun 25 '19 at 10:16
I have tried the code from the possible duplicate, to no avail. Don't know if that matters, but that question uses an aws_elb, not aws_lb. — Joseph Tura, Jun 25 '19 at 11:32
@JosephTura please check I test with Terraform version `Terraform v0.12.1` — Adiii, Jun 26 '19 at 10:04

score 3 · Answer 1 · answered Sep 18 '19 at 20:53

It looks like the API will request the ACL of the bucket to see if it has permission, and populate the initial folder structure, therefore the even though the aws_elb_service_account has permissions to putObject in the bucket the api call will fail. This policy is what the AWS web console creates when it creates the S3 bucket for you, and it solved it for me.

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}
data "aws_elb_service_account" "main" {}
resource "aws_s3_bucket_policy" "lb-bucket-policy" {
  bucket = aws_s3_bucket.lb-log-storage-s3.id

  policy = <<POLICY
{
    "Id": "Policy",
    "Version": "2012-10-17",
    "Statement": [{
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "${data.aws_elb_service_account.main.arn}"
                ]
            },
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "${aws_s3_bucket.lb-log-storage-s3.arn}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "${aws_s3_bucket.lb-log-storage-s3.arn}/AWSLogs/${data.aws_caller_identity.current.account_id}/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": [
                "s3:GetBucketAcl"
            ],
            "Resource": "${aws_s3_bucket.lb-log-storage-s3.arn}"
        }
    ]
}
POLICY
}

Adiii · Answer 2 · 2019-06-26T10:17:18.420

It seems the issue is with your policy but you can try my code using aws_lb, Here is the complete configuration to launch to LB in default VPC and create bucket named test-bucket-1-unique-name, policy and LB named test-http-lb. Along with SG and Route53 entry that is commented.

# Creating Load Balancer
resource "aws_lb" "httplb" {
  name                       = "test-http-lb"
  internal                   = false
  load_balancer_type         = "application"
  security_groups            = ["${aws_security_group.lbsg.id}"]
  subnets                       = ["subnet-99fdf8e0", "subnet-902b0ddb"]
  enable_deletion_protection = false
  access_logs {
    bucket  = "${aws_s3_bucket.bucket.bucket}"
    prefix  = "http-lb"
    enabled = true
  }
  tags = {
    Environment = "test-http"
  }
}


# Creating Security Groups for Load Balancer
resource "aws_security_group" "lbsg" {
  name        = "test-loadbalancer-sg"
  description = "test-Allow LB traffic"
  tags = {
    Name = "test-SG-Balancer"
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "HTTP"
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}










#uncomment this if you want to add route53 record
# resource "aws_route53_record" "web" {
#   zone_id = "${data.aws_route53_zone.primary.zone_id}"
#   name    = "${var.env_prefix_name}.ironman.co
#   type    = "A"

#   alias {
#     name                   = "${aws_lb.httplb.dns_name}"
#     zone_id                = "${aws_lb.httplb.zone_id}"
#     evaluate_target_health = true
#   }
# }




data "aws_elb_service_account" "main" {}

# Creating policy on S3, for lb to write
resource "aws_s3_bucket_policy" "lb-bucket-policy" {
  bucket = "${aws_s3_bucket.bucket.id}"

  policy = <<POLICY
{
  "Id": "testPolicy1561031527701",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "testStmt1561031516716",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::test-bucket-1-for-lb-logs/http-lb/*",
      "Principal": {
        "AWS": [
           "${data.aws_elb_service_account.main.arn}"
        ]
      }
    }
  ]
}
POLICY
}

resource "aws_s3_bucket" "bucket" {
  bucket = "test-bucket-1-for-lb-logs"
  acl    = "private"
  region = "us-west-2"

  versioning {
    enabled = false
  }
  force_destroy = true


}

Then go to your S3 bucket and verify TestFile. Here are the logs from terraform

score 0 · Answer 3 · answered Sep 26 '22 at 20:40

This one stumped me also, but I got it working with this

        {
          Effect : "Allow",
          Principal : {
            "AWS" : "arn:aws:iam::127311923021:root"
          },
          Action : [
            "s3:PutObject"
          ],
          Resource : "${aws_s3_bucket.logging_bucket.arn}/AWSLogs/${data.aws_caller_identity.current.account_id}/*",
        }

Where does 127311923021 come from, you ask? Believe me, I did too! This AWS document says it is the ID of the AWS account for Elastic Load Balancing for your Region (us-east-1 in this case). It has a large table of these magic numbers!

Terraform: Setting up logging from AWS LoadBalancer to S3 bucket

3 Answers3