AWS RDS Proxy unavailalbe - how to debug?

Question

I have created an RDS Proxy using Terraform. However, it does not seem to be working.

My application code cannot connect to the proxy (timeout) and aws rds describe-db-proxy-targets gives the following:

{
    "Targets": [
        {
            "Endpoint": "mydb.aaaaaaaaaaaa.eu-west-2.rds.amazonaws.com",
            "RdsResourceId": "mydb",
            "Port": 5432,
            "Type": "RDS_INSTANCE",
            "TargetHealth": {
                "State": "UNAVAILABLE",
                "Description": "DBProxy Target unavailable due to an internal error"
            }
        }
    ]
}

How can I go about debugging this?

Here is the Terraform script for the proxy. The RDS instance is described elsewhere, but is working.

data "aws_subnet" "mydb_rds" {
  filter {
    name   = "availability-zone"
    values = [ aws_db_instance.mydb.availability_zone ]
  }
}

resource "aws_secretsmanager_secret" "mydb_rds_proxy" {
  name = "mydb-rds-proxy"
}

resource "aws_secretsmanager_secret_version" "mydb_rds_proxy" {
  secret_id     = aws_secretsmanager_secret.mydb_rds_proxy.id
  secret_string = var.db_password
}

resource "aws_iam_role" "mydb_rds_proxy" {
  name = "mydb-rds-proxy"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": "rds.amazonaws.com"
      }
    }
  ]
}
EOF
}

resource "aws_iam_policy" "mydb_rds_proxy_policy" {
  name = "mydb-rds-proxy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GetSecretValue",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_secretsmanager_secret.mydb_rds_proxy.arn}"
      ]
    },
    {
      "Sid": "DecryptSecretValue",
      "Action": [
        "kms:Decrypt"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_secretsmanager_secret.mydb_rds_proxy.arn}"
      ]
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "mydb_rds_proxy_policy_attachment" {
  role       = aws_iam_role.mydb_rds_proxy.name
  policy_arn = aws_iam_policy.mydb_rds_proxy_policy.arn
}

resource "aws_db_proxy" "mydb" {
  name                   = "mydb-rds-proxy"
  debug_logging          = false
  engine_family          = "POSTGRESQL"
  idle_client_timeout    = 1800
  require_tls            = true
  role_arn               = aws_iam_role.mydb_rds_proxy.arn
  vpc_security_group_ids = [ aws_security_group.mydb_rds.id ]
  vpc_subnet_ids         = [
    data.aws_subnet.mydb_rds.id,
    aws_default_subnet.subnet_a.id,
    aws_default_subnet.subnet_b.id
  ]

  auth {
    auth_scheme = "SECRETS"
    iam_auth    = "DISABLED"
    secret_arn  = aws_secretsmanager_secret.mydb_rds_proxy.arn
  }
}

resource "aws_db_proxy_default_target_group" "mydb" {
  db_proxy_name = aws_db_proxy.mydb.name

  connection_pool_config {
    connection_borrow_timeout    = 120
    max_connections_percent      = 100
    max_idle_connections_percent = 50
  }
}

resource "aws_db_proxy_target" "mydb" {
  db_instance_identifier = aws_db_instance.mydb.id
  db_proxy_name          = aws_db_proxy.mydb.name
  target_group_name      = aws_db_proxy_default_target_group.mydb.name
}

locals {
  proxied_pg_connection_string = "postgres://${aws_db_instance.mydb.username}:${var.db_password}@${aws_db_proxy.mydb.endpoint}:5432/postgres?client_encoding=UTF8"
}

Answer 1

There are several things you need to get right for this to work:

• Username / password stored in the secret
• Security group rules from Lambda -> RDS Proxy
• Security group rules from RDS Proxy -> RDS
• RDS Proxy, Lambda and RDS in the same VPC
• RDS Proxy role can access the secret

A useful debugging query is:

aws rds describe-db-proxy-targets --db-proxy-name <proxy-name>

To understand the error message it gives back, see this page.

The username / password is the hardest thing to discover, since Terraform does not support it yet. What you need to do is construct a JSON string in Terraform that matches what RDS Proxy can understand:

resource "aws_secretsmanager_secret_version" "my_db_proxy" {
  secret_id     = aws_secretsmanager_secret.my_db_proxy.id
  secret_string = jsonencode({
    "username"             = aws_db_instance.my_db.username
    "password"             = var.db_password
    "engine"               = "postgres"
    "host"                 = aws_db_instance.my_db.address
    "port"                 = 5432
    "dbInstanceIdentifier" = aws_db_instance.my_db.id
  })
}

You then need ensure these security group rules allowing TCP traffic on port 5432 (for Postgres) exist:

• ingress Lambda to RDS Proxy
• ingress RDS Proxy to RDS
• egress RDS Proxy to "0.0.0.0/0"

The RDS Proxy role should have a policy like this:

resource "aws_iam_policy" "my_rds_proxy_policy" {
  name = "my-rds-proxy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Action": [
        "rds:*"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_db_instance.my_db.arn}"
      ]
    },
    {
      "Sid": "GetSecretValue",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_secretsmanager_secret.my_rds_proxy.arn}"
      ]
    },
    {
      "Sid": "DecryptSecretValue",
      "Action": [
        "kms:Decrypt"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    },
    {
      "Sid": "DecryptKms",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.${var.aws_region}.amazonaws.com"
        }
      }
    }
  ]
}
EOF
}

Good luck!