Auto create S3 Buckets on localstack

Question

Using localstack in my docker-compose mainly to mimic S3.

I know I can create buckets, that's not the issue. What I would like to do is automatically create the buckets when I run a docker-compose up.

Is there something built in already for localstack?

Matthew Warman · Accepted Answer · 2023-01-05T18:07:10.690

A change that came in with this commit since version 0.10.0.

When a container is started for the first time, it will execute files with extensions .sh that are found in /docker-entrypoint-initaws.d. Files will be executed in alphabetical order. You can easily create aws resources on localstack using awslocal (or aws) cli tool in the initialization scripts.

Before v1.1.0

version: '3.7'
services:
  localstack:
    image: localstack/localstack
    environment:
      - SERVICES=s3
    ports:
      - "4566:4566"
      # - "4572:4572" Old S3 port
    volumes:
      - ./aws:/docker-entrypoint-initaws.d

After v1.1.0

The /docker-entrypoint-initaws.d directory usage has now been deprecated. The Pluggable initialization hooks in /etc/localstack/init/<stage>.d have replaced the legacy Init scripts, and the former will be removed entirely in the 2.0 release.

Reference: localstack/localstack/issues/7257

Init hooks official documentation.

version: '3.7'
services:
  localstack:
    image: localstack/localstack
    environment:
      - SERVICES=s3
    ports:
      - "4566:4566"
    volumes:
      - ./aws:/etc/localstack/init/ready.d

With a script in directory ./aws/buckets.sh:

#!/usr/bin/env bash
awslocal s3 mb s3://bucket

Will produce this output:

Before v1.1.0

...
localstack_1  | Starting mock S3 (http port 4572)...
localstack_1  | Waiting for all LocalStack services to be ready
localstack_1  | Ready.
localstack_1  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initaws.d/buckets.sh
localstack_1  | make_bucket: bucket
localstack_1  |

After v1.1.0

localstack-1  | Ready.
localstack-1  | 2023-01-05T18:07:50.163  INFO --- [-functhread8] l.services.motoserver      : starting moto server on http://0.0.0.0:40443
localstack-1  | 2023-01-05T18:07:50.165  INFO --- [   asgi_gw_0] localstack.services.infra  : Starting mock S3 service on http port 4566 ...
localstack-1  | 2023-01-05T18:07:50.324  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.CreateBucket => 200
localstack-1  | make_bucket: bucket

The script part of the answer is slightly wrong. The `mb` and `s3` are the wrong way round! Should be `awslocal s3 mb s3://bucket`. Thanks for the answer though, saved me a lot of hassle! — Matt Keen, Nov 21 '19 at 21:52
@MattKeen where is the directory `docker-entrypoint-initaws.d` and its contents (`.sh` scripts) supposed to be created in the localstack container? I've tried creating it in `/usr/local/bin/` and `/opt/code/localstack/`, but it doesn't look like the `.sh` scripts are getting run when the container starts. — feature_not_bug, Dec 23 '19 at 18:59
They should be in `/docker-entrypoint-initaws.d`, therefore `./aws/buckets.sh` would be `/docker-entrypoint-initaws.d/buckets.sh`, based on the configuration above. — Matthew Warman, Jan 06 '20 at 08:39
That's perfect, thank you. Do you have any idea how I can set my healthcheck to wait for this bucket to be created ? — Boris Le Méec, Apr 08 '21 at 10:24
This might work as a test for healthcheck `test: "awslocal s3api head-bucket --bucket bucket"` — Matthew Warman, Apr 12 '21 at 15:59
One little sidenote: localstack needs aws cli credentials. So either set them within the script or as env-variables in the compose file. Script: ```export AWS_ACCESS_KEY_ID=foobar export AWS_SECRET_ACCESS_KEY=foobar``` — Tobias Stangl, Feb 08 '22 at 09:28
Looks like `/docker-entrypoint-initaws.d` is deprecated. See below answer https://stackoverflow.com/a/74819396/9711710 for more information and updated solution. — Matti, Jan 04 '23 at 22:00

score 8 · Answer 2 · edited Oct 03 '22 at 19:54

I have been able to achieve this using Localstack with kind of "workaround":

Start Localstack

Create expected buckets, e.g.:

aws --endpoint-url=http://localhost:4572 s3 mb s3://test1

Above line will update the s3_api_calls.json file in the Localstack directory (by default on Linux it's /tmp/localstack/data
Backup the file
Put the copied file in the Localstack directory ( /tmp/localstack/data by default) before starting the stack again
You should be able to see something like 2019-03-21T08:38:28:INFO:localstack.utils.persistence: Restored 2 API calls from persistent file: /tmp/localstack/data/s3_api_calls.json the in startup log after you start Localstack again and the bucket should be available: aws --endpoint-url=http://localhost:4572 s3 ls s3://test1

score 6 · Answer 3 · answered Dec 16 '22 at 01:44

Important notice for users of LocalStack v1.1.3 and above

Since the 1st of December 2022, LocalStack has announced the deprecation of legacy init scripts (/docker-entrypoint-initaws.d) with the release of v1.3.0. The replacement - plugggable initialisation hooks - was introduced in v1.1.0.

They will be fully removed in v2.0.0, posing a risk for anyone that would be using /docker-entrypoint-initaws.d with the latest version of LocalStack.

This would be a breaking change.

For future-proofing, the accepted answer is still valid but I would recommend replacing /docker-entrypoint-initaws.d with /etc/localstack/init/ready.d.

This would mimic the previous intended behaviour, as it would be hooking into the READY stage where LocalStack can actually create your S3 buckets, while making sure you can still continue to update LocalStack.

This would be the best current Dockerfile:

version: '3.8'
services:
  localstack:
    image: localstack/localstack:latest
    environment:
      - SERVICES=s3
    ports:
      - "4566:4566"
    volumes:
      - ./aws:/etc/localstack/init/ready.d

score 0 · Answer 4 · answered Jun 22 '23 at 16:21

Just spend a day making it right. Here we go!

The goal is: run docker-compose with localstack, create S3 bucket s3://this-is-my-bucket and copy over some json files from project source

LOCALSTACK_BUILD_VERSION: latest today, 2.1.1.dev

Docker compose:

localstack:
  image: localstack/localstack:latest
  networks:
    - hmpps
  container_name: localstack
  ports:
    - "4566-4597:4566-4597"
    - 8999:8080
    - 9080:9080
  environment:
    - SERVICES=s3
    - PORT_WEB_UI=9080
    - DEBUG=${DEBUG- }
    - DATA_DIR=/tmp/localstack/data
    - DOCKER_HOST=unix:///var/run/docker.sock
    - DEFAULT_REGION=eu-west-2
  volumes:
    - "./src/test/resources/test-data:/tmp/localstack/test-data"
    - "$PWD/src/test/resources/localstack/setup-s3.sh:/etc/localstack/init/ready.d/init-aws.sh"
    - "/var/run/docker.sock:/var/run/docker.sock"

Two important things here:

src/test/resources/test-data - is the source folder for your json files
src/test/resources/localstack/setup-s3.sh - is a script file to run during docker-compose

Script file setup-s3.sh

#!/usr/bin/env bash
set -e
export TERM=ansi
export AWS_ACCESS_KEY_ID=foobar
export AWS_SECRET_ACCESS_KEY=foobar
export AWS_DEFAULT_REGION=eu-west-2
export PAGER=

echo "S3 Configuration started"

aws --endpoint-url=http://localhost:4566 s3 mb s3://this-is-my-bucket
aws --endpoint-url=http://localhost:4566 s3 cp /tmp/localstack/test-data/ s3://this-is-my-bucket/chart --recursive

echo "S3 Configured"

Look at "--recursive" param for "aws cp" command. That took us 4 hours!

Does not work without "--recursive". Crazy!

logstack docker logs:

2023-06-22 17:07:00 S3 Configuration started
2023-06-22 17:07:01 2023-06-22T16:07:01.070  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.CreateBucket => 200
2023-06-22 17:07:01 make_bucket: this-is-my-bucket
2023-06-22 17:07:01 2023-06-22T16:07:01.431  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200
upload: ../../../tmp/localstack/test-data/1a.json to s3://this-is-my-bucket/chart/1a.json
2023-06-22 17:07:01 S3 Configured

UI for localstack S3 buckets is here http://s3.localhost.localstack.cloud:4566/this-is-my-bucket

Not sure if required, we have application-test.yml properties

aws:
  region-name: eu-west-2
  endpoint-url: "http://localhost:4566"
  access_key: foobar
  secret_access_key: foobar
  s3:
    access_key: foobar
    secret_access_key: foobar
    bucket_name: this-is-my-bucket

Next goal is to write an integration test, point to localstack s3://this-is-my-bucket bucket, load "chart/1a.json" file, and run the assertion magic.

Thanks!

score -2 · Answer 5 · answered Nov 19 '19 at 08:22

-2

DATA_DIR: Local directory for saving persistent data (currently only supported for these services: Kinesis, DynamoDB, Elasticsearch, S3)

answered Nov 19 '19 at 08:22

Bigo

93
1
12

Auto create S3 Buckets on localstack

5 Answers5

Important notice for users of LocalStack v1.1.3 and above

Linked