0

I'm using localstack so that I can run AWS Kinesis locally and test out an integration pipeline entirely locally in docker (using docker-compose).

I want to run localstack and then automatically create a Kinesis data stream. I've tried various things e.g. hooking into the docker-entrypoint.sh shell script of the localstack Dockerfile, using wait-for-it.sh, etc., but I can't seem to automatically create my stream...

If I run localstack and then manually create the stream after the container is up and running via the LocalStack AWS CLI at the command line (of the docker host), the stream gets created, but not quite sure how to get this working automatically...

Here is what I run to get create the stream aws --endpoint-url=http://localhost:4566 kinesis create-stream --stream-name test --shard-count 1

What's the best way to automate this with docker-compose?

Ryan.Bartsch
  • 3,698
  • 1
  • 26
  • 52
  • LocalStack supports [initialization hooks](https://docs.localstack.cloud/references/init-hooks/), so you can mount a shell script in `/etc/localstack/init/ready.d` that runs `awslocal kinesis create-stream ...`. [Auto create S3 Buckets on localstack](https://stackoverflow.com/questions/53619901/auto-create-s3-buckets-on-localstack) also describes this mechanism. – David Maze Apr 26 '23 at 01:42

2 Answers2

2

Create a Docker-Compose configuration:

version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "127.0.0.1:4566:4566"
    environment:
      - DEBUG=1
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${PWD}/init-aws.sh:/etc/localstack/init/ready.d/init-aws.sh"  # ready hook
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

Ensure that your init-aws.sh file is in the same directory as the Compose configuration.

Create a init-aws.sh file with the following content:

#!/bin/bash
awslocal kinesis create-stream --stream-name samplestream --shard-count  1

Make the script executable: chmod +x init-aws.sh.

Start the Docker Compose: docker-compose up.

You will find the following in your logs:

localstack_main  | 2023-04-26T03:04:14.890  INFO --- [   asgi_gw_0] l.s.k.kinesis_mock_server  : Creating kinesis backend for account 000000000000
localstack_main  | 2023-04-26T03:04:18.935  INFO --- [-functhread7] l.s.k.kinesis_mock_server  : [io-compute-blocker-2] INFO  2023-04-26 03:04:18,933 k.m.KinesisMockService contextId=09267c32-e3df-11ed-a039-bb213401b87c, cacheConfig={"awsAccountId":"000000000000","awsRegion":"us-east-1","createStreamDuration":{"length":500,"unit":"MILLISECONDS"},"deleteStreamDuration":{"length":500,"unit":"MILLISECONDS"},"deregisterStreamConsumerDuration":{"length":500,"unit":"MILLISECONDS"},"initializeStreams":null,"mergeShardsDuration":{"length":500,"unit":"MILLISECONDS"},"onDemandStreamCountLimit":10,"persistConfig":{"fileName":"000000000000.json","interval":{"length":5,"unit":"SECONDS"},"loadIfExists":true,"path":"/var/lib/localstack/tmp/state/kinesis","shouldPersist":true},"registerStreamConsumerDuration":{"length":500,"unit":"MILLISECONDS"},"shardLimit":100,"splitShardDuration":{"length":500,"unit":"MILLISECONDS"},"startStreamEncryptionDuration":{"length":500,"unit":"MILLISECONDS"},"stopStreamEncryptionDuration":{"length":500,"unit":"MILLISECONDS"},"updateShardCountDuration":{"length":500,"unit":"MILLISECONDS"}} - Logging Cache Config
localstack_main  | 2023-04-26T03:04:20.856  INFO --- [-functhread7] l.s.k.kinesis_mock_server  : [io-compute-blocker-6] INFO  2023-04-26 03:04:20,855 k.m.KinesisMockService  - Starting Kinesis TLS Mock Service on port 50733
localstack_main  | 2023-04-26T03:04:20.856  INFO --- [-functhread7] l.s.k.kinesis_mock_server  : [io-compute-blocker-6] INFO  2023-04-26 03:04:20,856 k.m.KinesisMockService  - Starting Kinesis Plain Mock Service on port 54499
localstack_main  | 2023-04-26T03:04:20.863  INFO --- [-functhread7] l.s.k.kinesis_mock_server  : [io-compute-3] INFO  2023-04-26 03:04:20,863 k.m.KinesisMockService contextId=0a4fe103-e3df-11ed-a039-bb213401b87c - Starting persist data loop
localstack_main  | 2023-04-26T03:04:22.177  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS kinesis.CreateStream => 200

Check your Kinesis streams:

awslocal kinesis list-streams

I hope this helps!

Harsh Mishra
  • 563
  • 3
  • 10
  • I can confirm that this also works on Testcontainers Java. There it may be an alternative to KINESIS_INITIALIZE_STREAMS, which is deprecated – earthling paul Jul 28 '23 at 07:40
1

I guess it is simpler, check the docs: https://docs.localstack.cloud/user-guide/aws/kinesis/

You have KINESIS_INITIALIZE_STREAMS env variable to define the stream name, like

$ docker run --name kinesis --rm -p 4566:4566 -e"KINESIS_INITIALIZE_STREAMS=my-stream" localstack/localstack

and you should be able to see it:

$ docker exec kinesis awslocal kinesis list-streams
Vercingetorix
  • 71
  • 1
  • 3