I am building a data pipeline with Airbyte, PostgreSQL and dbt. PostgreSQL and DBT I can easily set up via my main docker-compose.yml but with Airbyte I am not sure. Airbyte itself is a multi-container app so it has it's own docker-compose.yml.
To deploy Airbyte, I follow the steps here and run it with the script `./run-ab-platform.sh. I am not sure though how to integrate this into the main docker-compose.yml.
Current docker-compose.yml structure:
version: "3.8"
services:
db:
container_name: postgres
image: postgres:latest
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=mysecretpassword
- POSTGRES_DB=training
ports:
- '5432:5432'
volumes:
- ./postgres:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 60s
timeout: 30s
retries: 5
dbt:
image: fishtownanalytics/dbt:latest
environment:
DBT_PROFILES_DIR: /dbt/profile
volumes:
- ./my_dbt_profiles:/dbt/profile
depends_on:
- postgres
I already tried to use build or extend but can't seem to get it right.