Running integration tests in Github actions: issues with connecting with postgres

Question

I have some integration tests that, in order to succesfully run, require a running postgres database, setup via docker-compose, and my go app running from main.go. Here is my docker-compose:

version: "3.9"
services:

  postgres:
    image: postgres:12.5
    user: postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: my-db
    ports:
      - "5432:5432"
    volumes:
      - data:/var/lib/postgresql/data
      - ./initdb:/docker-entrypoint-initdb.d
networks:
  default:
    driver: bridge

volumes:
  data:
    driver: local

and my Github Actions are as follows:

jobs:
  unit:
    name: Test
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:12.5
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: password
          POSTGRES_DB: my-db
        ports:
          - 5432:5432
    env:
      GOMODCACHE: "${{ github.workspace }}/.go/mod/cache"
      TEST_RACE: true 
   steps:
     - name: Initiate Database
       run: psql -f initdb/init.sql postgresql://postgres:password@localhost:5432/my-db

     - name: Set up Cloud SDK
       uses: google-github-actions/setup-gcloud@v0

     - name: Authenticate with GCP
       id: auth
       uses: "google-github-actions/auth@v0"
       with: credentials_json: ${{ secrets.GCP_ACTIONS_SECRET }}

     - name: Configure Docker
       run: |
         gcloud auth configure-docker "europe- docker.pkg.dev,gcr.io,eu.gcr.io"

     - name: Set up Docker BuildX
       uses: docker/setup-buildx-action@v1

     - name: Start App
       run: |
         VERSION=latest make images
         docker run -d -p 3000:3000 -e      POSTGRES_DB_URL='//postgres:password@localhost:5432/my-db?sslmode=disable' --name='app' image/app

     - name: Tests
       env:
        POSTGRES_DB_URL: //postgres:password@localhost:5432/my-db?sslmode=disable
      GOMODCACHE: ${{ github.workspace }}/.go/pkg/mod
       run: | 
         make test-integration
         docker stop app

My tests run just fine locally firing off the docker-compose with docker-compose up and running the app from main.go. However, in Github actions I am getting the following error:

failed to connect to `host=/tmp user=nonroot database=`: dial error (dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory

What am I missing? Thanks

Answer 1

What you are missing is setting up the actual Postgres Client inside the Github Actions server (that is why there is no psql tool to be found).

Set it up as a step.

     - name: Install PostgreSQL client
        run: |
          apt-get update
          apt-get install --yes postgresql-client

Apart from that, if you run everything through docker-compose you will need to wait for postgres to be up and running (healthy & accepting connections).

Consider the following docker-compose:

version: '3.1'

services:
  api:
    build: .
    depends_on:
      - db
    ports:
      - 8080:8080
    environment:
      - RUN_UP_MIGRATION=true
      - PSQL_CONN_STRING=postgres://gotstock_user:123@host.docker.internal:5432/gotstockapi?sslmode=disable
    command: ./entry
    
  db:
    image: postgres:9.5-alpine
    restart: always
    environment:
      - POSTGRES_USER=root
      - POSTGRES_PASSWORD=password
    ports:
      - "5432:5432"
    volumes:
      - ./db:/docker-entrypoint-initdb.d/

There are a couple of things you need to notice. First of all, in the environment section of the api we have PSQL_CONN_STRING=postgres://gotstock_user:123@host.docker.internal:5432/gotstockapi?sslmode=disable which is the connection string to the db being passed as an env variable. Notice the host is host.docker.internal.

Besides that we have command: ./entry in the api section. The entry file contains the following #!/bin/ash script:

#!/bin/ash
NOT_READY=1
while [ $NOT_READY -gt 0 ] # <- loop that waits till postgres is ready to accept connections
do 
    pg_isready --dbname=gotstockapi --host=host.docker.internal --port=5432 --username=gotstock_user
    NOT_READY=$?
    sleep 1
done;
./gotstock-api # <- actually executes the build of the api
sleep 10
go test -v ./it # <- executes the integration-tests

And finally, in order for the psql client to work in the above script, the docker file of the api is looking like this:

# syntax=docker/dockerfile:1

FROM golang:1.19-alpine3.15

RUN apk add build-base

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download && go mod verify

COPY . .

RUN apk add postgresql-client

RUN go build -o gotstock-api

EXPOSE 8080

Notice RUN apk add postgresql-client which installs the client.

Happy hacking! =)

Answer 2

I think this code has more than one problem.

Problem one: In your code I don't see you run docker-compose up, therefore I would assume that Postgres is not running.

Problem two: is in this line: docker run -d -p 3000:3000 -e POSTGRES_DB_URL='//postgres:password@localhost:5432/my-app?sslmode=disable' --name='app' image/app

You point the host of Postgres to localhost, which on your local machine works. As there localhost is your local comuter. Though, as you use docker run you are not running this on your local machine, but in a docker container. There localhost is pointing to inside the conmtainer.

Posible solution for both

As you are already using docker-compose I suggest you to also add your test web server there.

Change your docker-compose file to:

version: "3.9"
services:

  webapp:
    build: image/app
    environment:
      POSTGRES_DB_URL='//postgres:password@postgres:5432/my-app?sslmode=disable'
    ports:
      - "3000:3000"
    depends_on:
      - "postgres"

  postgres:
    image: postgres:12.5
    user: postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: my-app
    ports:
      - "5432:5432"
    volumes:
      - data:/var/lib/postgresql/data
      - ./initdb:/docker-entrypoint-initdb.d
networks:
  default:
    driver: bridge

volumes:
  data:
    driver: local

If you now run docker-compose up, both services will be available. And it should work. Though I am not a github-actions expert, so I might have missed something. At least like this, you can run your tests locally the same way as in CI, something that I always see as a big plus.