flyway unable to connect to postgres container within docker-entrypoint-initdb.d script

Question

I'm trying to extend the postgres Docker image to potentially (via an environment variable flag) execute flyway DB migrations on DB init. My Dockerfile is here:

FROM postgres:9.6

# Install curl and java (for Flyway)
RUN set -x \
    && apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates curl openjdk-8-jre 

# Install Flyway
ENV FLYWAY_VERSION 4.2.0
ENV FLYWAY_INSTALL_DIR /usr/src/flyway
ENV FLYWAY_CONF ${FLYWAY_INSTALL_DIR}/flyway-${FLYWAY_VERSION}/conf/flyway.conf
ENV FLYWAY_EXE ${FLYWAY_INSTALL_DIR}/flyway-${FLYWAY_VERSION}/flyway
RUN mkdir -p ${FLYWAY_INSTALL_DIR} && \
    curl -L https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/${FLYWAY_VERSION}/flyway-commandline-${FLYWAY_VERSION}.tar.gz | \
    tar -xzC ${FLYWAY_INSTALL_DIR} && \
    chmod +x ${FLYWAY_EXE}

# Copy migration scripts
ENV MIGRATIONS_LOCATION /flyway/migrations
COPY migrations $MIGRATIONS_LOCATION

COPY init_db.sh /docker-entrypoint-initdb.d/init_db.sh

With my init_db.sh startup script:

#!/bin/bash
set -e

RUN_MIGRATIONS="${RUN_MIGRATIONS:-false}"
DB_URL="jdbc:postgresql://localhost:5432/$DB_NAME"

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
    CREATE DATABASE $DB_NAME;
EOSQL

if [ "$RUN_MIGRATIONS" == "true" ]; then
        echo "running migrations ..."
        ${FLYWAY_EXE} -user=$POSTGRES_USER -password=$POSTGRES_PASSWORD -url=$DB_URL -locations="filesystem:$MIGRATIONS_LOCATION" migrate
fi

However, when running the container with RUN_MIGRATIONS=true, flyway fails to connect to postgres:

docker build . -t postgres-flyway && docker run -e DB_NAME=db -e RUN_MIGRATIONS=true -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres postgres-flyway
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....LOG:  database system was shut down at 2018-08-06 02:19:32 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections
 done
server started
ALTER ROLE


/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/init_db.sh
CREATE DATABASE
running migrations ...
Flyway 4.2.0 by Boxfuse

ERROR:
Unable to obtain Jdbc connection from DataSource (jdbc:postgresql://localhost:5432/db) for user 'postgres': Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State  : 08001
Error Code : 0
Message    : Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

The postgres image runs postgres on port 5432 (as usual) so I'm at a loss on why flyway is unable to connect to postgres over localhost:5432.

I also noticed that within this context, pg_isready states that postgres is accepting connections but when specifying the hostname as localhost or 127.0.0.1 it is unable to reach postgres either. That is, by inserting a few pg_isready commands in my init_db.sh script:

...
pg_isready
pg_isready -p 5432
pg_isready -h localhost -p 5432
...

I see the following log output on postgres init:

...
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
localhost:5432 - no response
...

I'm suspicious that I've reached a limitation of postgres' initialize context, but I would like to understand why postgres is unreachable over localhost/127.0.0.1:5432 at this point of initialization.

Answer 1

I had the same problem running flyway when creating a docker image for my database based on the postgres:10.5 image. I added the following to my entrypoint.sh before running flyway, to confirm that the problem I was seeing was caused by the docker-entrypoint.sh change @Nick Maraston posted in his answer:

echo "$(date) - waiting for database to start"
while ! pg_isready -h localhost -p 5432 -d $POSTGRES_DB
do
    echo "$(date) - waiting for database to start"
    sleep 10
done

The result was that the above code looped for ever. I then replaced it with the following code to restart the database listening for TCP/IP connections on localhost:

pg_ctl -D "$PGDATA" -m fast -w stop
pg_ctl -D "$PGDATA" \
            -o "-c listen_addresses='localhost'" \
            -w start

Rather than restarting the database like this, a cleaner solution would be to use the JDBC -socketFactory option explained here.

Answer 2

I discovered the problem while digging through the images entry point script. A recent change to the image restricts postgres to only listen for connections over a unix domain socket during internal initialization: https://github.com/docker-library/postgres/pull/440

Answer 3

It is true that postgres docker is listening to a unix socket, eg. /var/run/postgresql/.s.PGSQL.5432. But it is not necessary to force the server to switch its listening address. Postgres database URI allows a connection string to point to a unix socket.

Reference: Connect to a database over a unix socket using SQLAlchemy

The example provided:

export DATABASE_URL=postgres://user:password@/dbname?host=/path/to/unix/socket

I was able to omit the host, and decided to use this environment variable in my /docker-entrypoint-initdb.d/*.sh script instead. Note that no string follows the @ symbol, nor is there a host query string here. You may need to explicitly define the host depending on your application.

Solution:

export DATABASE_URL="postgres://$POSTGRES_USER:$POSTGRES_PASSWORD@/$POSTGRES_DB"