Can't keep postgres data persistent using Github CodeSpaces with Docker-Compose

Question

I set up a Github codespaces environment using devcontainer.json and docker-compose.yaml. Everything works fine, but the postgres database defined in docker-compose.yml loses its data every time the container needs to be re-built.

Here's the bottom part of the docker-compose.yml

      db:
        image: postgres:latest
        restart: unless-stopped
        volumes:
          - postgres-data:/var/lib/postgresql/data
        environment:
          POSTGRES_USER: test_user
          POSTGRES_DB: test_db
          POSTGRES_PASSWORD: test_pass
       
   volumes:
     postgres-data:

as you can see, I am trying to map the postgres data volume into a postgres-data volume, but this doesn't work for some reason.

What am I doing wrong that's preventing postgres data from persisting between container builds?

Answer 1

Another option would be to look into using Spawn. (Disclaimer - I'm one of the devs working on it).

We've written some documentation about exactly how to use Spawn-hosted databases with GitHub codespaces here: https://docs.spawn.cc/blog/2021/08/01/spawn-and-codespaces

This will allow you to provision a database thats independent from the GitHub codespace and preserve data between restarts.

You get some extra features with Spawn like arbitrary save points, resets and loading back to saved revisions with Spawn - but the key functionality of spinning up a database for a GitHub codespace and preserving data is one of the things it works extremely well for.

Answer 2

according to https://docs.github.com/en/codespaces/customizing-your-codespace/configuring-codespaces-for-your-project#dockerfile , only docker images can be pulled from source and set-up, nowhere they mention that volume persistence is guaranteed.

and after going through this https://code.visualstudio.com/docs/remote/devcontainerjson-reference looks like mounts and few other features related to volumes are not supported for codespaces.

workspaceMount : Not yet supported in Codespaces or when using Clone Repository in Container Volume.

workaround :

in .devcontainer folder where your dockerfile is present add a line like this

RUN curl https://<your_public_cloud>/your_volume.vol -O

here <your_public_cloud> can be google drive, aws or any endpoint where you have access to download the volume. its also the volume you needed to be persist.

• and once its downloaded you can mount the volume to postgres service or make a hotswap.

• and when you want to save, just upload the volume to your cloud storage provider.

• repeat the process every time you build, and save and upload before "unbuild" or dismissing your codespace whatever you like to call.

hope that eases your issue, happy coding!

Answer 3

As long as you don't remove the volume with docker-compose down --volumes as an example, the data should persist.

I had the same issue; and it turned out that I had put a crontab running docker system prune -af every 15 minutes!

Answer 4

As you don't have access to VM, maybe the directory containing your docker-compose.yml changes.

In that case, volume name may change too.

Indeed, by default, your volume name would be the following :

<directory_name>_postgres-data

Could you try a named volume (starting with compose 3.4):

  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: test_user
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: test_pass
       
volumes:
  postgres-data:
    external: false
    name: postgres-data

documentation of docker-compose can be found here :

https://docs.docker.com/compose/compose-file/compose-file-v3/#name

EDIT 1

If your VM is created at each build, docker dependencies too.

volumes, networks, etc...

A persistent volume is needed somewhere (surviving VM builds).

You may have to create a directory in your local workspace, like:

/local/workspace/postgres-data/

which become in codespaces according to my understanding :

./postgres-data

Check permissions, your user may not exist in the container.

As a result your compose file become:

  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - ./postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: test_user
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: test_pass

Answer 5

You could just mount a host directory, instead of using a docker volume:

volumes:
  - /home/me/postgres_data:/var/lib/postgresql/data

This guarantees that no volume cleanup (accidental or deliberate) nukes your database.

Indeed the postgres docs do this in their examples. See the PGDATA environment variable.