0

I have a Django app and Postgres DB running on two separate containers. The DB connection from Django continues to be set whenever I startup the containers. I tried restarting Django, but get the same result. I'm able to go into psql from the Django container and view the tables.

Kernel: 5.11.0-36-generic x86_64 bits: 64 compiler: N/A 
  Desktop: Cinnamon 4.6.7 wm: muffin 4.6.3 dm: LightDM 1.30.0 
  Distro: Linux Mint 20 Ulyana base: Ubuntu 20.04 focal

Docker version 20.10.8, build 3967b7d

docker-compose version 1.25.0, build unknown

Error

traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/utils/autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/channels/management/commands/runserver.py", line 76, in inner_run
    self.check_migrations()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 486, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/loader.py", line 220, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
    if self.has_table():
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 55, in has_table
    with self.connection.cursor() as cursor:
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

Watching for file changes with StatReloader
Performing system checks...

Docker Compose

  web:
    build: .
    ports:
      - "8000:80"
    depends_on:
      - db
      - rasa_server
      - rasa_actions_server
      - redis
    extra_hosts:
      - "host.docker.internal:host-gateway" # for docker.host.internal to work on Linus
    volumes:
      - type: bind
        source: .
        target: /code
    command: [ "python", "manage.py", "runserver" ,"0.0.0.0:8000" ]
  ...
  db:
    build: ./databasedata/
    ports:
      - "8003:5432"
    volumes:
      - type: volume
        source: db
        target: /var/lib/postgresql/data
volumes:
  db:
  redis:

Psql root@ddf41bff0bb4:/code# psql -h host.docker.internal -p 8003 -U postgres

Docker volume (DB)

docker volume inspect ava_db
[
    {
        "CreatedAt": "2021-09-26T03:14:56-04:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "ava",
            "com.docker.compose.version": "1.25.0",
            "com.docker.compose.volume": "db"
        },
        "Mountpoint": "/var/lib/docker/volumes/ava_db/_data",
        "Name": "ava_db",
        "Options": null,
        "Scope": "local"
    }
]

Postgres container logs

...
pg_restore: creating FK CONSTRAINT "public.newweb_product_tags newweb_product_tags_tag_id_f6ff674f_fk_newweb_tag_id"
pg_restore: creating ACL "SCHEMA public"

2021-09-26 15:33:42.510 UTC [49] LOG:  received fast shutdown request
waiting for server to shut down....2021-09-26 15:33:42.512 UTC [49] LOG:  aborting any active transactions
2021-09-26 15:33:42.513 UTC [49] LOG:  background worker "logical replication launcher" (PID 56) exited with exit code 1
2021-09-26 15:33:42.514 UTC [51] LOG:  shutting down
2021-09-26 15:33:42.551 UTC [49] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2021-09-26 15:33:42.652 UTC [1] LOG:  starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-09-26 15:33:42.653 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-09-26 15:33:42.653 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-09-26 15:33:42.658 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-09-26 15:33:42.666 UTC [80] LOG:  database system was shut down at 2021-09-26 15:33:42 UTC
2021-09-26 15:33:42.672 UTC [1] LOG:  database system is ready to accept connections
user994165
  • 9,146
  • 30
  • 98
  • 165
  • Seems this container received a shutdown command pg_ctl -m fast stop, any idea which or who sent this command? – jlandercy Sep 26 '21 at 19:29
  • @jlandercy, no, not sure where that's coming from. I didn't see the container in shutdown state. – user994165 Sep 27 '21 at 07:05
  • It does not happen at the container level but inside it. This LOG line `received fast shutdown request` means postgres service inside the container receive a command like `pg_ctl -m fast stop` asking him to shut down. There is not enough information in your post to understand what is happening, so here we are guessing... This command can be sent by any service which has access to the virtual network of your container (which is exposed) and has sufficient access to the service (what kind of security you have implemented). Find who have sent this command and you will understand what to do. – jlandercy Sep 27 '21 at 07:31
  • Have you found the root cause of you issue? – jlandercy Oct 04 '21 at 06:00
  • @jlandercy, unfortunately not. However, since Postures restarts, I simply put in a sleep in the web container command so Postgres would be up by the time the web tries to reach it. – user994165 Oct 05 '21 at 14:14
  • You can use health check for that and force depends_on to have reached the healthy status. See https://github.com/peter-evans/docker-compose-healthcheck. Anyway it is a workaround because we know there is a unexpected restart we have not understood yet. – jlandercy Oct 05 '21 at 15:43
  • @jlandercy, yeah I tried that but it looks like it's not supported in Docker compose 3. https://stackoverflow.com/questions/47710767/what-is-the-alternative-to-condition-form-of-depends-on-in-docker-compose-versio. It seems that implementing retries from the app is the better practice anyhow. – user994165 Oct 06 '21 at 18:31
  • This answer dates from 2017. My advise: it is better to understand why your database behaves like this. Try to start a fresh one to check if it happens, limit network to rightful services, enforce security and integrity (password, ACL, SSL,...). Health check are still a valid way to do it with docker. And of course any extra mechanism to recover in a clean way are always welcome in a professional service. Cheers – jlandercy Oct 06 '21 at 22:59

0 Answers0