No response after connecting from celery to redis via ssl

Question

I'm following this tutorial, and adjusting the Celery-background related code to my project.

In my case I am operating in a Docker environment, and I have a secured site (i.e. https://localhost). which requires secured ssl communication.

I adjusted the code for secure connection.
I had initial connection problems, which created log error messages, but I was able to solve - see here.

Now the log file is quite, but I think that I still have connection problems. As a result, at runtime, when triggerring a task, nothing happens.

What could be the reason for no connection in the secured case?
Should I expect a message if the keys are incorrect? Is there a way to test the connection from celery/web containers to redis container from the command line?

Why do you ask for something "besides logs"? Have you looked at logs from all the parts involved, ie the Flask process, the Celery worker and Redis? — Miguel Grinberg, Feb 15 '20 at 22:12
@Miguel, redis log looks the same with/without ssl. celery log looks different (I added the log to the post - "with ssl" is missing the connection but I don't see any error messages. — Avner Moshkovitz, Feb 15 '20 at 23:35
To test, I provided wrong keys and I'm still getting the same log (i.e. no message that the keys are wrong). The keys were generated for localhost (outside the container), and CELERY_BROKER_URL uses rediss://redis:6380/0 (not rediss://localhost:6380/0). Could this be a problem? — Avner Moshkovitz, Feb 15 '20 at 23:38
What I see is that you are making a lot of assumptions about things you think are okay, so you are not mentioning them. For example, how you configure and/or start redis with/without ssl isn't mentioned, when it is clear that when you use your ssl configuration Celery is unable to connect. You could have also tested connecting to redis with the redis-cli to have another client. Also Celery has options to make the log more verbose which you haven't used. — Miguel Grinberg, Feb 16 '20 at 17:01
@Miguel, thanks for the leads. I made changes to the configuration of the containers, which I describe the solution. — Avner Moshkovitz, Feb 18 '20 at 05:06

score 0 · Accepted Answer · answered Feb 18 '20 at 05:04

I was able to fix the problem by making changes to the configuration of the containers.
Specifically, I made the following changes:

In redis container:

followed this tutorial to add stunnel, and create certs for the redis container
used this git code to configure stunnel within a Docker container

In celery container:
- elevated the log level to debug

I first tested that I can connect from my localhost to the redis docker container over ssl. This is described here

Then, I tested that I can connect from the celery container to the redis container over ssl. The docker-compose file is:

version: '3'

services:
  web:
    restart: always
    build:
      context: ./web
      dockerfile: Dockerfile
    expose:
      - "8000"
    volumes:
      - /home/webServer/web:/home/flask/app/web
      - /home/webServer/redis/ssl:/etc/certs
      - data2:/home/flask/app/web/project/img
    command: /usr/local/bin/gunicorn -w 2 -t 3600 -b :8000 project:app
    depends_on:
      - postgres
    stdin_open: true
    tty: true

  nginx:
    restart: always
    build:
      context: ./nginx
      dockerfile: Dockerfile
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /home/webServer/web:/home/flask/app/web
      - data2:/home/flask/app/web/project/img
    depends_on:
      - web

  postgres:
    restart: always
    build:
      context: ./postgresql
      dockerfile: Dockerfile
    volumes:
      - data1:/var/lib/postgresql/data
    expose:
      - "5432"

  redis:
    build:
      context: ./redis
      dockerfile: Dockerfile
    restart: always

    command: sh -c "stunnel /stunnel-redis-server.conf && /usr/local/bin/redis-server /etc/redis/redis.conf"
    expose:
      - '6380'
    ports:
     - "6380:6380"
    volumes:
      - /home/webServer/redis/ssl:/etc/certs
      - /home/webServer/redis/conf:/etc/redis

  celery:
    build:
      context: ./web
    command: watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery worker -A project.celery  --loglevel=debug
    volumes:
      - /home/webServer/web:/home/flask/app/web
      - /home/webServer/redis/ssl:/etc/certs
      - data2:/home/flask/app/web/project/img
    depends_on:
      - redis

volumes:
  data1:
  data2:

Other related files are:

files on the redis docker container: (the settings in these files are described here)

redis container Dockerfile
redis/conf/redis.conf
redis/stunnel-redis-server.conf

settings on the celery docker container:

cat web/project/flask_celery.py
...

key_file = '/etc/certs/localhost.key'
cert_file = '/etc/certs/private.pem'
ca_file = '/etc/certs/myCA.pem'
...    

celery = Celery(app.import_name,
                backend=app.config['CELERY_RESULT_BACKEND'],
                broker=app.config['CELERY_BROKER_URL'],
                broker_use_ssl = {
                    'ssl_keyfile': key_file,
                    'ssl_certfile': cert_file,
                    'ssl_ca_certs': ca_file,
                    'ssl_cert_reqs': ssl.CERT_REQUIRED
                },
                redis_backend_use_ssl = {
                    'ssl_keyfile': key_file,
                    'ssl_certfile': cert_file,
                    'ssl_ca_certs': ca_file,
                    'ssl_cert_reqs': ssl.CERT_REQUIRED
                })

------------------

cat project/__init__.py
...
app.config['CELERY_BROKER_URL'] = 'rediss://webserver_redis_1:6380/0'
app.config['CELERY_RESULT_BACKEND'] = 'rediss://webserver_redis_1:6380/0'

No response after connecting from celery to redis via ssl

1 Answers1

Linked