rq worker unable to connect to EC2 through Heroku app (SSL: CERTIFICATE_VERIFY_FAILED)

Question

My Heroku app uses Redis and an rq worker. I recently enabled SSL on my Heroku app via the dashboard, and was met with SSL errors. Initially I was met with the following error when hitting an endpoint that used Redis:

app[redis-cylindrical-86839]: Error accepting a client connection: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca

but I resolved that by following advice read on a similar Stack Overflow thread by initializing Redis in-code with ssl_cert_reqs=None. However, I'm still being met with the following error in my rq worker:

app[worker.1]: ERROR:root:Error 1 connecting to ec2-XX-XXX-XX-XXX.compute-1.amazonaws.com:11459. [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1002).

and the worker then crashes. Nothing I've tried thus far has worked (mainly different methods of initializing Redis found on StackOverflow or suggested by Heroku support).

My current initialization code:

from redis import Redis, from_url as redis_from_url
from rq import Queue
from dotenv import load_dotenv

# Setup
load_dotenv()

redis_url = os.getenv('REDIS_URL')
if redis_url:
    parsed_redis_url = urlparse(redis_url)
    redis =  Redis(host=parsed_redis_url.hostname, port=parsed_redis_url.port, password=parsed_redis_url.password, ssl=True, ssl_cert_reqs=None)
else:
    # for local development
    redis = Redis()

Any insight as to what's going on, or better yet how to fix my RQ worker would be greatly appreciated. Also I understand there are similar threads to this on StackOverflow, but none of them mention the issue as it pertains to RQ workers, and none of the solutions have worked for me.

Answer 1

If you connect via AWS EC2 and have not configured certs on the remote machine, why not use the HTTP endpoint provided in your Heroku Config when you provisioned the Redis instance?

Also, if you are using ssl_cert_reqs=None, try using the from_url method and passing in the URL as {REDIS_HTTP_URL}?ssl_cert_reqs=None: https://redis-py.readthedocs.io/en/stable/connections.html#redis.Redis.from_url

I can connect from my local Windows machine without issues using this setup.

Try the following:

from redis import Redis, from_url
from rq import Queue
from dotenv import load_dotenv

# Setup
load_dotenv()

# Load the URL value from dotenv/config. Prod should use SSL/secure connections.
redis_url = f"{os.getenv('REDIS_URL')}?ssl_cert_reqs=None"

# Create the connection - look into connection pools/globally passing.
redis =  Redis.from_url(redis_url)

# Pass the connection to RQ
queue = rq.Queue(connection=redis)

For the worker issue, what do you have in your Heroku Procfile? To help more, why are you using EC2 to interact with a Redis instance provisioned in Heroku? You can have everything run via Heroku without needing the EC2.

Also, suppose the goal is to have the EC2 be a remote worker that processes jobs placed into the queue from your Flask App hosted on Heroku. In that case, you need to ensure the application code is available on both the remote worker running on EC2 and the Heroku app - this is due to how RQ works and serializes the job function + parameters.

If you use ASYNC endpoints in Flask, arq is an alternative to RQ with async support. Arq offers more flexibility in not having the remote worker share the same codebase as the queuing Flask app.

Depending on the goal, the Flask application and worker can run directly from Heroku without needing EC2.