Django Channels 2.4 and Websockets giving 502 error on Elastic Beanstalk and ElastiCache

Question

I have integrated a chat component into my django web app via django channels in a very similar way as documented here. I used elasticache to create my redis instance. My settings.py looks as follows:

ASGI_APPLICATION = 'MyApp.routing.application'
CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.core.RedisChannelLayer',
        'CONFIG': {
            "hosts": [('my-redis-instance.cache.amazonaws.com', 6379)], #This is a placeholder in the question for the cluster's actual endpoint
        },
    },
}

In deploying my application to elastic beanstalk, I have tried following this tutorial and this tutorial with no luck. My django.config file currently looks as follows:

container_commands:
    01_collectstatic:
        command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput"
    02_migrate:
        command: "django-admin.py migrate"
        leader_only: true
    03_load-data:
        command: "python manage.py load_database"
        leader_only: true

option_settings:
    aws:elasticbeanstalk:application:environment:
        DJANGO_SETTINGS_MODULE: MyApp.settings
    aws:elasticbeanstalk:container:python:
        WSGIPath: MyApp/wsgi.py
    "aws:elasticbeanstalk:container:python:staticfiles":
        /static/: "static/"
    aws:elbv2:listener:80:
        DefaultProcess: http
        ListenerEnabled: 'true'
        Protocol: HTTP
        Rules: ws
    aws:elbv2:listenerrule:ws:
        PathPatterns: /ws/*
        Process: websocket
        Priority: 1
    aws:elasticbeanstalk:environment:process:http:
        Port: '80'
        Protocol: HTTP
    aws:elasticbeanstalk:environment:process:websocket:
        Port: '5000'
        Protocol: HTTP

I also tried creating a Procfile to configure gunicorn and daphne. It looks like this:

web: gunicorn --bind :8000 --workers 3 --threads 2 MyApp.wsgi:application
websocket: daphne -b 0.0.0.0 -p 5000 MyApp.asgi:application

The security groups that are attached to my ElastiCache redis instance have an inbound rule with Custom TCP set to port 6379 and the source set to Any. In trying all these different deployment methods, I have continued to get the following error:

[Error] WebSocket connection to 'ws://my-site-url.com/ws/messages/' failed: Unexpected response code: 502

I am not sure what to do. I have seen several stack overflow posts related to this, but none have helped. This post seemed to have a valid solution, but I believe it is now outdated.

Update:

In doing some more searching, I created a websockets-python.config file in my .ebextensions folder that is supposed to configure the Apache proxy server to allow for the use of web sockets. I also removed the Procfile that configured Daphne and Gunicorn. The websockets-python.config file looks like so:

files:
  "/etc/httpd/conf.d/proxy-pass.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      ProxyPass /ws/ ws://127.0.0.1:5000/
      ProxyPassReverse /ws/ ws://127.0.0.1:5000/

  "/etc/httpd/conf.modules.d/99-mod_proxy_wstunnel.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      <IfModule !proxy_wstunnel_module>
      LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
      </IfModule>

After redeploying with this file, I got an invalid state error on the web socket and then the web socket connection eventually timed out. I believe that the web socket is trying to connect, but is unable to for some reason (maybe it's how my security groups are configured) and then eventually times out.

Security Group Configuration:

ELB: Inbound rule from anywhere on port 80. Outbound rule from port 80 and 5000 to anywhere

My Server: Inbound rule from ELB on port 80 and 5000. Outbound rule of All Traffic to Anywhere.

ElastiCache Instance: Inbound rule on port 6379 from anywhere and on port 6439 from server. Outbound rule of All Traffic to Anywhere.

Answer 1

When using ElastiCache, you have to use the cluster's primary endpoint. You can go to ElastiCache's dashboard, go to Redis, and click on the dropdown. Find the primary endpoint and use that instead of my-redis-instance.cache.amazonaws.com

Edit since the endpoint may just be correct: the problem may be in your security group on ElastiCache. The ElastiCache cluster should only allow inbound from your server's SG (on port 6439), which only allows inbound from your ELB's security group (on port 5000?), which only allows inbound from port 443 and port 80 from anywhere.

Finally, I find it odd you're configuring Gunicorn and Daphne. Aren't they one in the same except Daphne can run ASGI? (Only a Uvicorn worker with Gunicorn allows for ASGI; you should just use Daphne and don't enable Gunicorn).

This is all in theory that your AWS instance has enough memory and CPU; that too can be a problem. I'd recommend Sentry for devs (free) so that you can take a look. A lot of my 502 errors were solved because of some neat errors.

Answer 2

I found a solution to this after a few days of work. I couldn't make the port forwarding on the load balancer work, however i configured mine to forward to port 5000 on Nginx.

redis url:

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [(REDIS_URL, 6379)],
        },
    },
}

Nginx config:

.platform/nginx/nginx.conf

#Elastic Beanstalk Nginx Configuration File

user                    nginx;
error_log               /var/log/nginx/error.log warn;
pid                     /var/run/nginx.pid;
worker_processes        auto;
worker_rlimit_nofile    32137;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    include       conf.d/*.conf;

    map $http_upgrade $connection_upgrade {
        default     "upgrade";
    }

    server {
        listen        80 default_server;
        access_log    /var/log/nginx/access.log main;

        client_header_timeout 60;
        client_body_timeout   60;
        keepalive_timeout     60;
        gzip                  off;
        gzip_comp_level       4;
        gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        # Include the Elastic Beanstalk generated locations
        include conf.d/elasticbeanstalk/*.conf;
        

        location /ws {
            proxy_pass          http://127.0.0.1:5000;
            proxy_http_version  1.1;

            proxy_set_header    Connection          $connection_upgrade;
            proxy_set_header    Upgrade             $http_upgrade;
            proxy_set_header    Host                $host;
            proxy_set_header    X-Real-IP           $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
            proxy_set_header    X-Forwarded-Proto   https;
        }
    }

}

Procfile:

web: gunicorn --bind 127.0.0.1:8000 --workers=1 --threads=15 project.wsgi:application
websocket: daphne -b 0.0.0.0 -p 5000 project.asgi:application

As for security group, when creating redis, on the security group you need to add the id of the security group currently being used on your EC2 instance, in the destination field for inbound.