Django CSRF cookie not set correctly

Question

Update 7-18:

Here is my nginx config for the proxy server:

server {
    listen 80;
    server_name blah.com; # the blah is intentional

    access_log /home/cheng/logs/access.log;     
    error_log /home/cheng/logs/error.log;       

    location / {
        proxy_pass http://127.0.0.1:8001;         
    }

    location /static {
        alias /home/cheng/diandi/staticfiles;  
    }

    location /images {
        alias /home/cheng/diandi/images;
    }

    client_max_body_size 10M;
}

Here is nginx.conf:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip_disable "msie6";

        # Enable Gzip compressed.
        gzip on;

        # Enable compression both for HTTP/1.0 and HTTP/1.1.
        gzip_http_version  1.1;

        # Compression level (1-9).
        # 5 is a perfect compromise between size and cpu usage, offering about
        # 75% reduction for most ascii files (almost identical to level 9).
        gzip_comp_level    5;

        # Don't compress anything that's already small and unlikely to shrink much
        # if at all (the default is 20 bytes, which is bad as that usually leads to
        # larger files after gzipping).
        gzip_min_length    256;

        # Compress data even for clients that are connecting to us via proxies,
        # identified by the "Via" header (required for CloudFront).
        gzip_proxied       any;

        # Tell proxies to cache both the gzipped and regular version of a resource
        # whenever the client's Accept-Encoding capabilities header varies;
        # Avoids the issue where a non-gzip capable client (which is extremely rare
        # today) would display gibberish if their proxy gave them the gzipped version.
        gzip_vary          on;

        # Compress all output labeled with one of the following MIME-types.
        gzip_types
                application/atom+xml
                application/javascript
                application/json
                application/rss+xml
                application/vnd.ms-fontobject
                application/x-font-ttf
                application/x-web-app-manifest+json
                application/xhtml+xml
                application/xml
                application/x-javascript
                font/opentype
                image/svg+xml
                image/x-icon
                text/css
                text/plain
                text/javascript
                text/js
                text/x-component;

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

---

Update 7-15:

When copying code to the linux machines, I simply replaced the original source code file but didn't delete the old .pyc files which I don't think will cause trouble right?

---

Here is the view code:

from django.contrib.auth import authenticate, login
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
from django.shortcuts import render

def login_view(request):
    if request.method == 'POST':
        username = request.POST['username']
        password = request.POST['password']
        user = authenticate(username=username, password=password)
        next_url = request.POST['next']
        if user is not None:
            if user.is_active:
                login(request, user)
                if next_url:
                    return HttpResponseRedirect(next_url)
                return HttpResponseRedirect(reverse('diandi:list'))
        else:
            form = {'errors': True}
            return render(request, 'registration/login.html', {'form': form})

    else:
        form = {'errors': False}
        return render(request, 'registration/login.html', {'form': form})

---

I got one of those CSRF cookie not set error from Django, but this is not because I forgot to include the {% csrf_token %} in my template.

Here is what I observed:

Access login page #1 try

Inside the Request Header, the cookie value is:

csrftoken=yNG8ZmSI4tr2xTLoE9bys8JbSuu9SD34;

In the template:

<input type="hidden" name="csrfmiddlewaretoken" value="9CVlFSxOo0xiYykIxRmvbWyN5iEUHnPB">

In a cookie plugin that I installed on chrome, the actual csrf cookie value is set to:

9CVlFSxOo0xiYykIxRmvbWyN5iEUHnPB

Access login page #2 try:

Inside the Request Header, the cookie value is:

csrftoken=9CVlFSxOo0xiYykIxRmvbWyN5iEUHnPB;

In the template:

<input type="hidden" name="csrfmiddlewaretoken" value="Y534sU40S8iTubSVGjjh9KQl0FXesVsC">

In a cookie plugin that I installed on chrome, the actual csrf cookie value is set to:

Y534sU40S8iTubSVGjjh9KQl0FXesVsC

The pattern

As you can see from the examples above, the cookie value inside the Request Header differs from the actual csrfmiddlewaretoken in the form and the actual cookie value being set.

The cookie value of the current request matches the next request header's cookie value.

---

To help debugging, here is a portion of my `settings.py:

DJANGO_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
)

THIRD_PARTY_APPS = (
    'compressor',
    'crispy_forms',
    'django_extensions',
    'floppyforms',
    'multiselectfield',
    'admin_highcharts',
)

LOCAL_APPS = (
    'diandi_project',
    'uer_application',
)

INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS

MIDDLEWARE_CLASSES = (
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [str(ROOT_DIR.path('templates'))],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.media',
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

I am using Django 1.9.5 and python 2.7.10.

One "solution"

I have encountered this problem before, I can clear all my browser cookies and the site will function properly. But this problem will eventually come up again, so I am really hoping someone can help me out (I probably just made a really dumb mistake somewhere).

Update

Originally, I thought I made some mistakes while overriding the django.contrib.auth.view page, so I wrote my own login page handler and it still causes the issue.

Here is the core part of my login template:

{% block content %}
...

                <form method="post" action="{% url 'login' %}">
                    {% csrf_token %}

                    <div class="form-group">
                        <label for="username">username</label>
                        <input type="text" class="form-control" id="id_username" name="username">
                    </div>
                    <div class="form-group">
                        <label for="password">password</label>
                        <input type="password" class="form-control" id="id_password" name="password">
                    </div>

                    <input type="submit" class="btn btn-default" value="login" />
                    <input type="hidden" id="next" name="next" value="" />
                </form>

...

{% endblock %}

On the Linux machines, I have a nginx server setup as a reverse proxy which direct request on port 80 to 8001, and I am running the server using ./manage runserver localhost:8001 This is the only difference I can think of in terms of setup. Otherwise, all of the source code and settings file are identical.

---

I started deleting cookies but not all of them, this is what I see before deleting them:

I deleted all the cookies other than djdt and csrftoken, then the page worked. Could the deleted cookies somehow go over some limit which prevent the csrftoken which is further down the list from being set?

Here is the cookie value of the image above in the request header:

Cookie:PSTM=1466561622; BIDUPSID=6D0DDB8084625F2CEB7B9D0F14F93391; BAIDUID=326150BF5A6DFC69B6CFEBD67CA7A18B:FG=1; BDSFRCVID=Fm8sJeC62leqR8bRqWS1u8KOKg9JUZOTH6ao6BQjXAcTew_mbPF_EG0PJOlQpYD-hEb5ogKK0mOTHvbP; H_BDCLCKID_SF=tJPqoCtKtCvbfP0k-tcH244HqxbXq-r8fT7Z0lOnMp05EnnjKl5M3qKOqJraJJ585Gbb5tOhaKj-VDO_e6u-e55LjaRh2PcM2TPXQ458K4__Hn7zep0aqJtpbt-qJjbOfmQBbfoDQCTDfho5b63JyTLqLq5nBT5Ka26WVpQEQM5c8hje-4bMXPkkQN3T-TJQL6RkKTCyyx3cDn3oyToVXp0njGoTqj-eJbA8_CtQbPoHHnvNKCTV-JDthlbLetJyaR3lWCnbWJ5TMCo1bJQCe-DwKJJgJRLOW2Oi0KTFQxccShPC-tP-Ll_qW-Q2LPQfXKjabpQ73l02VhcOhhQ2Wf3DM-oat4RMW20jWl7mWPQDVKcnK4-Xj533DHjP; BDUSS=5TNmRvZnh2eUFXZDA5WXI5UG1HaXYwbzItaWt3SW5adjE1Nn5XbUVoWHZuYXBYQVFBQUFBJCQAAAAAAAAAAAEAAAC0JtydAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO8Qg1fvEINXSU; Hm_lvt_a7708f393bfa27123a1551fef4551f7a=1468229606; Hm_lpvt_a7708f393bfa27123a1551fef4551f7a=1468229739; BDRCVFR[feWj1Vr5u3D]=I67x6TjHwwYf0; BDRCVFR[dG2JNJb_ajR]=mk3SLVN4HKm; BDRCVFR[-pGxjrCMryR]=mk3SLVN4HKm; cflag=15%3A3; H_PS_PSSID=1424_20515_13289_20536_20416_19861_14994_11792; csrftoken=xUgSHybzHeIwusN0GvMgB1ATeRrPgcV1

Since the site functions now, all I have are five cookies instead of 14 like the image above:

Answer 1

Here is the issue: You cannot have a cookie which key contains either the character '[' or ']'

I discovered the solution following @Todor's link, then I found out about this SO post. Basically there was a bug in python 2.7.x that does not parse cookies with ']' in the value. The bug was fixed in 2.7.10.

I thought it would be good to just confirm this issue. So I dug through all of the cookies and found one with the following key/value:

key: BDRCVFR[feWj1Vr5u3D]
val: I67x6TjHwwYf0

So I inserted the following cookie locally and submitted to the server:

key: test
val: BDRCVFR[feWj1Vr5u3D]

The login page worked, which means 2.7.10 indeed fixed the bug.

But then I realized that the square brackets are actually in the key name not in the value, so I did the following tests:

key: [
val: I67x6TjHwwYf0

and

key:]
val: I67x6TjHwwYf0

Both cookies break the login process and django displays:

CSRF cookie not set

So either django or a python library it relies on cannot parse cookies with square brackets in names properly. If anybody knows where I should submit this bug please let me know (django or python).

I would like to thank everybody who left a comment in the OP: @raphv, @trinchet, @Phillip, @YPCrumble, @PeterBrittain and @Todor. Thank you guys so much for debugging with me!

---

Update: July 20, 2016

This bug is fixed in Django 1.10, just have to wait for the release

Update: July 19, 2016

I filed a bug report to Django as the result of this post. We will see if it will be fixed in future releases.