Python Django truncates the response

Question

I've created a little Django application and deployed it on Ubuntu, views.py

from django.shortcuts import render
import json
import psycopg2
import csv
from django.http import HttpResponse
from ratelimit.decorators import ratelimit
def create_connection():
    return psycopg2.connect(database="data",user="ag",host="localhost",port=5432,password='pass')

def execute_query(query, parameters):
    with create_connection() as connection:
        result = []
        cursor = connection.cursor()
        cursor.execute(query, parameters)
        for row in cursor:
            result.append(row)
        for i in range(len(result)):
            result[i] = list(result[i])
            for j in range(len(result[i])):
                result[i][j] = result[i][j] if result[i][j] else ""
        return result

@ratelimit(key='ip', rate='10/m')
def to_csv(request):
    j_body = json.loads(request.GET.get('data', ''))
    query = j_body['query']
    print(query)
    print("****************************************************************")
    parameters = j_body['params']
    headers = j_body['headers']
    rows = execute_query(query, parameters)
    response = HttpResponse(content_type='text_csv')
    response['Content-Disposition'] = 'attachment; filename="result.csv"'
    writer = csv.writer(response)
    writer.writerow(headers)
    writer.writerows(rows)
    return response

The main goal of this app - to get sql query and return csv response as file, which size can be more than 200 mb. I have a problem - my response is always truncated to 696 kb (regardless on count of rows in result of cursor.execute). There is example. You can see the last truncated line.

Server was ran by 'python3 manage.py runserver 0.0.0.0:8000' command. Please, help me to solve my problem.

UPDATED I redeployed it with Nginx and gunicorn, and tried solution suggested here ngnix + gunicorn throws truncated response body Despite this, I am still getting the same error

I fixed my code, following this article - https://docs.djangoproject.com/en/1.10/howto/outputting-csv/#streaming-large-csv-files

#Also changed execute_query method, so it doesn't create redurant copy of response from database
def execute_query(query, parameters):
    with create_connection() as connection:
        result = []
        cursor = connection.cursor()
        cursor.execute(query, parameters)
        return cursor

class Echo(object):
    """An object that implements just the write method of the file-like
    interface.
    """
    def write(self, value):
        """Write the value by returning it, instead of storing in a buffer."""
        return value

@ratelimit(key='ip', rate='10/m')
def to_csv(request):
    try:
        j_body = json.loads(request.GET.get('data', ''))
        query = j_body['query']
        print("****************************************************************")
        parameters = j_body['params']
        headers = j_body['headers']
        cursor = execute_query(query, parameters)
        pseudo_buffer = Echo()
        writer = csv.writer(pseudo_buffer)
        writer.writerow(headers)
        print('Writing began')
        response = StreamingHttpResponse((writer.writerow(row) for row in cursor),
                                     content_type="text/csv")
        response['Content-Disposition'] = 'attachment; filename="result.csv"'
        print('Sending began')
        return response
    except:
        return HttpResponse('Error')

Now all work fine.

score 1 · Accepted Answer · answered Nov 29 '16 at 18:51

Sounds like you might need a StreamingHttpResponse. Have you tried implementing this? It appears that you are suffering from some sort of issue with a buffered/chunked response. Something like gunicorn or nginx could be used to increase this buffer size, but it is not a real solution.

Python Django truncates the response

1 Answers1