62

I'm trying to get the 'hello world' of streaming responses working for Django (1.2). I figured out how to use a generator and the yield function. But the response still not streaming. I suspect there's a middleware that's mucking with it -- maybe ETAG calculator? But I'm not sure how to disable it. Can somebody please help?

Here's the "hello world" of streaming that I have so far:

def stream_response(request):
    resp = HttpResponse( stream_response_generator())
    return resp

def stream_response_generator():
    for x in range(1,11):
        yield "%s\n" % x  # Returns a chunk of the response to the browser
        time.sleep(1)
muudscope
  • 6,780
  • 4
  • 21
  • 20

2 Answers2

52

You can disable the ETAG middleware using the condition decorator. That will get your response to stream back over HTTP. You can confirm this with a command-line tool like curl. But it probably won't be enough to get your browser to show the response as it streams. To encourage the browser to show the response as it streams, you can push a bunch of whitespace down the pipe to force its buffers to fill. Example follows:

from django.views.decorators.http import condition

@condition(etag_func=None)
def stream_response(request):
    resp = HttpResponse( stream_response_generator(), content_type='text/html')
    return resp

def stream_response_generator():
    yield "<html><body>\n"
    for x in range(1,11):
        yield "<div>%s</div>\n" % x
        yield " " * 1024  # Encourage browser to render incrementally
        time.sleep(1)
    yield "</body></html>\n"
Pēteris Caune
  • 43,578
  • 6
  • 59
  • 81
Leopd
  • 41,333
  • 31
  • 129
  • 167
  • 5
    In my tests the Django GZipMiddleware can prevent this from working. – Xealot Dec 21 '10 at 05:55
  • 1
    Yeah, I expect a lot of middlewares are likely to mess with it, so if it's not working, try disabling all your middlewares and re-enable them incrementally. GZip requires having the entire response before it compresses it, so it won't let you stream. – Leopd Jan 03 '11 at 01:25
  • 1
    @Xealot: I'm having a similar problem with GzipMiddleware. Filed a bug because that middleware does not support generators (it actually clears the generator accidentally): [Django ticket #15066](http://code.djangoproject.com/ticket/15066) – AndiDog Jan 12 '11 at 23:54
  • 1
    I disabled the GZipMiddleware on this call by inserting a fake header into the StreamingHttpResponse: `response['Content-Encoding'] = 'identity'`. This is probably invalid HTTP in eighteen different ways, but it was just an internal endpoint in my case, so this was good enough for me. – Henrik Heimbuerger Oct 19 '15 at 21:57
  • Maybe a better way would be to subclass it and customize it to ignore all responses with a header X-Streaming or something like than on. – skywalker Jan 11 '16 at 15:31
43

A lot of the django middleware will prevent you from streaming content. Much of this middleware needs to be enabled if you want to use the django admin app, so this can be an annoyance. Luckily this has been resolved in the django 1.5 release. You can use the StreamingHttpResponse to indicate that you want to stream results back and all the middleware that ships with django is aware of this and acts accordingly to not buffer your content output but send it straight down the line. Your code would then look like the following to use the new StreamingHttpResponse object.

def stream_response(request):
    return StreamingHttpResponse(stream_response_generator())

def stream_response_generator():
    for x in range(1,11):
        yield "%s\n" % x  # Returns a chunk of the response to the browser
        time.sleep(1)

Note on Apache

I tested the above on Apache 2.2 with Ubuntu 13.04. The apache module mod_deflate which was enabled by default in the setup I tested will buffer the content you are trying to stream until it reaches a certain block size then it will gzip the content and send it to the browser. This will prevent the above example from working as desired. One way to avoid this is to disable mod_deflate by putting the following line in your apache configuration:

SetEnvIf Request_URI ^/mysite no-gzip=1

This is discussed more in the How to disable mod_deflate in apache2? question.

Community
  • 1
  • 1
Marwan Alsabbagh
  • 25,364
  • 9
  • 55
  • 65
  • It works! but how am I supposed to render it to the template `real-time` ? Currently I am using Javascript with an ajax call.. but it posts the output.. only when it's finished processing.. So it's not `Streaming` when it comes to rendering it to the `browser`. Any help would be appreciated.. Thank you :) – First Blood Nov 05 '13 at 20:56
  • @FirstBlood With ajax you can get callbacks as it is streaming and not just when the download is complete. Check out this answer http://stackoverflow.com/a/4488132/1699750 for showing progress on an Ajax call. – Marwan Alsabbagh Nov 06 '13 at 12:02
  • Thanks.. I want to achieve `Ping` output status.. to be printed real-time on the browser.. So for ex: `ping -c 3 www.google.com` .. how the command prompt gives the output.. is it possible in `Django`, to spill out the output to the browser as I read via `subprocess`? I tried the `StreamingHttpResponse`, but it doesn't seem to stream real-time.. I even tried setting `headers: Keep-Alive`.. but no effect :( – First Blood Nov 06 '13 at 20:30
  • your webserver (eg Apache) will likely also buffer the output, if the process is not very long running or doesn't send much output this may have the effect that the webserver buffers it all and then delivers it in one lump at the end – Anentropic Mar 29 '14 at 09:38
  • @Anentropic That is true. I have updated the answer with a note about Apache – Marwan Alsabbagh Mar 29 '14 at 13:19