uWSGI and Django segmentation fault

Question

I've been dealing with a strange error that is intermittent and hard to trace. One of the pages on my django-powered site allows users to download music they have purchased.

The problem: Some users are reporting a 502 error upon downloading. A further investigation of the problem shows that there is a segmentation fault in uWSGI. I don't really know how to debug this, and am looking for anything that might help find a resolution.

Attempts to solve the problem:
I thought it may be that I was running too many uWSGI processes, so I've lowered that, but since the problem only happens sometimes (some users have no problem, some try again with success), I can't be sure the error is resolved.

Thoughts on the issue:
I am thinking it might be due to the way I'm handling the files here. The files that I'm zipping are also being sent to other users in other views. Could it be an issue when the uWSGI processes are both trying to read a file? No processes write to the files.

Next steps:
I'm looking for some guidance on this, or any ideas on how I can get a little more information on what's going on here.

uWSGI config:

[uwsgi]
plugin=/etc/uwsgi/python_plugin.so
wsgi-file           = /path/to/my/project/projectname/wsgi.py
home                = /path/to/my/project/
master              = true
socket              = /tmp/uwsgi.sock
chmod-socket        = 666
vacuum              = true
processes           = 3
workers             = 15
min-worker-lifetime = 45
max-requests        = 100
reload-mercy        = 5
harakiri            = 20
buffer-size         = 16384

Relevant view:

def zipForDownload(album):

    bonus = BonusContent.objects.filter(album=album)
    bonus_files = [open(f.bonus_file.path, 'rb') for f in bonus]

    tracks = trackSort(list(Track.objects.filter(album=album)))
    track_files = [open(f.audio_file.path, 'rb') for f in tracks]


    zipped_file = StringIO.StringIO()
    with zipfile.ZipFile(zipped_file, 'w') as zip:
        for i, f in enumerate(track_files):
            f.seek(0)
            num = tracks[i].track_number
            name = tracks[i].name
            ext = os.path.basename(f.name).split('.')[-1]
            zip.writestr("{0} - {1}.{2}".format(num, name, ext), f.read())
        for i, f in enumerate(bonus_files):
            name = bonus[i].name
            ext = os.path.basename(f.name).split('.')[-1]
            zip.writestr("{0}.{1}".format(name, ext), f.read())

    zipped_file.seek(0)
    response = HttpResponse(zipped_file, content_type='application/octet-stream')
    response['Content-Disposition'] = 'attachment; filename=%s.zip' % (album.name)
    return response

def downloadPostPin(request, purchase):
    if request.POST.get('PIN').encode('utf-8') == purchase.download_pin.encode('utf-8'):
        # Get zip response with all tracks and bonus content
        resp = zipForDownload(purchase.album)
        return resp

    else:
        # Otherwise, return the same page again with an error
        error = "Invalid PIN. Please try again!"
        return downloadPrePin(request, purchase, error)

uWSGI error log:

!!! uWSGI process 8094 got Segmentation Fault !!!
*** backtrace of 8094 ***
/home/web/.envs/music/bin/uwsgi(uwsgi_backtrace+0x2e) [0x46a1be]
/home/web/.envs/music/bin/uwsgi(uwsgi_segfault+0x21) [0x46a581]
/lib/x86_64-linux-gnu/libc.so.6(+0x36c30) [0x7fdc1ffd6c30]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Malloc+0x248) [0x7fdc206ef508]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyString_FromStringAndSize+0xa2) [0x7fdc206de232]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3be7) [0x7fdc206fe0e7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8) [0x7fdc206fedd8]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x162310) [0x7fdc20701310]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x7d30d) [0x7fdc2061c30d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fdc20687837]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff706) [0x7fdc2069e706]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2920) [0x7fdc206fce20]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x161883) [0x7fdc20700883]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a7d02) [0x7fdc20746d02]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PySequence_List+0x2c) [0x7fdc207471bc]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PySequence_Fast+0x3d) [0x7fdc2074807d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1aa995) [0x7fdc20749995]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4abc) [0x7fdc206fefbc]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x162310) [0x7fdc20701310]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_CallFunction+0xbb) [0x7fdc20706efb]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x16a15d) [0x7fdc2070915d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyObject_GenericSetAttrWithDict+0x107) [0x7fdc205f00b7]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_SetAttr+0x8f) [0x7fdc206aeadf]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1cda) [0x7fdc206fc1da]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1623e5) [0x7fdc207013e5]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x7d30d) [0x7fdc2061c30d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x12e48f) [0x7fdc206cd48f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x12c4df) [0x7fdc206cb4df]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2316) [0x7fdc206fc816]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7fdc206ff059]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7fdc206ff059]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1623e5) [0x7fdc207013e5]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xeb1) [0x7fdc206fb3b1]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8) [0x7fdc206fedd8]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x162310) [0x7fdc20701310]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x7d30d) [0x7fdc2061c30d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x12e5f5) [0x7fdc206cd5f5]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fdc20687837]
/home/web/.envs/music/bin/uwsgi(python_call+0x11) [0x4808a1]
/home/web/.envs/music/bin/uwsgi(uwsgi_request_wsgi+0x116) [0x482a96]
/home/web/.envs/music/bin/uwsgi(wsgi_req_recv+0xa2) [0x41f1f2]
/home/web/.envs/music/bin/uwsgi(simple_loop_run+0xc4) [0x466664]
/home/web/.envs/music/bin/uwsgi(uwsgi_ignition+0x194) [0x46a7d4]
/home/web/.envs/music/bin/uwsgi(uwsgi_worker_run+0x2dd) [0x46f02d]
/home/web/.envs/music/bin/uwsgi(uwsgi_run+0x3b4) [0x46f554]
/home/web/.envs/music/bin/uwsgi(_start+0) [0x41e8ae]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fdc1ffc1ec5]
*** end of backtrace ***
DAMN ! worker 9 (pid: 8094) died :( trying respawn ...
Respawned uWSGI worker 9 (new pid: 8371)

score 3 · Accepted Answer · edited May 23 '17 at 12:23

The problem happens because your process fails to allocate a chunk of memory.

When I first look at your code, the first thing that drew my attention was that you are loading entire content of file into memory and read them all at once at each iteration.

zip.writestr("{0} - {1}.{2}".format(num, name, ext), f.read())

f.read() here will read the entire content of the file all at once and put that into memory, then will try to write it into another section of the memory, the StringIO object, which in turn ending up storing same data twice in different parts of the RAM.

I would recommend you to read from file one line at a time or at least reasonable amount of data and don't forget to close the file when you are done with it:

for i, f in enumerate(track_files):
    f.seek(0)
    ...
    while True:
        data = f.read(2**16)
        if not data:
            break
        else:
            zip.write(data)
    ...
    f.close()

You might want to take a look at this SO answer about writing to a zipfile.ZipFile instance chunk by chunk.

Beautiful. Thanks for providing such a complete answer. – Jamie Counsell Feb 22 '15 at 18:17 — Jamie Counsell, Feb 22 '15 at 18:17

uWSGI and Django segmentation fault

1 Answers1