Django Subprocess live/unbuffered reporting of stdout from batch scripts

Question

Basically the back story is that i've built a selecting of python scripts for a client to process importing and exporting batch jobs between their operations database and their ecomms site database. this works fine. these scripts write to stdout to update the user on the status of the batch script.

I'm trying to produce a framework for these scripts to be run via a django view and to post the stdout to the webpage to show the user the progress of these batch processes.

the plan was to - call the batch script as a subprocess and then save stdout and stderr to a file. - return a redirect to a display page that will reload every 2 seconds and display line by line the contents of the file that stdout is being written to.

however the problem is, that the stdout/stderr file is not being actually written to until the entire batch script has finished running or errors out.

i've tried a number of things, but none seem to work.

heres the current view code.

def long_running(app, filename):
    """where app is ['command', 'arg1', 'arg2'] and filename is the file used for output"""
    # where to write the result (something like /tmp/some-unique-id)
    fullname = temppath+filename
    f = file(fullname, "a+")
    # launch the script which outputs something slowly
    subprocess.Popen(app, stdout=f, stderr=f)# .communicate()
    # once the script is done, close the output
    f.close()

def attributeexport(request):
    filename = "%d_attribute" %(int(time.time())) #set the filename to be the current time stamp plus an identifier
    app = ['python','/home/windsor/django/applications/attribute_exports.py']
    #break thread for processing.
    threading.Thread(target=long_running, args=(app,filename)).start()
    return HttpResponseRedirect('/scripts/dynamic/'+filename+'/')
    pass

def dynamic(request, viewfile):
    fileobj = open(temppath+viewfile, 'r')
    results = []
    for line in fileobj:
        results.append(line)
        if '~END' in line:
            #if the process has completed
            return render_to_response('scripts/static.html', {'displaylist':results, 'filename':viewfile})
    return render_to_response('scripts/dynamic.html', {'displaylist':results, 'filename':viewfile})
pass

Your problem is output buffering; this may help you, or may not since you're not reading the output directly: http://stackoverflow.com/questions/803265/getting-realtime-output-using-subprocess — eternicode, Nov 19 '10 at 06:26
i think i might have to look into processing the output via the original script, setting key points to save an updated log file and pass the file name as an argument to the original script... i was hoping to avoid doing this as there are a lot of these scripts that will have to change. — BenDog, Nov 22 '10 at 05:02

score 1 · Accepted Answer · edited May 26 '12 at 06:47

1

It helps if you use the following:

['python','-u','path/to/python/script.py']

edited May 26 '12 at 06:47

Mat

202,337
40
393
406

answered Nov 23 '10 at 06:23

BenDog

366
1
13

2

Note that if you need the autoreloader, you should use the `PYTHONUNBUFFERED` environment variable since the autoreloader doesn't pass Python-level CLI arguments to the subprocess. – Joe Apr 25 '14 at 17:02

Django Subprocess live/unbuffered reporting of stdout from batch scripts

1 Answers1