Django get data from user, process it offline to send CSV to user

Question

I may not be able to provide you anything regarding code I tried. But none of searches or research could get me a working result.

What I am trying to do? I have a form where user submit a CSV file. This file is processed in backend to retrieve data from DB. Data recovered from DB is again written in CSV and mailed to user.

This processing takes around 5-10 minutes. So I want to show a message to user that your work is under process and you will get CSV file once it is completed.

My code as of now.

def bulkserachpro(request):
    with request.FILES['searchnum'] as csvfile:
                spamreader = csv.reader(csvfile,dialect=excel)
                next(spamreader)
                a = [] 
                b = []
                for row in spamreader:
                    a.append((row[0]).upper())
                data_list = nummodel.objects.filter(number__in=a)
                csvfile = StringIO.StringIO()
                csvwriter = csv.writer(csvfile)
                csvwriter.writerow(['a','b'])
                for data in data_list:
                    csvwriter.writerow([data.a,data.b])
                message = EmailMessage("Hello","Your report","email@gmail.com",["emailto@gmail.com"])
                message.attach('invoice.csv', csvfile.getvalue(), 'text/csv')
                message.send()
            return render(request,'bulkpro.html',messages.add_message(request, messages.SUCCESS,'File uploaded Succesfully.'))

This code does its job. But as user cannot wait for 5-10 minutes, he will get timeout error message. What should I change here so that user gets following message and once processing is done he receives email with attachment.

Your file is received, it will be processed and sent to your mail id.

score 0 · Answer 1 · answered May 21 '18 at 11:48

0

Waiting in the view will definitely timeout, any tricks to solve the issue is not straight forward. Instead do AJAX calls at intervals to know the status as below.

After posting the CSV, start a celery task which does the work.
AJAX query the status of task(task-id passed in previous call) to know the status of the job.
Display completion or progress of the task accordingly.
Send the email with CSV in celery task.

answered May 21 '18 at 11:48

Vinay P

617
3
13

I dont want to use celery. Can it be done with pure Django? I thought of sending file with Jquery-Ajax and alert message to user. But uploading file with Ajax is not possible it seems!!! – Sapna Sharma May 21 '18 at 11:52
Or let view process the file and send user a message that its processing. I do not want to track progress. – Sapna Sharma May 21 '18 at 11:53
Send the file using Form/FormURL post method but the processing of the file should be handled in celery. Waiting in a view for a long duration of time blocks lot of things, not advisable and possibility that it can close connection at any stage of pipeline(client browser->TCP-IP->WebServer->Django/Uwsgi) is high. – Vinay P May 21 '18 at 11:54
Ok lets put it in this way. User send file. Cut user connection and show him message processing will be emailed. Let Django do the processing and mail it in view. Its possible? – Sapna Sharma May 21 '18 at 11:56
When you initiate disconnect from client, django thread/view which is running will also be killed. – Vinay P May 21 '18 at 11:58

score 0 · Accepted Answer · answered May 21 '18 at 16:15

Although Celery is often used for this sort of thing, for this simple case you could potentially just run the csv processing in a thread.

For example, create a thread that encapsulates the processing code:

import threading

class ProcessCsv(threading.Thread):

    def __init__(self, csvfile):
        self.csvfile = csvfile
        self.daemon = True

    def run(self):
        spamreader = csv.reader(self.csvfile,dialect=excel)
        next(spamreader)
        a = [] 
        b = []
        for row in spamreader:
            a.append((row[0]).upper())
        data_list = nummodel.objects.filter(number__in=a)
        csvfile = StringIO.StringIO()
        csvwriter = csv.writer(csvfile)
        csvwriter.writerow(['a','b'])
        for data in data_list:
            csvwriter.writerow([data.a,data.b])
        message = EmailMessage("Hello","Your report","email@gmail.com",["emailto@gmail.com"])
        message.attach('invoice.csv', csvfile.getvalue(), 'text/csv')
        message.send()

And then your view can start the thread, and just display the "Your file is received, it will be processed..." message:

def bulkserachpro(request):
    process = ProcessCsv(request.FILES['searchnum'])
    process.start()  # Start the csv processing in a separate thread
    # Display "your file is being processed" message
    return render(request,'bulkpro.html',messages.add_message(request, messages.SUCCESS,'File uploaded Succesfully.'))

was reading this https://stackoverflow.com/a/3044626/9362919 Will threading create any issue if multiple users use this function at same time? — Sapna Sharma, May 21 '18 at 17:08
Using threads should be fine in this case. It's only an issue if the threads are sharing data, in which case you need to control access using a lock. In your case, the threads are not sharing data, they read from their own separately uploaded csv file and write to their own local variables. — Will Keeling, May 21 '18 at 17:57
You really really should be doing this. Unless you REALLY know your threading (and 99% of people either don't or worse dont know it as well as they think they do) AND you really understand Djangos lifecycle , your in for a world of pain. Keep it siloed out in Celery or something similar thats specifically designed for the task. And try and use Djangos file backends, or your going to have a hard time with scaleability (especially if you need to change things out to S3 or whatever). — Shayne, Nov 15 '20 at 06:50

Django get data from user, process it offline to send CSV to user

2 Answers2