Python Thread not returning the value

Question

Using: Django with Python

Overall objective: Call a function which processes video conversion (internally makes a curl command to the media server) and should immediately return back to the user.

Using message queue would be an overkill for the app. So I had decided to use threads, I have written a class which overwrites the init and run method and calls the curl command

class process_video(Thread):
    def __init__ (self,video_id,video_title,fileURI):
        Thread.__init__(self)
        self.video_id = video_id 
        self.video_title = video_title
        self.fileURI = fileURI
        self.status =-1

    def run(self):
        logging.debug("FileURi" + self.fileURI)
        curlCmd = "curl --data-urlencode \"fileURI=%s\" %s/finalize"% (self.fileURI, settings.MEDIA_ROOT)
        logging.debug("Command to be executed" + str(curlCmd))
        #p = subprocess.call(str(curlCmd), shell=True)
        output_media_server,error = subprocess.Popen(curlCmd,stdout = subprocess.PIPE).communicate()
        logging.debug("value returned from media server:")
        logging.debug(output_media_server)

And I instantiate this class from another function called createVideo which calls like this success = process_video(video_id, video_title, fileURI)

Problem: The user gets redirected back to the other view from the createVideo and the processVideo gets called, however for some reason the created thread (process_video) doesn't wait for the output from the media server.

I need some more information here. What does the code using `process_video` look like? I'm assuming you do something more than just create an instance of `process_video`. — Arlaharen, Nov 22 '10 at 06:52
subprocess.Popen call seems to get stuck as it doesn't seem to log any error or either output. I also checked the Media server logs and no calls are being made to it. Setting the instance of process_video as daemon doesn't help too. Any ideas? — Pranav Garg, Nov 22 '10 at 14:20

score 0 · Answer 1 · edited May 23 '17 at 12:29

I wouldn't rely on threads being executed correctly within web applications. Depending on the web server's MPM, the process that executes the request might get killed after a request is done (I guess).

I'd recommend to make the media server request synchronously but let the media server return immediately after it started the encoding without problems (if you have control over its source code). Then a background process (or cron) could poll for the result regularly. This is only one solution - you should provide more information about your infrastructure (e.g. do you control the media server?).

Also check the duplicates in another question's comments for some answers about using task queues in such a scenario.

BTW I assume that no exception occurs in the background thread?!

score 0 · Answer 2 · answered Nov 22 '10 at 18:15

Here is the thing what I did for getting around the issue which I was facing.

I used django piston to create an API for calling the processvideo with the parameters passed as GET, I was getting a 403 CSRF error when I was trying to send the parameters as POST. and from the createVideo function I was calling the API like this cmd = "curl \"%s/api/process_video/?video_id=%s&fileURI=%s&video_title=%s\" > /dev/null 2>&1 &" %(settings.SITE_URL, str(video_id),urllib.quote(fileURI),urllib.quote(video_title))

and this worked. I feel it would have been helpful if I could have got the session_id and post parameters to work. Not sure how I could get off that csrf thing to work.

Python Thread not returning the value

2 Answers2