Django - How to properly send a request to server and ask to run a matlab program?

Question

I have a django web project on a server. I want it to run a matlab code to produce some text file(which will be used later). Here is my code:

if(request.method == "POST"):
    run_octave(Dataset,is_multiclass,require_mapper,mapper,require_aspect_ratio,aspect_ratio)
    return redirect('meta2db.views.meta2db')

def run_octave(dataset,is_multiclass,require_mapper,mapper,require_aspect_ratio,aspect_ratio):

    origWD = os.getcwd()
    args = ["octave", "dbEval.m",dataset,is_multiclass,require_mapper,\
    mapper,require_aspect_ratio,aspect_ratio]

    os.chdir(os.path.join(os.path.abspath(sys.path[0]), "../scripts/"))

    #subprocess call here
    process = subprocess.Popen(args, stdout=subprocess.PIPE)

    for line in process.stdout:
        time.sleep(0.5)
        Group("eval_status").send({"text": line.decode('utf-8')},immediately=True)

    if process.poll() is None:
        process.kill()
    else:
        print(process.communicate())

    os.chdir(origWD)

I ues a post request to run the octave code with subprocess call. However the matlab code take awhile to be finished and always make the client timeout which is not acceptable. My question is how to solve this kind of problem in another way. A post request seems not a good solution.

Answer 1

You could make it async.

And for async i mean that in the post request you will generate a uuid which identifies the operation. You will launch the operation inside the post request using a different thread/process.

To check if the operation is done, you will have to setup a status view where given the operation id will return his status.

Adding just a code snippet as a proof of concept. I did not test it!

from concurrent.futures import ThreadPoolExecutor
import uuid

class Operation(object):
    operations = {}

    def __init__(self, *task):
        self.id = str(uuid.uuid4())
        self.task = task
        self.done = False
        self._thread = None
        self._executor = ThreadPoolExecutor()
        self.__class__.operations[self.id] = self

    def run(self):
        self._thread = self.executor.submit(*self.task)
        self._thread.add_done_callback(self._callback)
        return self.id

    def _callback(self):
        self.done = True

    @classmethod
    def is_operation_done(cls, id):
        try:
            return cls.operations[id].done 
        except IndexError:
            raise Exception("Operation not found") # FIXME Custom exception here!!



if(request.method == "POST"):
    operatiorn = Operation(run_octave, 
                            Dataset, 
                            is_multiclass, 
                            require_mapper,
                            mapper,
                            require_aspect_ratio, 
                            aspect_ratio)
    id = operatiorn.run()
    return id # jsonize it if you wanna pass it to the frontend


def is_operatiorn_done(request, operation_id):
    # This is implementing a polling but would be bettere to have a socket!
    return Operation.is_operation_done(operation_id)

def run_octave(dataset,is_multiclass,require_mapper,mapper,require_aspect_ratio,aspect_ratio):
    .....
    .....

Answer 2

This would be an asyncronous operation. This is not built in to django by default, but there are several ways to make it possible.

The most common choice is probably to use Celery. This is a distributed task queue that can be combined with django. It also requires that you install a message broker such as RabbitMQ.

http://www.celeryproject.org/

A newer alternative is django channels, which is part of the django project, but not a default part of django (at least not yet).

https://github.com/django/channels

See this question for more comparison of the two projects. How Django channels are different than celery?

Both of these libraries are quite complex. If you are looking for more lightweight alternatives, see this question:

Simple approach to launching background task in Django