0

I have a situation where I have a processing function that can take a lot of time to run. This function takes a progress observer as an argument and notifies the observer about advancement.

I would like to send async http requests to report said progress, and return back to the processing function ASAP.

def longProcessingFunction(progressObserver):
    # processing in a loop ...
        progressObserver.updateProgress(x)
    # more processing ...

class MyClass():
    def runLongProcessingFunction(slef):
        results = longProcessingFunction(self)
        # send last request to report results (ideally *after* last progress message) like:
        # requests.patch(self.reportingUrl, json={'results': results, 'progress': 1.0})

    def updateProgress(self, progress):
        # send async http request to report progress like:
        # requests.patch(self.reportingUrl, json={'progress': progress})
        # ideally handle the response on next progress update
        # (if it arrived by that time - otherwise on the one after it)

I tried using asyncio, but it seems that its main intention is to use a main event loop, and add multiple async tasks to it. I even tried applying a somewhat bizarre run_once approach inside the updateProgress function, but apart from it really forcing my scenario on asyncio I couldn't get the app to wait for the last pending progress task at the end and send the results request after it (any wait for all tasks attempt fails since I'm never really in an asyncio loop - I'm using it to 'run once').

In my case, compared to how normal asyncio code works, whenever I'm running the longProcessingFunction() I'm blocked until it finishes processing (apart from the progress observer being called).

Looking for any reasonable architecture whether with asyncio or w/o it.

ArnonZ
  • 3,822
  • 4
  • 32
  • 42
  • 1
    Not sure if it is the best approach, but I would in `updateProgress` only write the progress to a queue and exit. Then, in another task, I would process what I find in the queue. Whether those two tasks are implemented as asyncio or threads, it should work in the same manner. – zvone Nov 20 '20 at 20:52
  • 2
    Whether or not asyncio will be helpful depends on what `longProcessingFunction` does. If processing involves a lot of I/O, asyncio can help; if it's CPU-intensive, asyncio won't help. You'll want to looking into using multiple processes instead. – dirn Nov 20 '20 at 21:43
  • Thanks, that's really helpful. For the record, it's a 100% CPU guzzler, weekend runner on relatively small inputs, shameless game theory algorithm I wish to understand someday. Surprising though that python would require another process to send a request and pickup the response later. – ArnonZ Nov 20 '20 at 23:08

0 Answers0