Running multiple instances of Tornado gives ioloop is already running error

Question

This is how my code looks like, (have excluded some details that aren't really relevant)

from multiprocessing.pool import ThreadPool as Pool

class GetUsers(BaseTask):
    def foo(self):
        pool = Pool()
        try:
            pool.map(self.bar, users)
        finally:
            pool.close()
            pool.join()

    def bar(self, users):
        uuid = users[0]
        ioloopInstance = ioloop.IOLoop().instance()
        isInExperiment = self.isInExperiment(uuid, ioloopInstance)
        log.info(str(uuid)+str(isInExperiment))

    def isInExperiment(self, uuid, ioloop):
        isInExpTag_response =ioloop.run_sync(lambda: self.
                                             fetch_isInExperiment_response(uuid))
        if len(isInExpTag_response.body) > 0:
            return True
        return False

    @gen.coroutine
    def fetch_isInExperiment_response(self, uuid):
        response = yield baz
        raise gen.Return(response)

When I run this, I get ioloop is already running error. I feel that this is because several process running are trying to access the same instance of Tornado and so this error is seen. I have tried reading the documentation of tornado and seen other resources online trying to tackle the same error, but could not find anything helpful.

Can anyone please help me out?

You can't call `run_sync` on a running loop. That's what's causing the error. Either use couroutines and `yield` everywhere OR use callbacks everywhere. You are unnecessarily complicating your code by mixing the two approaches. — xyres, Mar 20 '20 at 17:16
using run_sync in a loop worked when I wasn't multiprocessing though. Will try this. — Swastik Udupa, Mar 21 '20 at 01:49

score 2 · Accepted Answer · answered Mar 24 '20 at 11:54

This worked after I changed from multiprocessing.pool import ThreadPool as Pool to from multiprocessing import Pool

I had used to Threadpool as a workaround for fixing Can't pickle <type 'instancemethod'> when using multiprocessing Pool.map() this error, but went on to use copy_reg as defined here https://laszukdawid.com/2017/12/13/multiprocessing-in-python-all-about-pickling/ to resolve the entire issue.

Running multiple instances of Tornado gives ioloop is already running error

1 Answers1