0

I'm currently using tornado to develop an API that will have multiple users. For performance reasons, it would be beneficial for the entire API to be asynchronous. In the current code, I utilize cursor.fetchmany(# of rows) to fetch many lines from a database and write to a specified file (csv,json). I am attempting to make this process asynchronous but am having trouble doing so. The fetch deals with a large amount of data so if multiple people use the API, it will significantly slow it down as people are "waiting" for the fetch of other users to be completed.

I have tried using IOLoop with little success

Also I tried something like this 

async def loop(self):
        I = await val.cursor.fetchmany(100)

But there are errors with this because the function fetchmany isn't a native tornado function and isn't structured to support this. Some of my questions are...

1)Has anyone been able to implement a solution to make this async fetching process possible

2) Is this even possible or does the database limit the functionality of tornado asynchronous coding

3) If this is not possible, could I implement something like "ThreadPoolExecutor"to speed up the process by multithreading, or using a substitute for fetchmany? (I don't think this is possible but just trying to outline some of my thought process)

Any help would be greatly appreciated.

Adam
  • 1
  • 1
    Possible duplicate of [How to make SQLAlchemy in Tornado to be async?](https://stackoverflow.com/questions/16491564/how-to-make-sqlalchemy-in-tornado-to-be-async) – Ben Darnell Oct 11 '18 at 01:28

1 Answers1

1

Generally I wrapper the SQLAlchemy function in other method with @gen.coroutine decorator, so it can be an asynchronous method.

E.g.:

@gen.coroutine
def fetchmany():
   return val.cursor.fetchmany(100)

@gen.coroutine
def loop(self):
   I = yield fetchmany()

I hope to help you

Thiago Barros
  • 66
  • 5