0

I'm using Tornado Async framework for the implementation of a REST Web-Server.

I need to run a high-CPU-load periodic task on the background of the same server.

It is a low-priority periodic task. It should run all the time on all idle cores, but I don't want it to affect the performance of the Web-Server (under a heavy HTTP-request load, it should take lower priority).

Can I do that with the Tornado IOLoop API?

I know I can use tornado.ioloop.PeriodicCallback to call a periodic background task. But if this task is computational-heavy I may cause performance issues to the web service.

ShaharA
  • 860
  • 8
  • 19

1 Answers1

1

Tornado (or asyncio in python 3) or any other single-process-event-loop based solution is not meant to be used for CPU intensive tasks. You should use it only for IO intensive tasks.

The word "background" means, that you're not waiting for a result (I call it sometimes unattended task). More over, if a background task blocks, the rest of a application have to wait, the same way as a request handler blocks, the other parts, including the background, are blocked.

You might be thinking of use of threads, but in python this is not a solution either, due to GIL.

The right solutions are:

kwarunek
  • 12,141
  • 4
  • 43
  • 48
  • Thanks for the reply. If I decouple the worker from the web server, it means I can't let the worker run on the same server without worrying whether it may affect the webserver. So you suggest using another process on a different isolated core? ( I now let Tornado spawn a process-per-core) – ShaharA Sep 10 '17 at 19:18
  • I don't know what is your configuration. The kernel scheduler will handle processe management (and its resource) pretty well, but of course each process could make cpu-, io-, mem- starvation on others, solution could be to use cgroups (plain or with containers like docker). Again the key information is how much resource do you have and how much the worker, webserver needed, without tests you need crystal ball ;) – kwarunek Sep 12 '17 at 05:29