3

I want to know whether it is possible to limit the number of threads python interpreter can spawn when running a script.

I am not using threading myself, but my script uses modules like pandas, numpy, and others which I assume would be using threading in the background. The script works fine, but when I run it on an LSF cluster, it spawns too many threads crossing thread limit per job (for the cluster) and as a result, crashes.

I would like to restrict the python interpreter to not allow more than, say, 100, threads at a time. Is it possible to run a python script on an LSF server with such a restriction on thread count?

Thanks for your help.

P.S.: My colleague faces the same issue with java apps, so a solution for that would be cool as well.

Manish Goel
  • 843
  • 1
  • 8
  • 21
  • 1
    Are you using vanilla python to run scripts? Are you aware of the [Python GIL](https://wiki.python.org/moin/GlobalInterpreterLock) ? Also, see https://stackoverflow.com/questions/30791550/limit-number-of-threads-in-numpy – saud Nov 08 '18 at 10:30
  • you could monkey patch `threading` module, to limit thread number in python. – georgexsh Mar 06 '19 at 17:48

0 Answers0