2

I have a simple question.

I often run complex computations in pandas with large datasets. Sometimes python keep up using RAM more an more up to the point where everything is saturated and my computer basically crashes. I use the Anaconda distribution on a windows machine.

Is it possible to set a threshold (say 98%) such that if the RAM usage goes above that threshold, then the kernel should be restarted orthe process should be killed?

ℕʘʘḆḽḘ
  • 18,566
  • 34
  • 128
  • 235

1 Answers1

1

There seems to be relatively easy on Unix systems (https://docs.python.org/2/library/resource.html#resource.setrlimit). However, for windows systems there does not seem to be an internal python way of limiting the amount of memory. You will then have to rely on the Windows API. As far as I can tell from this answer, you would be best off creating a Job object, and associate it with the python process. Then you would use JOB_OBJECT_LIMIT_JOB_MEMORY to limit the memory of the process. Or possibly you can get away with using SetProcessWorkingSetSize.

Community
  • 1
  • 1
Vidar
  • 1,777
  • 1
  • 11
  • 15