0

Looking for some advice on the following problem.

I have a number of jobs running using mpi4py on a SLURM system. I have noticed that when a given job is too big (i.e. too much data to process) I get the following error: 

mpirun noticed that process rank 0 with PID 62208 on node node1 exited on signal 9 (Killed).

I have tried breaking some jobs down into smaller chunks before submitting them, but I was wondering if there is a way to anticipate a Killed signal and add an except statement to break the job into chunks when the need arises.

berkelem
  • 2,005
  • 3
  • 18
  • 36
  • That completely depends on your jobs and your programs. Only you or the author of the program can know what your program can or cannot compute. – Vladimir F Героям слава May 27 '18 at 14:39
  • I'm the author of the code and I know what it's computing. I'm just looking for a way of knowing when the processors are reaching their maximum capacity so I can break up the data before the job is killed. – berkelem May 28 '18 at 12:39
  • But only you can know what that happens, how can we? It depends on your code! – Vladimir F Героям слава May 28 '18 at 12:54
  • I don't see how it depends on my code. The Killed signal, as I understand it, is sent when the processors are overloaded. So I'm looking for a way to get a readout from the processors when their memory is reaching its limit, before the Killed signal is sent. – berkelem May 28 '18 at 15:36
  • Than just monitor the amount of free memory like in any other Linux program. Guarding mallocs will not help as OS will hapilly give the process more memory than what is actually available. I am sure you will find many Q&As about that here. – Vladimir F Героям слава May 28 '18 at 15:46

2 Answers2

0

The KILL signal cannot be caught, blocked, or ignored, but it is often preceded by a INT or TERM signal, which you can catch and take the opportunity to act upon. See an example in Python for the INT signal here

damienfrancois
  • 52,978
  • 9
  • 96
  • 110
0

Your application is allocating too much memory. Unfortunately, SIGKILL can not be handled in your application; so you need to be proactive (making sure you never more memory than possible) rather than reactive (catch the signal and take action). It is unclear what exactly you mean by "too much data to process" or how you are breaking up the problem into smaller chunks, so I can only give some general advice.

If you can reasonably estimate the memory requirements of the problem you are trying to solve, you can warn the user and abort the job early if the problem size is larger than the nodes can handle. The script can then reduce the problem size or increase the number of nodes until it would fit in the memory. If the nodes are shared or you do not know the amount of available memory beforehand, you can use functions like get_phys_pages() and get_avphys_pages to determine that.

Sourav
  • 379
  • 7
  • 13