3

I have a python written a python module to query a database and read this into a dataframe. Some of these queries are quite big and are causing the module to exit. i.e. I get:

Exited

printed to the screen. Digging a bit deeper, I find

Memory cgroup out of memory: Kill process

So it's running out of memory - my question is how do I capture that kill signal so I can print a useful error message e.g. you need to request more resources to run this command... Currently I have:

import signal
import pandas as pd

kill_now = False
def exit_gracefully(signum, frame, ):
    kill_now = True

signal.signal(signal.SIGINT, exit_gracefully)
signal.signal(signal.SIGTERM, exit_gracefully)
sql_reader = pd.read(query, conn, chunksize=1000)
table_data = []
while not kill_now:
    for data in sql_reader:
        table_data.append(data)
    break
if kill_now:
   print("ran out of memory...")

But this doesn't catch the "Killed" signal

sophros
  • 14,672
  • 11
  • 46
  • 75
Conor
  • 535
  • 2
  • 8
  • 16
  • Are you sure the process being killed is python, not some spawned database driver process? – sophros Jan 13 '21 at 11:50
  • how could a check that? – Conor Jan 13 '21 at 11:56
  • Have you checked answers and links to [this question](https://stackoverflow.com/questions/726690/what-killed-my-process-and-why)? – sophros Jan 13 '21 at 12:12
  • I'm not sure that helps - I don't have control over the RAM, all I can do is capture the error and give a more descriptive message to the user - i.e. that they need to increase their RAM. But I agree that that question is likely the issue i.e. process is most likely being killed OOMKiller. I'd guess I need to specify a memory limit, and exit if that is reached - that memory limit being below the actual amount of memory available - but I couldn't find a way to get the max memory available – Conor Jan 13 '21 at 13:43

0 Answers0