8

I've got a python application which uses a lot of memory, which should be fine as I handle it within loop with try / except MemoryError. Unfortunately, the MemoryError exception is never raised - before that happens, python is killed (on Debian Linux) by the OOM Killer.

Question is why... and how I can catch the error within Python. If I can catch it I have an easy mitigation, but without the exception I can't invoke my mitigation.

For info, the application is processing videos, with each frame being ~15MB numpy objects. If I run out of memory I'm happy to reduce the frame rate and try again.

I've also tried tracking memory usage as I load each frame, using psutil.available, but the process is killed with still ~350MB showing as available memory (of 2GB total). I assume that's a fragmentation problem.

Problem I therefore have is that I can arbitrarily set some limits, e.g. if I get to <500MB of free memory then start again with a lower frame rate, but it all feels a bit arbitrary and not very robust. If the application or perhaps the OS or the hardware changes, I might find that next time it crashes out at 501MB remaining, or something... that's why I'd rather handle it via the MemoryError exception.

Sadly this doesn't seem to be a common issue, with "python invoked oom-killer exception" giving me just two pages of Google search results! Mostly the answer on here previously has been "don't use so much memory", which isn't very helpful - in my case I want to use as much as available, but am happy to use less if I need to. Just that Python doesn't give me the opportunity to do so before it gets killed!

Any thoughts greatly appreciated.

DaveWalker
  • 521
  • 1
  • 4
  • 15
  • https://backdrift.org/oom-killer-how-to-create-oom-exclusions-in-linux ? – user2864740 Sep 09 '18 at 20:51
  • 1
    *"Unfortunately, the MemoryError exception is never raised ..."* - just guessing, but Linux oversubscribes memory. At the point of allocation you usually get success even if there's not enough virtual memory available. Later, when you try to use the memory and the system learns there is a shortage, it invokes the OOM killer. If you want to fail at the point of allocation then use Solaris. Solaris does not oversubscribe memory. – jww Sep 09 '18 at 20:51
  • Also see [User-space out-of-memory handling](https://lwn.net/Articles/590960/), [Toward a smarter OOM killer](https://lwn.net/Articles/359998/), [Taming the OOM killer](https://lwn.net/Articles/317814/), [Improving the OOM killer](https://lwn.net/Articles/684945/), [Avoiding the OOM killer with mem_notify](https://lwn.net/Articles/267013/), etc on LWN. The LWN articles keep flowing :) – jww Sep 09 '18 at 20:52
  • Thanks both, think I now understand far more about Linux memory allocation than I should ever need to! Having read around the subject, I think the real answer is actually that there isn't a "proper" solution to this, and so I'll have to work around it. One of the supposed advantages of Python is the cross-platform nature, easy portability, etc, and so whilst I can make some of the changes on my own systems to make it fail "in the right way", I can't expect that when I deploy elsewhere. So I'll have to make up some other metric as a proxy for a maximum amount of memory to use... – DaveWalker Sep 17 '18 at 18:44

1 Answers1

0

numpy tends to put everything in memory, so you can check how many video files you can have in memory at the same time.

"Simple" test of how many files you can load into memory

import numpy as np
import sys
import math
import random


def real_size(obj):
    size_bytes = sys.getsizeof(obj)
    if size_bytes == 0:
        return "0B"
    size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB")
    i = int(math.floor(math.log(size_bytes, 1024)))
    p = math.pow(1024, i)
    s = round(size_bytes / p, 2)
    return "%s %s" % (s, size_name[i])


def object_15():
    """
    This creates an numpy object with ~15MB
    """
    row = [random.random()] * 2080000
    data = np.array(row).astype(float)
    return data


def my_read_file(number_of_files):
    data = object_15()
    for i in range(number_of_files):
        data = np.append(data, object_15())
    return data

print(f'Just checking that each object has ~15MB: {real_size(object_15())}')
number_of_files = 100
print(f'Objects loaded into memory = {number_of_files}\nTotal memory_used = {real_size(my_read_file(number_of_files))}')

def object_15(): """ This creates an numpy object with ~15MB """ row = [random.random()] * 2080000 data = np.array(row).astype(float) return data

def my_read_file(number_of_files): data = object_15() for i in range(number_of_files): data = np.append(data, object_15()) return data

print(f'Just checking that each object has ~15MB: {real_size(object_15())}') number_of_files = 100 print(f'Objects loaded into memory = {number_of_files}\nTotal memory_used = {real_size(my_read_file(number_of_files))}')

According to this test, you should be able to have ~100 numpy objects, each of size ~15MB and not using more than 2 GB. Please adapt the code and try to your specific needs/object type.

More information here:

Other topics you might want to consider are:

Jose Santos
  • 1
  • 1
  • 2