0

Sometimes a Python program stops with an exception like the following, when there is not enough memory:

OSError: [Errno 12] Cannot allocate memory

Can I make it wait until memory is available again instead of dying unrecoverably? Or at least freeze until the user sends a SIGCONT or something to it?

It's not my program, so I prefer not to modify its source code, but I think it would still be cool if I can do that by modifying only the outmost calling part.

Thank you!

noname
  • 343
  • 4
  • 14
  • Are you willing to do low-level programming like writing a C library to use with `LD_PRELOAD`? Or do you want a pure Python solution? Is this on Linux? – John Zwinck Dec 14 '16 at 13:30
  • @JohnZwinck Thanks for commenting. Actually I don't want to do low-level things like hooking.. So pure Python solution would be better. Yes, it's on Ubuntu. – noname Dec 14 '16 at 14:38
  • @JohnZwinck Just to clarify, by "I don't want to do low-level things", I meant I don't want to insert low-level things into this project. I guess it will be okay if its reliable enough though. – noname Dec 14 '16 at 17:32

1 Answers1

2

You can catch the OSError exception but this may not help your program to continue where it left off.

To do this well you need to interpose some code between Python and malloc. You can do that using LD_PRELOAD as per the details here: How can I limit memory acquired with `malloc()` without also limiting stack?

The idea is you implement a wrapper for malloc which calls the real malloc and waits to retry if it fails. If you prefer not to use LD_PRELOAD then building Python with your interposing code baked in is a possibility (but a bit more work).

The library you'll write for LD_PRELOAD will end up being usable with just about any program written in C or C++. You could even open-source it. :)

Community
  • 1
  • 1
John Zwinck
  • 239,568
  • 38
  • 324
  • 436
  • 3
    Note: You don't need to wrap `malloc` itself on modern Python; in 3.4 and higher, [PEP 445](https://www.python.org/dev/peps/pep-0445/) provides APIs to replace the default allocators used by Python for arbitrary purposes. There is no actual guarantee that Python ultimately devolves to `malloc` at all (the small object allocator uses arenas which can be directly acquired via `mmap` for instance), so those hooks and the PEP guarantees are important. – ShadowRanger Dec 15 '16 at 02:19