Cpython Extension forcing Memory Leak

Question

I am trying to leak memory from my C-extension forcefully.

The code for the below is

Pyobect* myfunc(PyObject* self, PyObject* args)
{
    static int i = 213123;
    PyObject* temp = PyLong_FromLong(i);  // This is supposed to leak.
    i = i + 1 ;
    return Py_None;
}

I am using sys.gettotalrefcount() to check whether the total ref counts are increasing. After the first call to the above function, the ref count increases, but any other subsequent calls to the function does not increase the ref counts any further.

I am not sure whether any leaks are happening or not, or whether this is not the correct way to check leaks in a C-extension.

Do you want to force a memory leak? Or Want to check where the memory leak is happening? — Gaurav Pathak, Mar 27 '23 at 11:15
@GauravPathak . I want to force a memory leak. I am trying to understand memory management in C extensions and hence wanted to force a memory leak. The line where I have commented is supposed to create a memory leak. — daniel, Mar 27 '23 at 11:20
@climb4: Do you mean the variable "i"? This variable is incremented with each call to the function as it is a static variable. And hence the variable "i" is different for each call. — daniel, Mar 27 '23 at 11:25

CristiFati · Accepted Answer · 2023-05-20T10:30:58.407

What we've got here is Undefined Behavior.
According to [Python.Docs]: The None Object - Py_RETURN_NONE (emphasis is mine):

Properly handle returning Py_None from within a C function (that is, increment the reference count of None and return it.)

So, there is a memory leak, but that's masked by None's RefCount decrease (as after function exit, the value is garbage collected).

I prepared a small example:

Unfortunately, I don't have a debug Python build (having sys.gettotalrefcount)
I didn't build a full extension module, but a regular .dll, loaded via [Python.Docs]: ctypes - A foreign function library for Python (might also want to check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for a common pitfall when working with CTypes (calling functions))

dll00.c:

#include <Python.h>

#if defined(_WIN32)
#  define DLL00_EXPORT_API __declspec(dllexport)
#else
#  define DLL00_EXPORT_API
#endif


#if defined(__cplusplus)
extern "C" {
#endif

DLL00_EXPORT_API PyObject* func(PyObject *self, PyObject *args);

#if defined(__cplusplus)
}
#endif


PyObject* func(PyObject *self, PyObject *args)
{
    static int i = 213123;
    PyObject *temp = PyLong_FromLong(i);  // This is supposed to leak.
    i += 1;
#if defined(_UB)
    return Py_None;
#else
    Py_RETURN_NONE;
#endif
}

code00.py:

#!/usr/bin/env python

import ctypes as cts
import sys


def main(*argv):
    dll_name = "./{:s}.{:s}".format(argv[0] if argv else "dll00", "dll" if sys.platform[:3].lower() == "win" else "so")
    dll = cts.PyDLL(dll_name)
    func = dll.func
    func.argtypes = (cts.py_object, cts.py_object)
    func.restype = cts.py_object

    sentinel = 10  # @TODO - cfati: Chose an arbitrary value to make sure we don't run out of RefCounts in the 1st wave
    cnt = sys.getrefcount(None) - sentinel
    print("\nRunning function {:s} from {:s} {:d} times...".format(func.__name__, dll_name, cnt))
    for _ in range(cnt):
        func(0, 0)
    print("RefCounts - None: {:d}, total: {:d}\nRunning function {:d} (remaining) times...".format(
        sys.getrefcount(None), getattr(sys, "gettotalrefcount", lambda: -1)(), sentinel))
    for _ in range(sentinel):
        func(0, 0)


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.\n")
    sys.exit(rc)

Output:

[cfati@cfati-5510-0:/mnt/e/Work/Dev/StackExchange/StackOverflow/q075854845]> ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[064bit prompt]> ls
code00.py  dll00.c
[064bit prompt]>
[064bit prompt]> gcc dll00.c -fPIC -I/usr/include/python3.9 -L/usr/lib/x86_64-linux-gnu -lpython3.9 -shared -D_UB -o dll00_ub.so
[064bit prompt]> gcc dll00.c -fPIC -I/usr/include/python3.9 -L/usr/lib/x86_64-linux-gnu -lpython3.9 -shared -o dll00.so
[064bit prompt]>
[064bit prompt]> ls
code00.py  dll00.c  dll00.so  dll00_ub.so
[064bit prompt]>
[064bit prompt]> python3.9 ./code00.py dll00
Python 3.9.16 (main, Dec  7 2022, 01:11:51) [GCC 9.4.0] 064bit on linux


Running function func from ./dll00.so 4522 times...
RefCounts - None: 4532, total: -1
Running function 10 (remaining) times...

Done.

[064bit prompt]>
[064bit prompt]> python3.9 ./code00.py dll00_ub
Python 3.9.16 (main, Dec  7 2022, 01:11:51) [GCC 9.4.0] 064bit on linux


Running function func from ./dll00_ub.so 4555 times...
RefCounts - None: 10, total: -1
Running function 10 (remaining) times...
Fatal Python error: none_dealloc: deallocating None
Python runtime state: initialized

Current thread 0x00007fb115180740 (most recent call first):
  File "/mnt/e/Work/Dev/StackExchange/StackOverflow/q075854845/./code00.py", line 23 in main
  File "/mnt/e/Work/Dev/StackExchange/StackOverflow/q075854845/./code00.py", line 29 in <module>
Aborted

Might also be interesting:

[SO]: How to create python C++ extension with submodule that can be imported (@CristiFati's answer)

Cpython Extension forcing Memory Leak

1 Answers1

Linked