Memory leak when using cythonized python in c ++ library

Question

I have a script written in python. I am cytonize it and plugging it in as a C ++ module. From this, I create the dll and connect it to the project in c #, where the library call must go through several times.

The problem arises precisely with the repeated launch of the library, because the first time the script is processed, the RAM is not cleared, which prevents it from being restarted. Python consists of modules that take up a lot of memory, so a single use of the library takes 160MB of RAM. I tried to use Py_Finalize (), but as I understand it, he deleted only dynamic sections (~ 86MB) for me, so re-initialization turns out to be an error. If you do not use Py_Finalize (), then each restart will take up + 80-90MB of memory, which after repeated starts becomes a very big problem.

C++ library: Method for running python

void MLWrapper::outputInfo(char * curDirPath) {
    auto err = PyImport_AppendInittab("runML", PyInit_runML);
    wchar_t* szName = GetWC(curDirPath);
    Py_SetPythonHome(szName);
    Py_Initialize();
    auto module = PyImport_ImportModule("runML");
    mlDataG.predictionResult = runTab(&mlDataG);
    Py_Finalize();}

C#: Class for working with dll

public class ExternalHelpers : IDisposable
{
    private IntPtr _libraryHandle;
    private OutputInfoDelegate _outputInfo;

    private delegate void OutputInfoDelegate([MarshalAs(UnmanagedType.LPStr)]string dirPath);

    public ExternalHelpers()
    {
        _libraryHandle = UnsafeMethods.LoadLibrary("MLWrapper.dll");

        if (_libraryHandle == IntPtr.Zero)
        {
            Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        }

        _outputInfo = LoadExternalFunction<OutputInfoDelegate>(@"?outputInfo@MLWrapper@@YAXPEAD@Z") as OutputInfoDelegate;
    }

    public void OutputInfo(string path)
    {
        _outputInfo(path);
    }

    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    ~ExternalHelpers()
    {
        Dispose(false);
    }


    private Delegate LoadExternalFunction<Delegate>(string functionName)
        where Delegate : class
    {
        IntPtr functionPointer =
            UnsafeMethods.GetProcAddress(_libraryHandle, functionName);

        if (functionPointer == IntPtr.Zero)
            Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        // Marshal to requested delegate
        return Marshal.GetDelegateForFunctionPointer(functionPointer, typeof(Delegate)) as Delegate;
    }

    private void Dispose(bool disposing)
    {
        if (disposing)
        {
            _outputInfo = null;
        }

        if (_libraryHandle != IntPtr.Zero)
        {
            while (UnsafeMethods.FreeLibrary(_libraryHandle) == true)
            {
                continue;
            }

            //if (!UnsafeMethods.FreeLibrary(_libraryHandle))
            //    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());

            _libraryHandle = IntPtr.Zero;
        }
    }
}

C#: Method call

using (ExternalHelpers e = new ExternalHelpers())
{
    ...                     
    e.OutputInfo(@"C:\Users\user\source\repos\Project\bin\x64\Debug");...}

How can I fix this problem?

I also had the idea of reconnecting the library dynamically. So I would be able to completely shut down the library and the memory should be freed, but when you clear the module’s memory, the library closes with the exit code: 1 and the main application ends.

Perhaps I forgot to describe some other details, so correct me in the comments if additional information is needed

You have written a python program that runs in c++ that your calling from c#? Is there an oppertunity to move it fully into one language? — Mark Davies, Jan 28 '20 at 08:37
Why not initialize only if Python interpreter isn't already initialized, i.e. after queyring the state with `Py_IsInitialized()` and finalize only when program is closed? Some extensions (I think numpy is (or at least was) one of them) have problems being Initialized/Finalized in the same process multiple times (not sure memory leak is the worst that can happen). — ead, Jan 28 '20 at 09:13

score 1 · Accepted Answer · answered Jan 28 '20 at 11:21

You could initialize/finalize Python interpreter multiple times, but it will result in memory leaks. In practice you should call initialize/finalize only once in app lifetime. From the Python documentation of the Py_FinalizeEx [1] function:

Bugs and caveats: The destruction of modules and objects in modules is done in random order; this may cause destructors (del() methods) to fail when they depend on other objects (even functions) or modules. Dynamically loaded extension modules loaded by Python are not unloaded. Small amounts of memory allocated by the Python interpreter may not be freed (if you find a leak, please report it). Memory tied up in circular references between objects is not freed. Some memory allocated by extension modules may not be freed. Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.

Ref: [1] https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx

Memory leak when using cythonized python in c ++ library

1 Answers1