Valgrind error and memory leaks with Python/C API

Question

I'm actually developping a game in C++ and trying to do the AI with a script langage. To do so, i choosed Python2 with Python/C api. My AI is actually working but there is a big problem : when I run valgrind on my program, there is a lot of error and memory leaks. So, I would know if this happened because of my code or by the API ?

Here is a summary of my class AI :

 IA::IA()
{
  setenv("PYTHONPATH",".",1);
  Py_Initialize();
  PyRun_SimpleString("import sys");
  pName = PyBytes_FromString((char*)"Test");
  pModule = PyImport_Import(pName);
  pDict = PyModule_GetDict(pModule);
  pFunc = PyDict_GetItemString(pDict, "push_f");
}

IA::~IA()
{
  Py_DECREF(pValue);
  Py_DECREF(pModule);
  Py_DECREF(pName);
  Py_Finalize();
}

void IA::LaunchIA(float x, float y, float z)
{
  PyObject *toSend;

  toSend = Py_BuildValue("(OOO)", TlistMob, TlistPlayer, pDPosIA);
  pResult = PyObject_CallObject(pFunc, toSend);
  PyErr_Print();
  printf("return = %f\n", (float)PyInt_AsLong(pResult));

}

My (very) simple Python code :

def push_f(MobList, PlayerList, pos):
   return 0

And the valgrind error (x1000) :

==11602== Memcheck, a memory error detector
==11602== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==11602== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==11602== Command: ./a.out
==11602== 
==11602== Invalid read of size 4
==11602==    at 0x4FCE173: PyObject_Free (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F02FC2: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4FBDE9A: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F85BAD: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F872FF: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F88559: PyImport_ImportModuleLevel (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EFF697: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F4B1E2: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x5021446: PyEval_CallObjectWithKeywords (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EF45C5: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x502201B: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EF0B88: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==  Address 0x693c020 is 2,560 bytes inside a block of size 2,731 free'd
==11602==    at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11602==    by 0x4F81D28: PyMarshal_ReadLastObjectFromFile (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F85A22: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F872FF: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F88559: PyImport_ImportModuleLevel (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EFF697: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F4B1E2: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x5021446: PyEval_CallObjectWithKeywords (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EF45C5: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x502201B: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EF0B88: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F851B3: PyImport_ExecCodeModuleEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==  Block was alloc'd at
==11602==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11602==    by 0x4F81CDF: PyMarshal_ReadLastObjectFromFile (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F85A22: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F872FF: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F88559: PyImport_ImportModuleLevel (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EFF697: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F4B1E2: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x5021446: PyEval_CallObjectWithKeywords (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EF45C5: PyEval_EvalFrameEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x502201B: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4EF0B88: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602==    by 0x4F851B3: PyImport_ExecCodeModuleEx (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==11602== 
==11602== 
==11602== HEAP SUMMARY:
==11602==     in use at exit: 491,741 bytes in 204 blocks
==11602==   total heap usage: 3,301 allocs, 3,097 frees, 3,567,424 bytes allocated
==11602== 
==11602== LEAK SUMMARY:
==11602==    definitely lost: 0 bytes in 0 blocks
==11602==    indirectly lost: 0 bytes in 0 blocks
==11602==      possibly lost: 1,072 bytes in 2 blocks
==11602==    still reachable: 490,669 bytes in 202 blocks
==11602==         suppressed: 0 bytes in 0 blocks
==11602== Rerun with --leak-check=full to see details of leaked memory
==11602== 
==11602== For counts of detected and suppressed errors, rerun with: -v
==11602== Use --track-origins=yes to see where uninitialised values come from
==11602== ERROR SUMMARY: 497 errors from 25 contexts (suppressed: 0 from 0)

In my main, you need to know that I'm creating only one IA object.

Am I doing something wrong ? Or is it just the API ?

(This is not a duplicate because I run valgrind on my C++ executable and not Python, my c++ is running the script)

Thanks in advance !!!

Possible duplicate of [How can I use valgrind with Python C++ extensions?](http://stackoverflow.com/questions/3982036/how-can-i-use-valgrind-with-python-c-extensions) — DavidW, May 15 '17 at 07:16
I'm not 100% sure if the duplicate's right, so please disagree if you want, but it may help you ignore a lot of the errors. Also: you should be checking the return value from (most of) your Python calls against `NULL`. That's how you know if an error has occurred. — DavidW, May 15 '17 at 07:18
Thanks, that worked, but it stills remains an error : "Use of uninitialised value of size 8 ==10564== at 0x4FCE194: PyObject_Free (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)". I checked all the return of Python function in C++ and no error message appeared. — Kulbuto, May 15 '17 at 14:44
A skim over your code suggests that you're definitely leaking `pFunc`, `toSend`, `pResult` and possibly `TlistMob`, `TlistPlayer`, `pDPosIA` (I don't really know much about those last three). I don't know if that's the issue with the uninitialized value though. — DavidW, May 15 '17 at 15:32
Also, you should not call `PyErr_Print()` unless there is an error to print. — DavidW, May 15 '17 at 15:34

score 0 · Accepted Answer · edited May 23 '17 at 12:26

I initially suggested this as a duplicate. I don't think that's the case any more, but it still gives useful advice on removing false positives from the Valgrind output for Python programs.

Beyond that there are a number of specific issues with your program:

No error checking - the Python C API typically uses a NULL return value to indicate an error. There are clearly various ways of writing error checking code, but I'd be tempted to go for something like

IA::IA() :
pModule(NULL), pDict(NULL), pFunc(NULL), pName(NULL) // initially null initialize everything
{
  setenv("PYTHONPATH",".",1);
  Py_Initialize();

  // I don't think you actually use this line, so maybe remove it
  if (PyRun_SimpleString("import sys") == -1) goto error;

  pName = PyBytes_FromString((char*)"Test");
  if (!pName) goto error;
  pModule = PyImport_Import(pName);
  if (!pModule) goto error;
  pDict = PyModule_GetDict(pModule);
  if (!pDict) goto error;
  pFunc = PyDict_GetItemString(pDict, "push_f");
  if (!pFunc) goto error;

  return; // completed OK

  error:
  Py_XDECREF(pName); // XDECREF is OK with NULL...
  Py_XDECREF(pModule);
  Py_XDECREF(pDict);
  Py_XDECREF(pFunc);
  PyErr_Print();
  throw std::runtime_error(""); // ??? - possibly get and use the error string
            // see https://stackoverflow.com/a/1418703/4657412
}

I'm aware people are suspicious of goto but in this case it's a reasonably clean way of jumping to an error handling block. You can structure it differently if you like.

The destructor doesn't decref pFunc, which looks like a memory leak.
IA::LaunchIA similarly is lacking working error checking.
IA::LaunchIA never decrefs toSend, pResult, TlistMob, TlistPlayer, pDPosIA. Some of these are related to the incompleteness of the code you show, but if they aren't decrefed then you're leaking memory.
You call PyErr_Print() without checking for an error. The documentation says:

Call this function only when the error indicator is set. (Otherwise it will cause a fatal error!)

I suspect that this might be your biggest problem.

Those are all the issues I can see. Without a minimal complete example it's impossible to actually check what you're seeing. Given you're using C++ you might be well advised to use/make a decent, object oriented wrapper for PyObject* to avoid having to worry about reference counting yourself - Boost Python has one.

Valgrind error and memory leaks with Python/C API

1 Answers1