1

So, I want to call a Python callback function from C.

At some point, the function is sent to C and packed into a tuple like this

PyObject *userData = Py_BuildValue("Oi",py_callback,some_number);

Somewhere in that area, I do Py_INCREF(py_callback), too.
At some later time in the program, I want to call that function

PyObject *py_callback;
int some_number;
PyArg_ParseTuple((PyObject*)userData,"Oi",&py_callback,&some_number); // returns true
PyObject *py_result = PyObject_CallFunctionObjArgs(py_callback,
                                                   /* ... */
                                                   NULL);

and that last call throws a segmentation fault. Do you have any idea, why it would do such a thing?

bastibe
  • 16,551
  • 28
  • 95
  • 126
  • 1
    Compile in debug mode, load it up in GDB, and get a backtrace. – Conrad Meyer Feb 23 '11 at 12:17
  • 1
    Are you making sure you hold the GIL before calling back into Python? – ncoghlan Feb 23 '11 at 13:07
  • I spotted a different problem that is the more likely culprit (see my answer). If that still doesn't work, http://docs.python.org/dev/c-api/init.html#thread-state-and-the-global-interpreter-lock should provide you with enough Google fodder to look for some easier to follow explanations of how to handle the CPython interpreter lock. – ncoghlan Feb 23 '11 at 14:12
  • @ncoghlan: Your suggestion was correct: The issue can be solved with GIL states. If you formulate your comment as a question, I shall accept it. – bastibe Feb 28 '11 at 10:24

2 Answers2

5

When getting weird behaviour from the Python C API, it is always worth double checking that you are managing the state of the Global Interpreter Lock (aka "the GIL") correctly: http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock

ncoghlan
  • 40,168
  • 10
  • 71
  • 80
  • Oh, I'm sorry. That was a typo. I corrected the question accordingly. – bastibe Feb 23 '11 at 16:13
  • That still doesn't make sense - what are you passing as the second argument to `PyObject_CallFunctionArgs`? `&some_number` won't work correctly (it's the wrong type) and you aren't initialising `py_some_number` at all in the current version. Since you want to call back into Python, using `"OO"` and keeping the number as a `PyObject` is what you want to do. – ncoghlan Feb 23 '11 at 22:08
  • You are right, of course. I actually figured out that I would not need that number anyway. – bastibe Feb 28 '11 at 10:29
  • I replaced my original answer with a reference to the GIL management documentation, since that comment turned out to fix the real problem, while the tuple parsing problems only affected the SO question, not the real code. – ncoghlan Feb 28 '11 at 13:34
1

I'm not sure why everyone is talking about the GIL here. Are you releasing it anywhere? I notice a few possible trouble points which I'll list below. It's intermingled with suggestions.

I notice that you're not incrementing the reference count of userData. Why not? You are storing it aren't you? Why do you want to store it in the tuple? Why not just keep two variables (for callback and data) and then increment their refcounts so that you own a reference?

First of all, check the return values of all your functions to see if they're NULL. This will let you make sure that you're actually running properly. It's also probably a good idea to use PyObject_Print to print the objects you're interested in at all the points in the code to make sure things are going as you expect.

From your comment on where the segfault happens, my guess is that you're passing the wrong number of parameters to your callback function when you call it with PyObject_CallFunctionObjArgs. Can you double check it? Maybe show us your callback function definition and then check the invocation again.

Another possible issue I see is that from the naming, py_some_number sounds like you expect a Python integer object. This is not the case. After the PyArg_ParseTuple, it will contain an integer.

Noufal Ibrahim
  • 71,383
  • 13
  • 135
  • 169
  • The C-callback function has to conform to a function definition by Portaudio, so `userData` can only be one pointer. I checked and double checked all invocations and arguments and they are definitely correct. Thanks for the tip about incrementing the refcounts, though! – bastibe Feb 28 '11 at 14:21
  • 1
    I'm curious then. What change fixed the problem? – Noufal Ibrahim Feb 28 '11 at 14:26
  • Inserting `PyGILState_STATE gstate = PyGILState_Ensure();` before any calls to Python. The backend was actually creating a new thread, so I could not call Python functions without ensuring the thread state beforehand. I'm not sure if this makes sense. I am still struggling with the threading stuff in another question on SO. – bastibe Mar 01 '11 at 07:54
  • This is extremely weird. I've been doing this (calling python functions passed into C after a while) for a while now without hitting any such issues. I'm not using another thread but still. – Noufal Ibrahim Mar 01 '11 at 08:14
  • I ran into page 297 of the book "Expert Python Programming,: Become a master in Python by learning coding" on Google Books while searching for my issue. It does mention that the GIL limitations are famous limitation of coding C extensions in Python. – mds Jan 24 '20 at 00:50