INCREF needed when returning argument from Python C Extension function?

Question

This question is pretty simple but will help cement my understanding. I know that arguments to C extension functions are guaranteed to be live references for the duration of the C code (unless manually DECREFed). However, if I have a C extension code that returns a PyObject* that was present in its argument list, do I need to INCREF the argument before returning it? I.e., which of the following two is correct:

static PyObject return_item(PyObject *self, PyObject *item)
{
    // manipulate item
    return item;
}

or

static PyObject return_item(PyObject *self, PyObject *item)
{
    // manipulate item
    Py_INCREF(item);
    return item;
}

Based on https://docs.python.org/3/extending/extending.html#ownership-rules, which says

The object reference returned from a C function that is called from Python must be an owned reference — ownership is transferred from the function to its caller.

and Returning objects to Python from C I assume it's the latter (INCREFing is the way to go) but I want to be sure.

score 3 · Accepted Answer · answered Aug 26 '19 at 16:35

If someone is calling the return_item function from Python, they might do this:

something = Something()
something_else = return_item(something)
del something

If return_item did not return the argument which was passed in, but something else, you would expect that at this point the something which was passed in should be freed from memory, because its reference count falls to zero.

If you don't Py_INCREF and return the same object, that will still happen - the object's reference count will fall to 0 and you will have an invalid object in something_else.

TL;DR: Yes, you should Py_INCREF, because you created another reference to that object by returning it from the function.

Robin Betz · Answer 2 · 2019-09-26T22:53:01.203

You don't want to increment the reference count of the object before you return it. Doing so creates a memory leak that will prevent the object from being garbage collected.

Think of incrementing the reference count as saying "I am using this memory. Please don't free it." When you enter the C code, you "borrow" the reference from Python, but when you exit the C code, you're done with the object and don't need the reference anymore.

Variables and the underlying memory in Python are separate, which lets it be somewhat efficient with memory (read for more). The other answer misses the fact that assigning to something_else increments the reference count for the underlying memory. You can verify this yourself with sys.getrefcount.

import sys
something = "hello"
print(sys.getrefcount(something))       # 2 (getrefcount uses a reference)

something_else = something
print(sys.getrefcount(something_else))  # 3 (same memory as something)

del something
print(sys.getrefcount(something_else))  # 2
print(something_else)                   # "hello"

Both something and something_else referred to the same memory (a string containing the text "hello"). Deleting one doesn't affect the other. Even though your code uses a C function, the underlying principle is the same. Try printing out the reference count with both versions of your C function and it will be more clear.

What you don't want to do is call Py_DECREF before returning an object. In that case the reference count can drop to zero and the returned thing is totally invalid.

INCREF needed when returning argument from Python C Extension function?

2 Answers2