When extending Python with C, How do one dynamically build a complex structure in C?

Question

I'd like to make a C function which returns a list of tuples, a list of dicts or even better, a list of objects of a class I defined in another python module.

The documentation on the C API («The Python/C API») mentions that to build a complex object, I need to call to Py_BuildValue which uses a Variable number of parameters or Py_VaBuildValue which uses a va_list argument.

Since C cannot dynamically make a call with a variable number of arguments, I cannot use Py_BuildValue and must use Py_VaBuildValue, but looking for how to build a va_list variable, I find replies which the There's no ability to fill a va_list explicitly and tells me to make it from a variable parameters function, which defeats the purpose... and from the tests a made, all I get is segmentation faults and core dumps.

So, how am I Supposed to use those functions ?

Answer 1

You're not supposed to use those functions to build dynamically-sized data structures. The FAQ says to use PyTuple_Pack instead of Py_BuildValue for an arbitrary-sized tuple, but that's wrong too; I don't know why it says that. PyTuple_Pack has the same varargs issues as Py_BuildValue.

To build a variable-length tuple from C, use PyTuple_New to construct an uninitialized tuple of the desired length, then loop over the indices and set the elements with PyTuple_SET_ITEM. Yes, this mutates the tuple. PyTuple_SET_ITEM is only safe to use to initialize fresh tuples that have not yet been exposed to other code. Also, be aware that PyTuple_SET_ITEM steals a reference to the new element.

For example, to build a tuple of integers from 0 to n-1:

PyObject *tup = PyTuple_New(n);
for (int i = 0; i < n; i++) {
    // Note that PyTuple_SET_ITEM steals the reference we get from PyLong_FromLong.
    PyTuple_SET_ITEM(tup, i, PyLong_FromLong(i));
}

To build a variable-length list from C, you can do the same thing with PyList_New and PyList_SET_ITEM, or you can construct an empty list with PyList_New(0) and append items with PyList_Append, much like you would use [] and append in Python if you didn't have list comprehensions or sequence multiplication.