Passing arguments to tp_new and tp_init from subtypes in Python C API

Question

I originally asked this question on the Python capi-sig list: How to pass arguments to tp_new and tp_init from subtypes?

I'm reading the Python PEP-253 on subtyping and there are plenty of good recommendations on how to structure the types, call tp_new and tp_init slots, etc.

But, it lacks an important note on passing arguments from sub to super type. It seems the PEP-253 is unfinished as per the note:

(XXX There should be a paragraph or two about argument passing here.)

So, I'm trying to extrapolate some strategies well known from the Python classes subtyping, especially techniques that each level strips-off arguments, etc.

I'm looking for techniques to achieve similar effect to this, but using plain Python C API (3.x):

class Shape:
    def __init__(self, shapename, **kwds):
        self.shapename = shapename
        super().__init__(**kwds)

class ColoredShape(Shape):
    def __init__(self, color, **kwds):
        self.color = color
        super().__init__(**kwds)

What would be the equivalent in Python C API?

How to deal with similar situation but with arguments specific to derived class expected in different order? It is arguments given at the end of the args tuple (or kwds dict, I assume principle would be same).

Here is some (pseudo-)code that illustrates the situation:

class Base:
   def __init__(self, x, y, z):
      self.x = x
      self.y = y
      self.z = z

class Derived(Base):
   def __init__(self, x, y, a):
      self.a = a
      super().__init__(x, y, None):

Note, if the a was expected first:

Derived.__init__(self, a, x, y)

it would be similar situation to the Shape and ColoredShape above. It would also be easier to deal with, I assume.

Could anyone help to figure out the missing XXX comment mentioned above and correct technique for passing arguments from subtype up to super type(s) on construction?

UPDATE 2012-07-17:

Inspired by ecatmur's answer below I looked through Python 3 sources and I found defdict_init constructor of collections.defaultdict type object interesting. The type is derived from PyDictObject and its constructor takes additional argument of default_factory. The constructor signature in Python class is this:

class collections.defaultdict([default_factory[, ...]])

Now, here is how the default_factory is stripped from original args tuple, so the rest of arguments is forwarded to the tp_init of base type, it is PyDictObject:

int result;
PyObject *newargs;
Py_ssize_t n = PyTuple_GET_SIZE(args);
...
newargs = PySequence_GetSlice(args, 1, n);
...
result = PyDict_Type.tp_init(self, newargs, kwds);

Note, this snipped presence only the relevant part of the defdict_init function.

score 7 · Accepted Answer · answered Jul 12 '12 at 13:32

The problem is that PyArgs_ParseTupleAndKeywords doesn't provide a way to extract extra *args and **kwargs from the input args and keywords; indeed, any extra arguments result in a TypeError; "Function takes %s %d positional arguments (%d given)", or "'%U' is an invalid keyword argument for this function".

This means that you're going to have to parse args and keywords yourself; you're guaranteed that args is a tuple and keywords is a dict, so you can use the standard methods (PyTuple_GET_ITEM and PyDict_GetItemString) to extract the arguments you're interested in, and identify and construct a tuple and dict to pass on from the remainder. You obviously can't modify args, because tuples are immutable; and while popping items from keywords should be OK it does seem a little risky (example crash).

A more ambitious but definitely feasible route would be to copy vgetargskeywords from getargs.c (http://hg.python.org/cpython/file/tip/Python/getargs.c) and extend it to take optional out-parameters for remainder *args and **kwargs. This should be fairly straightforward as you just need to modify the parts where it detects and throws TypeError on extra arguments (extra args; extra keywords). Good luck if you choose this route.

I accepted your answer and awarded it with the bounty. Your solution directly addresses the problem and has been confirmed by my recent findings in the Python sources (see my update on `defdict_init`). Thanks very much! — mloskot, Jul 17 '12 at 09:04

score 2 · Answer 2 · edited Jul 13 '12 at 22:15

OK, to answer your first question: first of all you have a C Structure representing your class object, but from a C side. So in a header file called Shapes.h

typedef struct {
PyObject_HEAD
char *shapename;
} Shape;

typedef struct {
PyObject_HEAD
    char *shapename;
char *color;
} ColouredShape;

The things to note here are the following:

shapename and colour are effectively private variables. Python can't see them or interact with them
ColouredShape has to define all the parameters of Shape for inheritace to work as well as appearing in the same order

Next we need a type for our classes. The type basically defines the object from a Python point of view, i.e., its methods, members and any special methods its has. It's also where we tell Python that ColouredShape is a subclass of Shape. Its looks like the following:

PyTypeObject ShapeType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"mymod.Shape", /*tp_name*/
sizeof(ShapeType), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor) SimpleParameter_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
0, /*tp_compare*/
0, /*tp_repr*/
0, /*tp_as_number*/
0, /*tp_as_sequence*/
0, /*tp_as_mapping*/
0, /*tp_hash */
0, /*tp_call*/
Shape__str__, /*tp_str*/
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
"A class representing a Shape", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
Shape_methods, /* tp_methods */
0, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc) Shape_init, /* tp_init */
0, /* tp_alloc */
Shape_new, /* tp_new */
};

PyTypeObject ColouredShapeType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"mymod.ColouredShape", /*tp_name*/
sizeof(ColouredShape), /*tp_basicsize*/
0, /*tp_itemsize*/
(destructor) ColouredShape_dealloc, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
0, /*tp_compare*/
0, /*tp_repr*/
0, /*tp_as_number*/
0, /*tp_as_sequence*/
0, /*tp_as_mapping*/
0, /*tp_hash */
0, /*tp_call*/
0, /*tp_str*/
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/
"A class representing a coloured shape", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
ColouredShape_methods, /* tp_methods */
0, /* tp_members */
0, /* tp_getset */
&ShapeType, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc) ColouredShape_init, /* tp_init */
0, /* tp_alloc */
ColouredShape_new, /* tp_new */
};

One important point to note is that mymod must be the name of your Python C extension as imported from Python. In answer to your second question, your init function would look as follows:

int Shape_init(Shape *self, PyObject *args, PyObject *kwds){
   char *colour = null;
   static char *kwdlist[] = {"colour", NULL};
   if (!PyArg_ParseTupleAndKeywords(args, kwds, "s", kwdlist,
&colour)){
return -1;
}
   //Initialise your object here
}

There is no reason why ColouredShape_init cannot call Shape_init. However my understanding of PyArgs_ParseTupleAndKeywords is:

There is no format parameter for the rest of your positional and keyword arguments
It does not alter your args and keywords

Which is where your difficulty will be if you tried to go that way.

If you have any further questions on it let me know. But I suggest you look at the PyArgs_ParseTupleAndKeywords to get a better understanding of it

I'm afraid, this is not the answer to my question. I'm not asking about how to create a type object and process arguments passed to `tp_init` slot. I'm perfectly familiar with these elements. You haven't provided any more information than can be found in the PEP-253. It's rather modest answer regarding the matter I've asked about. I can't accept it. — mloskot, Jul 12 '12 at 14:43

score 0 · Answer 3 · answered Jul 12 '12 at 10:14

I have delved into the python c api at some length and its a strange beast to say the least. It may be worth looking into pyrex or cython which take care of a lot of the ugliness for you. From a python perspective there is no reason you can change arguments. however you can't change the order of members in C so for consistency I strongly advise you not to do it. If you want help on how to achieve this without pyrex or cython let me know. but i should warn it takes some understanding and a lot of boiler plate code to understand

Thanks for your answer, but I emphasised it is about using plain Python C API. No other methods are of interest here. Obviously, I can't accept your answer. — mloskot, Jul 12 '12 at 10:24

Passing arguments to tp_new and tp_init from subtypes in Python C API

3 Answers3

Linked