Pickling a Python Extension type defined as a C struct having PyObject* members

Question

I am running C++ code via Python and would like to pickle an extension type.

So I have a C++ struct (py_db_manager) containing pointers to a database object and a object manager object (both written in C++) that I wrapped with a python type object (t_db_manager). My problem is that this python type needs to know how to pickle the two pointers in order to send it to some child multicore processes. So I registered the type with the copy_reg module (this is equivalent to writing a reduce() method on the type. However, I'm not too sure what to put in it. Should I build a tuple with the PyObject* or just the integer pointers?. Can anyone help?

typedef struct
{
  PyObject_HEAD
  PyObject* man_inst_ ;
  PyObject* db_inst_ ;

}py_db_manager;`

Here's the Py_TypeObject

PyTypeObject t_db_manager = {
    PyObject_HEAD_INIT(0)               /* tp_head */
    0,                                  /* tp_internal */
    ".py_db_manager",                  /* tp_name */
    sizeof(py_db_manager)};

And here's the code that would be in the reduce method:

PyObject *pickle_manager(PyObject *module, PyObject *args)
{
    py_db_manager *cpp_manager =0;
    PyObject *values = NULL,
        *tuple = NULL;
    char text[512];

    if (!PyArg_ParseTuple(args, "O!", &t_db_manager, &cpp_manager))
        goto error;
    sprintf(text,"man_inst_, db_inst_");
    if ((values = Py_BuildValue("(sii)", text,
                                cpp_manager->man_inst_, cpp_manager->db_inst_)) == NULL)
        goto error;
    tuple = Py_BuildValue("(OO)", manager_constructor, values);

error:
    Py_XDECREF(values);
    return tuple;
}

score 2 · Answer 1 · answered May 14 '11 at 06:07

2

Because this will be passed to another process, pickling just integer pointers will not work as you would want it to. Different processes use different memory space therefore they don't see the same things.

So, to answer your question, you should pickle full objects and reconstruct them from the receiving end.

answered May 14 '11 at 06:07

Nam Nguyen

1,765
9
13

1

Actually, that's the whole idea. I want the child processes to look at the SAME database and have the same manager by looking at the SAME pointer. I plan to do this by using the sharedctypes module and calling shareddata=sharedctypes.Value(anInstanceof_t_db_manager). Still, shareddata will be pickled when being send to the children – octi May 14 '11 at 22:20
1

Still, you cannot just transfer the pointer! As an example, supposed that in the parent process, your pointer is pointing to 0x87654321. If you pass this pointer to the child process, it will still see 0x87654321, but at that address there could be something entirely differently than it is in the parent process. – Nam Nguyen May 15 '11 at 14:18
1

So when python spawns child processes does it copy the memory table of the parent process into each child process? This is what I see is happening while debugging: each child process has an instance of the database found at different memory addresses. What you are saying is that these memory addresses are not 'global' but process specific? Isn't there a way to tell the child process to use the databse found at the address of the parent process? – octi Sep 08 '11 at 17:54
1

There is a way. It is called shared memory. As far as I know, you cannot do that with pure Python. – Nam Nguyen Sep 09 '11 at 00:07

Pickling a Python Extension type defined as a C struct having PyObject* members

1 Answers1

Linked