python bytearray to C++ object

Question

I'm wondering if I could get some help. For context, I'm using some C++ libraries to generate some large (think hundreds of Mb) objects that I want to send over a network from a server to a client.

On the server, I've got the following:

PyObject*  CKKSwrapper::SerializePrivateKey() {
    std::string s;
    std::ostringstream os(s);
    Serial::Serialize(m_keys.publicKey, os, SerType::BINARY);

    auto msg = os.str();
    return PyBytes_FromString(&msg[0]);
}

which gives me some Python object. I then send this directly to the client via python sockets. I'm reading it in like

def _safe_recv_abstract(socket: Socket, deserializer_func):
    expected_length = _get_obj_size(socket)
    running_length = 0
    running_msg = bytearray()
    while running_length < expected_length:
        msg = socket.recv(expected_length)
        if msg:
            running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
            running_length += len(msg)

    socket.send(_add_header_to_payload(b"ACK"))
    logger.debug("_safe_recv_unenc_obj: Received all data")
    if optional_pycrypto_deserialize_func:
        return deserializer_func(running_msg)
    return running_msg

two things:

1. Accumulator (from cppwrapper.Accumulator() above) looks like

PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b){
    return PyByteArray_Concat(&a, &b);
}

2. deserializer_func calls an underlying C++ function that looks like

void CKKSwrapper::DeserializeX(
    const boost::python::list &pyvals) {

    auto msg= pythonListToCppVectorBytes(pyvals);

    LPPrivateKey<DCRTPoly> sk;
    std::istringstream is(string(msg.begin(), msg.end()));
    Serial::Deserialize(sk, is, SerType::BINARY);

    this->m_keys.secretKey = sk;
}

---

I'm running into the following error:

Boost.Python.ArgumentError: Python argument types in
    CKKSwrapper.Accumulator(bytearray, bytearray)
did not match C++ signature:
    Accumulator(pycrypto::CKKSwrapper {lvalue}, _object*, _object*)

I completely understand what it is saying and that the types are wrong but I'm not sure WHY. From the docs

PyObject* PyByteArray_Concat(PyObject *a, PyObject *b)
    Return value: New reference.

    Concat **bytearrays** a and b and return a new bytearray with the result.

If I understand correctly, I AM passing in bytearrays but it says that it is expecting objects?

The reason I'm trying to do it this way is that when I use a bytearray or a list for the accumulation, i.e

while running_length < expected_length:
    msg = socket.recv(expected_length)
    if msg:
        running_msg = cppwrapper.Accumulator(running_msg, bytearray(msg))
        running_length += len(msg)

the memory usage and runtime blow up

Answer 1

Partial answer for the simple mistake:

PyObject* CKKSwrapper::Accumulator(PyObject a, PyObject b)

This should be

PyObject* CKKSwrapper::Accumulator(PyObject* a, PyObject* b)

PyObject is never passed by value, always by pointer. In practice all useful Python objects look like:

struct Something{
    PyObject ob_base;
    Useful data;
};

This is essentially a C version of C++ inheritance. By passing by value you're losing everything but ob_base, exactly like passing a C++ derived class as its base by value.