Copy data from numpy to ctypes quickly

Question

I have a ctypes object shared between a c++ and python process. The python process takes the input values from that object, runs them through Tensorflow, and I'm left with a numpy array as output. As these arrays are pretty big I'm wondering if there is a better approach to copying the data from the output of tensorflow back to the shared ctypes object so the c++ process can act on them. (speed is the issue, yes.)

Right now I'm copying each value one by one:

output = np.array([12, 13, 11, 10]) # in reality this is fairly large (the Tensorflow result)
for i, value in enumerate(output):
    data.pressure[i] = ctypes.c_double(value)

where data is the ctypes object shared in memory. (structured after this example)

On the other hand, copying data from the ctypes object into numpy is easy, I'm wondering if there is something that does the opposite (from numpy to the ctypes array) Here is that easy code:

# Creating a numpy array from the ctypes array
input = np.reshape(data.velocity, (1, 29791))
# Tensorflow returns a numpy array
output = sess.run(final, feed_dict={Input: input}) 
# Now how do I get the data from output into data.pressure?

Edit: For reference, this is how the ctypes looks (python side)

class TransferData(ctypes.Structure):
    _fields_ = [
        ('statusReady', ctypes.c_bool),
        # ...
        ('velocity', ctypes.c_double * (31 * 31 * 31)),
        ('pressure', ctypes.c_double * (31 * 31 * 31))
    ]

Can you show the definition of `data`? – Michael Butscher Nov 05 '17 at 23:20 — Michael Butscher, Nov 05 '17 at 23:20
Sure thing, added to post. – Javid Pack Nov 05 '17 at 23:47 — Javid Pack, Nov 05 '17 at 23:47

score 5 · Accepted Answer · answered Nov 06 '17 at 00:13

This shows how to copy a whole data block from numpy array to ctypes array:

import numpy as np
import ctypes


# Preparing example

class TransferData(ctypes.Structure):
    _fields_ = [
        ('statusReady', ctypes.c_bool),
        ('velocity', ctypes.c_double * 4),
        ('pressure', ctypes.c_double * 4)
    ]


data = TransferData()
output = np.array([12., 13., 11., 10.])


# Actual code

# Both values should be equal but there could be problems with alignment settings
assert ctypes.sizeof(data.pressure) == output.nbytes

ctypes.memmove(ctypes.byref(data.pressure), output.ctypes.data, output.nbytes)


print(list(data.pressure))

Thanks, that works perfectly and is 15 to 30 times faster for my current use case. — Javid Pack, Nov 06 '17 at 19:47

Copy data from numpy to ctypes quickly

1 Answers1

Linked