Passing Numpy arrays to a C function for input and output

Question

Oh my word I'm a fool. I was simply omitting the second and third arguments when calling the function. Like a fool. Because that's what I am. Original silly question follows:

This seems like it must be a very common thing to do, but I can't find a relevant tutorial, and I'm too ignorant about Numpy and ctypes to figure it out myself.

I have a C function in file ctest.c.

#include <stdio.h>

void cfun(const void * indatav, int rowcount, int colcount, void * outdatav) {
    //void cfun(const double * indata, int rowcount, int colcount, double * outdata) {
    const double * indata = (double *) indatav;
    double * outdata = (double *) outdatav;
    int i;
    puts("Here we go!");
    for (i = 0; i < rowcount * colcount; ++i) {
        outdata[i] = indata[i] * 2;
    }
    puts("Done!");
}

(As you may guess, I originally had the arguments as double * rather than void *, but couldn't figure out what to do on the Python side. I'd certainly love to change them back, but I'm not picky as long as it works.)

I make a shared library out of it. gcc -fPIC -shared -o ctest.so ctest.c

Then in Python, I have a couple numpy arrays, and I'd like to pass them to the C function, one as input and one as output.

indata = numpy.ones((5,6), dtype=numpy.double)
outdata = numpy.zeros((5,6), dtype=numpy.double)
lib = ctypes.cdll.LoadLibrary('./ctest.so')
fun = lib.cfun
# Here comes the fool part.
fun(ctypes.c_void_p(indata.ctypes.data), ctypes.c_void_p(outdata.ctypes.data))

print 'indata: %s' % indata
print 'outdata: %s' % outdata

This doesn't report any errors, but prints out

>>> Here we go!
Done!
indata: [[ 1.  1.  1.  1.  1.  1.]
 [ 1.  1.  1.  1.  1.  1.]
 [ 1.  1.  1.  1.  1.  1.]
 [ 1.  1.  1.  1.  1.  1.]
 [ 1.  1.  1.  1.  1.  1.]]
outdata: [[ 0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  0.  0.]]

The outdata array is not modified. And in fact if I call the function again I get a segfault. Which doesn't surprise me -- I really don't know what I'm doing here. Can anyone point me in the right direction?

Sorry. It seems you are mixing `C` and `python` and I don't know how it works. The fact that the `puts` in `C` is called tells it is being called from `python` code. But I suspect what you are passing to the `C` function is causing the problems. — Mahesh, May 02 '11 at 22:15

score 87 · Answer 1 · edited Feb 12 '18 at 11:13

87

While not a direct answer to your original question, here's a much more convenient way to call your function. First, make the prototype of your C function exactly as you would do it in plain C. Since you don't need rowcount and colcount separately, I'll collapse them into a single size parameter:

void cfun(const double *indatav, size_t size, double *outdatav) 
{
    size_t i;
    for (i = 0; i < size; ++i)
        outdatav[i] = indatav[i] * 2.0;
}

Now define the ctypes prototype in the following way:

import ctypes
from numpy.ctypeslib import ndpointer
lib = ctypes.cdll.LoadLibrary("./ctest.so")
fun = lib.cfun
fun.restype = None
fun.argtypes = [ndpointer(ctypes.c_double, flags="C_CONTIGUOUS"),
                ctypes.c_size_t,
                ndpointer(ctypes.c_double, flags="C_CONTIGUOUS")]

Now, calls to your function will be really convenient:

indata = numpy.ones((5,6))
outdata = numpy.empty((5,6))
fun(indata, indata.size, outdata)

You could also define a wrapper to make this even more convenient:

def wrap_fun(indata, outdata):
    assert indata.size == outdata.size
    fun(indata, indata.size, outdata)

edited Feb 12 '18 at 11:13

Guillaume Jacquenot

11,217
6
43
49

answered May 03 '11 at 10:16

Sven Marnach

574,206
118
941
841

4

you might need `numpy.ascontiguousarray()` if the arrays are non-contiguous e.g., [`numpy.arange(1, 7)[::2]`](http://stackoverflow.com/a/22685515/4279). – jfs Mar 27 '14 at 11:11
4

@J.F.Sebastian: You can add `flags="C_CONTIGUOUS"` to the `ndpointer()` calls to dynamically check whether the array is C-contiguous. (Maybe I should add this to this answer.) – Sven Marnach Mar 27 '14 at 13:32
`flags` parameter leads to `TypeError` if input array is not contiguous. – jfs Mar 27 '14 at 16:42
@J.F.Sebastian: That's what I tried to imply by saying that it adds dynamic type checks. – Sven Marnach Mar 28 '14 at 14:04
Do you consider TypeError to be the correct answer? – jfs Mar 28 '14 at 15:42
@J.F.Sebastian: It can be argued whether this is a TypeError or a ValueError, but I think TypeError is fine. The main point is to raise an exception instead of trying to execute the C function on an input it wasn't designed for, which would probably lead to memory corruption. (I'm pretty sure you know all that, so I'm not quite sure what your point is.) – Sven Marnach Mar 28 '14 at 17:17
the point is: `outdata[:indata.size] = indata*2.0` doesn't break if the arrays are non-contiguous. The result of Python function should not depend on what memory layout Python object has. It is the responsibility of `wrap_fun()` to call `numpy.ascontiguousarray()` and make necessary adjustments if the change should be inplace. – jfs Mar 28 '14 at 21:08
@J.F.Sebastian: I don't disagree with calling `np.ascontiguousarray` if the function can potentially operate on a copy, but `wrap_fun` is written as an in-place operation. It doesn't return `outdata`. If it can't perform as designed with a non-contiguous array, a `TypeError` is correct. – Eryk Sun Mar 28 '14 at 22:33
@eryksun: the inplace operation: `outdata[:indata.size] = indata*2.0` doesn't raise TypeError on non-contiguous arrays – jfs Mar 28 '14 at 22:35
@J.F.Sebastian: OK, then you want to change `wrap_fun` to create a contiguous array and then copy it back to the original? Wouldn't it be better to force the caller to fix this a single time beforehand rather than making 2 copies? I'm just saying maybe `TypeError` is sometimes preferred. – Eryk Sun Mar 28 '14 at 22:42
@eryksun: np.ascontiguousarray() doesn't make unnecessary copies. – jfs Mar 28 '14 at 22:54
@J.F.Sebastian: But you have to check whether it has made a copy, and then copy it back to the original in that case. This double copy isn't an in-place operation, unlike `outdata[:indata.size] = indata*2.0` with non-contiguous arrays. BTW, I was awat that it checks `flags` to see if a copy is needed, which is why I said "potentially operate on a copy". – Eryk Sun Mar 28 '14 at 23:15

score 21 · Accepted Answer · answered May 02 '11 at 23:21

Just pass all four arguments to the C function. Change your Python code from:

fun(ctypes.c_void_p(indata.ctypes.data), ctypes.c_void_p(outdata.ctypes.data))

To:

fun(ctypes.c_void_p(indata.ctypes.data), ctypes.c_int(5), ctypes.c_int(6),
    ctypes.c_void_p(outdata.ctypes.data))

Passing Numpy arrays to a C function for input and output

2 Answers2

Linked