How can I embed a Python function that returns a string in C using cffi?

Question

I'm trying to embed a Python function in C using PyPy and cffi. I'm following this guide from the PyPy documentation.

The problem is, all the examples I've found operate on ints, and my function takes a string and returns a string. I can't seem to figure out how to embed this function in C, as C doesn't seem to really have strings, rather making do with arrays of chars.

Here's what I've tried:

# interface.py

import cffi

ffi = cffi.FFI()
ffi.cdef('''
struct API {
    char (*generate_cool_page)(char url[]);
};
''')

...


@ffi.callback("char[] (char[])")
def generate_cool_page(url):
    # do some processing with BS4
    return str(soup)

def fill_api(ptr):
    global api 
    api = ffi.cast("struct API*", ptr)
    api.generate_cool_page = generate_cool_page

--

// c_tests.c

#include "PyPy.h"
#include <stdio.h>
#include <stdlib.h>

struct API {
    char (*generate_cool_page)(char url[]);
};

struct API api;   /* global var */

int initialize_api(void)
{
    static char source[] =
        "import sys; sys.path.insert(0, '.'); "
        "import interface; interface.fill_api(c_argument)";
    int res;

    rpython_startup_code();
    res = pypy_setup_home(NULL, 1);
    if (res) {
        fprintf(stderr, "Error setting pypy home!\n");
        return -1;
    }
    res = pypy_execute_source_ptr(source, &api);
    if (res) {
        fprintf(stderr, "Error calling pypy_execute_source_ptr!\n");
        return -1;
    }
    return 0;
}

int main(void)
{
    if (initialize_api() < 0)
        return 1;

    printf(api.generate_cool_page("https://example.com"));

    return 0;
}

When I run gcc -I/opt/pypy3/include -Wno-write-strings c_tests.c -L/opt/pypy3/bin -lpypy3-c -g -o c_tests and then run ./c_tests, I get this error:

debug: OperationError:
debug:  operror-type: CDefError
debug:  operror-value: cannot render the type <char()(char *)>: it is a function type, not a pointer-to-function type
Error calling pypy_execute_source_ptr!

I don't have a ton of experience with C and I feel like I'm misrepresenting the string argument/return value. How do I do this properly?

Thanks for your help!

Answer 1

Note that you should not be using pypy's deprecated interface to embedding; instead, see http://cffi.readthedocs.io/en/latest/embedding.html.

The C language doesn't have "strings", but only arrays of chars. In C, a function that wants to return a "string" is usually written differently: it accepts as first argument a pointer to a pre-existing buffer (of type char[]), and as a second argument the length of that buffer; and when called, it fills the buffer. This can be messy because you ideally need to handle buffer-too-small situations in the caller, e.g. allocate a bigger array and call the function again.

Alternatively, some functions give up and return a freshly malloc()-ed char *. Then the caller must remember to free() it, otherwise a leak occurs. I would recommend that approach in this case because guessing the maximum length of the string before the call might be difficult.

So, something like that. Assuming you start with http://cffi.readthedocs.io/en/latest/embedding.html, change plugin.h to contain::

// return type is "char *"
extern char *generate_cool_page(char url[]);

And change this bit of plugin_build.py::

ffibuilder.embedding_init_code("""
    from my_plugin import ffi, lib

    @ffi.def_extern()
    def generate_cool_page(url):
        url = ffi.string(url)
        # do some processing
        return lib.strdup(str(soup))   # calls malloc()
""")
ffibuilder.cdef("""
    #include <string.h>
    char *strdup(const char *);
""")

From the C code, you don't need initialize_api() at all in the new embedding mode; instead, you just say #include "plugin.h" and call the function directly::

char *data = generate_cool_page("https://example.com");
if (data == NULL) { handle_errors... }
printf("Got this: '%s'\n", data);
free(data);   // important!