Which C functions are not gil-requiring in Cython?

Question

I try to compile the following Cython code, which uses C functions for file operations:

import tempfile
from libc.stdio cimport *

cdef extern from "stdio.h":
    FILE *fopen(const char *, const char *)
    int fclose(FILE *)
    size_t fwrite(const void *, size_t, size_t, FILE *)
    ssize_t getline(char **, size_t *, FILE *)

def run_io():
    cdef int ntasks
    cdef int i
    cdef string dump = "Some string"
    cdef string content = ""
    cdef char* fname
    cdef FILE* cfile
    cdef char* line = NULL
    cdef size_t l = 0
    tmpfile = tempfile.NamedTemporaryFile('w+')
    fname = tmpfile.name.encode("UTF-8")
    with nogil:
        cfile = fopen(fname, "wb")
        #fwrite(dump.data(), 1, dump.size(), cfile)
        #fclose(cfile)
        #cfile = fopen(fname, "rb")
        #if getline(&line, &l, cfile) == -1:
            #break
        #else:
            #printf("%s", line)
        fclose(cfile)
    tmpfile.close()

However, I get the following error:

Error compiling Cython file:
------------------------------------------------------------
...
        #cfile = fopen(fname, "rb")
        #if getline(&line, &l, cfile) == -1:
            #break
        #else:
            #printf("%s", line)
        fclose(cfile)
             ^
------------------------------------------------------------

test.pyx:31:14: Calling gil-requiring function not allowed without gil

I thought that only python functions are gil-requiring but not imported C ones. Nevertheless, it seems like it is not so.

Therefore, my questions are:

Which C functions can be used in Cython without GIL?
How to make file read/write without GIL?

Going to hazard a guess that the problem is `printf` since you haven't defined it properly (no mention in the extern block). — Dunes, Oct 27 '15 at 13:08
A separate issue to your GIL problem, but: you should use `printf("%s",line)` instead of `printf(line)`. See https://en.wikipedia.org/wiki/Uncontrolled_format_string — DavidW, Oct 27 '15 at 13:15
Second guess. Are you sure you can use `break` with a `with` statement in cython? In python that is not allowed. The compiler may have emitted a python break as there is no loop construct, and python break would not be allowed in a `with nogil` block. — Dunes, Oct 27 '15 at 13:30
Your code has multiple issues, and converting it with cython yields multiple errors. Provide something that reproduces __only__ the problem you have. — jepio, Oct 27 '15 at 13:37

jepio · Accepted Answer · 2015-10-27T14:05:29.123

You are shadowing the declarations from libc.stdio which are declared with

cdef extern from "stdio.h" nogil:

with your own definitions which do not have nogil. To answer the question in your title: only Python/C API functions require the gil.

This is your code with correct imports and trimmed of anything not relevant:

import tempfile
from libc.stdio cimport fopen, fclose, fwrite, getline, FILE
from libcpp.string cimport string

def run_io():
    cdef string dump = b"Some string"
    tmpfile = tempfile.NamedTemporaryFile('w+')
    cdef bytes py_fname = tmpfile.name.encode("UTF-8")
    cdef char* fname = py_fname
    cdef FILE* cfile
    with nogil:
        cfile = fopen(fname, "wb")
        fclose(cfile)
    tmpfile.close()

The following is necessary to ensure that the lifetime of the temporary returned by tmpfile.name.encode is extended.

cdef bytes py_fname = tmpfile.name.encode("UTF-8")
cdef char* fname = py_fname

It gives no errors when compiled with

cython -3 --cplus my_mod.pyx
g++ my_mod.cpp -shared -o my_mod.so $(python3.4 --cflags --ldflags)

Which C functions are not gil-requiring in Cython?

1 Answers1

Linked