27

I have a few functions written in C for a game project. These functions get called quite a lot (about 2000-4000 times per second). The functions are written in C for raw speed.

Now, the easiest way for me to include these functions into Python is to use ctypes. The alternative is to write a C extension to Python around these functions (which takes quite a bit of extra effort). So I wondered, not including the initial loading of the DLL, how big is the overhead of ctypes?

I'm using Python 2.7 (the standard CPython release), and I do not want to use an external library like Cython.

I know this question has been asked before, but I haven't seen much information about the performance comparison between the two options.

orlp
  • 112,504
  • 36
  • 218
  • 315
  • Well, PyPy's JIT can produce pretty great code for `ctypes` calls since a release or two. You may want to give it a shot. I don't post this as an answer because I don't have hard data at hand and it's not clear whether switching interpreters is an option for you. –  Nov 09 '11 at 15:25
  • I'd expect the overhead to be similar. – David Heffernan Nov 09 '11 at 15:26
  • @Delnan: This is a game I'm going to ship - also to linux. I cannot ask my users to install PyPy. – orlp Nov 09 '11 at 15:27
  • 1
    From python official documentation:"Do note that if your use case is calling C library functions or system calls, you should consider using the ctypes module rather than writing custom C code. Not only does ctypes let you write Python code to interface with C code, but it is more portable between implementations of Python than writing and compiling an extension module which typically ties you to CPython. – baco Mar 27 '13 at 17:32

2 Answers2

18

I've compared the performance of a C extension vs. a ctypes wrapper. In my particular test, the difference was about 250x. There were multiple calls into the C library so the ctypes wrapper was also executing Python code. The running time for the C library was very short which made the extra overhead for Python code even more significant. So the ratio will likely be different for you but was significant in my case.

casevh
  • 11,093
  • 1
  • 24
  • 35
9

The directly C coded interface has the potential to be much much faster. The bottleneck is the interface from Python to C and marshalling arguments and results may for example involve copying strings or converting Python lists to/from C arrays. If you have a loop that makes several hundred of these calls and some of the data doesn't have to be marshalled separately for each call then all you have to do is recode the loop in C and you may be able to massively reduce the bottleneck. ctypes doesn't give you that option: all you can do is call the existing functions directly.

Of course that all depends on exactly what sort of functions you are calling and what sort of data you are passing around. It may be that you can't reduce the overheads in which case I would still expect ctypes to be slower but perhaps not significantly.

Your best best would be to put together some sample of your code written each way and benchmark it. Otherwise there are just too many variables for a definitive answer.

Duncan
  • 92,073
  • 11
  • 122
  • 156
  • 1
    The data we are talking about is a bitmask (for collision). I have a C implementation ready, and the only data that is passed around in the Python code are booleans, coordinates and new python bitmask classes (which only contains a pointer to the actual data itself, which lives in the C code). – orlp Nov 09 '11 at 15:39
  • For bitmaps, maybe also explore numpy. The built-in Python bitmap support is rather slow for anything other than the underlying native type. – tripleee Jun 04 '21 at 10:11