24

When writing python code, my typical workflow is to use the interactive prompt and do something like

write function
repeat until working:
  test function
  edit function

Once I'm pretty sure everything's ok, I'll run the code in non-interactive mode and collect the results.

Sometimes the functions run a little too slow and must be optimized.

I'm interested in using cython to optimize these slow functions, but I want to keep my interactive workflow, i.e., run the functions, make changes, run them again.

Is there an easy way to do this?

So far I've tried putting my cython functions in a separate module "my_functions.pyx":

def fun1(int x):
    return x + 130

def fun2(int x):
    return x / 30

Then running (at the interative prompt) 

import pyximport; pyximport.install()
import my_functions as mf
mf.fun1(25)

This works the first time, but I want to make changes to my cython functions and reload them in the same interactive session.

running

import my_functions as mf

doesn't update the functions at all. And running

reload(mf)

gives an error: No module named my_functions

The only thing that works is to quit the current session, restart ipython, and import the module all over again. But this sort of kills the benefits of running interactively.

Is there a better way to optimize functions with cython interactively?

If not, can you describe some other ways to approach optimizing code with cython?

Any help is appreciated.

kith
  • 5,486
  • 1
  • 21
  • 21
  • The problem is that cython modules are compiled to shared libraries, which cannot be reloaded by the `reload` function. AFAIK at the moment you simply cannot unload a shared library, hence you'll have to restart the interpreter. Note that you can avoid to restart *ipython* if you use the Qt console: use `Ctrl+.` to restart the kernel only, without restarting the whole console. – Bakuriu Aug 08 '13 at 11:28

1 Answers1

41

I found a poorly documented feature in the "pyximport.install" function that allows a cython module to be reloaded. With this feature set to True, you can load/reload your cython modules interactively without having to restart python.

If you initialize your cython module with:

import pyximport
pyximport.install(reload_support=True)
import my_functions as mf

You can make changes to your cython module, and then reload with:

reload(mf)

Hopefully this will be of use to someone.

kith
  • 5,486
  • 1
  • 21
  • 21
  • Awesome. Anyone know why it's not the default? – daknowles Jul 24 '16 at 02:12
  • 2
    In case it helps someone: for some reason I have to do both `reload(mf)` and then do `import my_functions as mf` a second time, in order for it to update. But then it works. – N. Virgo Feb 22 '19 at 08:24
  • 1
    I am trying to use this to reload new versions of the .SO files. It doesn't seem to be working. In my python prompt, I did `import pyximport; pyximport.install(reload_support=True); import my_module as m`. If I replace the .SO file with a new one and try `reload(m)` it's still using the old one. Any idea how to get this working? – Vinayak Jan 25 '20 at 03:50
  • I am having the same issue as @Vinayak. I am guessing the problem is leated to the .SO file being compiled in a way that does not support reload – Salvatore Cosentino Jan 29 '21 at 07:55
  • @SalvatoreCosentino I didn't get this to work, so I ended up separating/splitting my program into two different parts. An unchanging part which simply imports the .SO file and runs a function within it and the .SO file with the core logic in it. The unchanging script checks for updates and when an updated .SO file is found, it downloads it, replaces the existing .SO file and restarts itself. When no update is found, it imports the .SO file and runs a function within it. This approach worked for me on both Windows and OSX. – Vinayak Jan 29 '21 at 15:47
  • Thank you @Vinayak. In my case I just gave up on the REPL for now and just running a test script that would compile the Cython code when required. – Salvatore Cosentino Jan 30 '21 at 05:52
  • @Vinayak - Can you provide code for the solution you mentioned? – SuperCodeBrah Sep 28 '21 at 03:14
  • @SuperCodeBrah You can take a look at my solution on my question about the same thing here: https://stackoverflow.com/a/69379937/1768141 – Vinayak Sep 29 '21 at 16:11