Scipy special functions with numba

Question

This code fails:

import numpy as np
from numba import jit
import scipy.special as sp

@jit(nopython=True)
def f(a):
    return sp.xlogy(a, a)
  

a = np.array([1,2,0,1], dtype=float)
f(a)

It gives the following error

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Unknown attribute 'xlogy' of type Module(<module 'scipy.special' from '/home/yair/.local/lib/python3.8/site-packages/scipy/special/__init__.py'>)

However, I believe it should work, becuase it says here that xlogy could be overloaded. An answer from 2015 says this could not be done (for some other function) but maybe I thought it changed.

Of course, I can implement it myself but it would be nice to use a known solution.

EDIT: Per Andras Deak's suggestion below, I tried using numba-scipy. I cannot get it to work and was hoping for some usage example. Here are my attempts - they all fail, see errors below:

import numpy as np
from numba import jit
import numba_scipy
import scipy as sp
from scipy import special as spc

@jit(nopython=True)
def f1(a):
    return numba_scipy.special.xlogy(a,a)

@jit(nopython=True)
def f2(a):
    return numba_scipy.xlogy(a,a)


@jit(nopython=True)
def f3(a):
    return sp.special.xlogy(a,a)

@jit(nopython=True)
def f4(a):
    return spc.xlogy(a,a)


a = np.array([0,1,2])

try:
    f1(a)
except Exception as e:
    print('f1 fails')
    print(e)

try:
    f2(a)
except Exception as e:
    print('f2 fails')
    print(e)

try:
    f3(a)
except Exception as e:
    print('f3 fails')
    print(e)

try:
    f4(a)
except Exception as e:
    print('f4 fails')
    print(e)

Complete error message:

/home/yair/.local/lib/python3.8/site-packages/numba/core/dispatcher.py:238: UserWarning: Numba extension module 'numba_scipy' failed to load due to 'ValueError(No function '__pyx_fuse_0pdtr' found in __pyx_capi__ of 'scipy.special.cython_special')'.
  entrypoints.init_all()
f1 fails
Failed in nopython mode pipeline (step: nopython rewrites)
module 'numba_scipy' has no attribute 'special'
f2 fails
Failed in nopython mode pipeline (step: nopython rewrites)
module 'numba_scipy' has no attribute 'xlogy'
f3 fails
Failed in nopython mode pipeline (step: nopython frontend)
Unknown attribute 'xlogy' of type Module(<module 'scipy.special' from '/home/yair/.local/lib/python3.8/site-packages/scipy/special/__init__.py'>)

File "example.py", line 19:
    def f3(a):
        return sp.special.xlogy(a,a)
        ^

During: typing of get attribute at example.py (19)

File "example.py", line 19:
    def f3(a):
        return sp.special.xlogy(a,a)
        ^

f4 fails
Failed in nopython mode pipeline (step: nopython frontend)
Unknown attribute 'xlogy' of type Module(<module 'scipy.special' from '/home/yair/.local/lib/python3.8/site-packages/scipy/special/__init__.py'>)

File "example.py", line 23:
    def f4(a):
        return spc.xlogy(a,a)
        ^

During: typing of get attribute at example.py (23)

File "example.py", line 23:
    def f4(a):
        return spc.xlogy(a,a)
        ^

Andras Deak -- Слава Україні · Accepted Answer · 2020-08-16T09:36:47.550

1

Since you're having install issues according to a comment, I would especially suggest using numba-scipy instead, from the creators of numba itself (link to pypi). It's always a good idea to use first-party extensions to packages.

The very documentation you linked relates to numba_special. If you look at the main page you'll see the first example:

>>> import numba
>>> import scipy.special as sc
>>> import numba_special  # The import generates Numba overloads for special
>>> @numba.njit
... def gamma_plus_1(x):
...     return sc.gamma(x) + 1.0
...
>>> gamma_plus_1(5.0)
25.0

Note the import numba_special line you're missing.

You will also have to install numba-special first. Also note the warning on the project's page:

Installing

Numba special depends on SciPy and Numba. Until the fix for this issue is included in a Numba release, numba_special must be installed against the master branch of Numba.

edited Aug 16 '20 at 09:36

answered Aug 16 '20 at 09:03

Andras Deak -- Слава Україні

33,737
11
83
111

Great, but now I cannot install it... ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. – Yair Daon Aug 16 '20 at 09:13
@Yair as the error says you have to read the rest of what was printed. There should be a mostly helpful error message. – Andras Deak -- Слава Україні Aug 16 '20 at 09:27
@Yair see my update: I suggest using numba's own extension package. – Andras Deak -- Слава Україні Aug 16 '20 at 09:29
Thanks! Can you give a usage example? I cannot get it to work (see updated question). – Yair Daon Aug 17 '20 at 07:12
@YairDaon I'll tell you when I get a response on https://github.com/numba/numba-scipy/issues/44 :) Things don't look good, but fingers crossed. – Andras Deak -- Слава Україні Aug 17 '20 at 10:39

score 0 · Answer 2 · answered Aug 16 '20 at 09:29

You are calling xlogy function from scipy.special, actually for the functions called by @jit decorated functions also need to be @jit decorated. So that's why xlogy is not compatible with your f function and numba throwing error. The workaround could be:

import numpy as np
from numba import jit


@jit(nopython=True)
def myxlogy(x,y):
    return x*np.log(y)


@jit(nopython=True)
def f(a):
    return myxlogy(a,a)

a = np.array([1,2,0,1], dtype=float)
f(a)
>>array([0.        , 1.38629436,        nan, 0.        ])

Actually this is not a good solution. We want 0log(0) = 0 and this solution does not do that. — Yair Daon, Oct 25 '22 at 13:45

Scipy special functions with numba

2 Answers2