SciPy error: RuntimeWarning: overflow encountered in _beta_ppf

Question

My SciPy version is 1.7.3 and I'm running on an Apple M1 chip (not sure if it's relevant). My Python version is 3.9.11, installed via Annaconda.

I get the error message /opt/anaconda3/lib/python3.9/site-packages/scipy/stats/_continuous_distns.py:624: RuntimeWarning: overflow encountered in _beta_ppf return _boost._beta_ppf(q, a, b) when running the following PyMC3 code:

import pymc3 as pm
from scipy import stats

Y = stats.bernoulli(0.7).rvs(20)

with pm.Model() as model:
    theta = pm.Beta("theta", alpha=0.1, beta=0.1)
    y_obs = pm.Binomial("y_obs", n=2, p=theta, observed=Y)
    idata = pm.sample(1000, return_inferencedata=True)

I don't think the issue is with PyMC3, because the PyMC3 code someone else wrote runs without any errors:

np.random.seed(123)
n_experiments=4
theta_real =0.35
data =stats.bernoulli.rvs(p=theta_real,size=n_experiments)
with pm.Model()  as our_first_model:
    theta =pm.Beta('theta',alpha=1,beta=1)
    y =pm.Bernoulli('y',p=theta,observed=data)
    start =pm.find_MAP()
    step =pm.Metropolis()
    trace =pm.sample(1000,step=step,start=start)
    burnin=100
    chain =trace[burnin:]
    pm.traceplot(chain,lines={'theta':theta_real})

The SciPy code snippet below, however, gives the same error:

import numpy as np
from scipy import stats
q = 0.999995
a = np.array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
b = np.array([99999, 99998, 99997, 99996, 99995, 99994, 99993, 99992, 99991, 99990, 99989, 99988, 99987, 99986, 99985, 99984, 99983, 99982, 99981])
stats.beta.ppf(q, a, b)

I Googled "overflow encountered in _beta_ppf" and found a GitHub issue (https://github.com/scipy/scipy/issues/14901) that points to problems with SciPy <=1.7.2 on Intel chips. However, I'm using 1.7.3 on an M1 chip.

Anaconda doesn't allow me to update to the newest SciPy version (1.8.0). Using pip to update (pip install scipy==1.8.0 ) results in incompatibilities with ArviZ (arviz 0.11.2 requires typing-extensions<4,>=3.7.4.3, but you have typing-extensions 4.1.1 which is incompatible). I guess I could downgrade typing-extensions, but it may cause further issues with other packages, so I'd rather not.

I wonder what I can do to solve this problem. Thanks!

"*however, I'm...on an M1 chip*" - yes, but you installed Anaconda, which doesn't have native M1 support and so must be running with x86_64 emulation. Check your `conda info` to see if platform is **osx-64** or **osx-arm64**. I wouldn't be surprised if compiled libraries (e.g., `boost`) using architecture-specific optimizations couldn't emulate properly (but that's just conjecture). — merv, Apr 04 '22 at 04:40
Just checked and says the platform is osx-64. I wonder what this then means in terms of package compatibility. — ramund, Apr 04 '22 at 06:25
It is possible to create native environments: https://stackoverflow.com/a/71463174/570918 — merv, Apr 04 '22 at 07:18
@OriolAbril That was my first thought by Anaconda also doesn't allow upgrading to the latest version of arviz. Sigh... — ramund, Apr 04 '22 at 17:43

score 1 · Answer 1 · answered Apr 04 '22 at 05:42

1

If you cannot upgrade scipy to 1.8.0, you can always downgrade it to 1.6.x (IIRC boost is not used in 1.6.x series)

answered Apr 04 '22 at 05:42

ev-br

24,968
9
65
78

Yup, downgrading to 1.6.2 does solve the problem! A ton of thanks for the suggestion! I hope that on the long run, 1.8.0 will become compatible... – ramund Apr 04 '22 at 17:43

score 0 · Answer 2 · answered Apr 11 '22 at 07:56

Native Conda Environment

FWIW, I use native Mambaforge for osx-arm64, and have no problem with creating a native Python v3.10 with SciPy v1.8.0. If you already have an Anaconda/Miniconda base (i.e., osx-64), you should be able to create a native environment with:

Shell Session

conda create -n arm64
conda activate arm64
conda config --env --add channel conda-forge
conda config --env --set subdir osx-arm64
conda install -y scipy=1.8 pymc3
python

Python Session

Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:39:37) [Clang 12.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from scipy import stats
>>> q = 0.999995
>>> a = np.array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
>>> b = np.array([99999, 99998, 99997, 99996, 99995, 99994, 99993, 99992, 99991, 99990, 99989, 99988, 99987, 99986, 99985, 99984, 99983, 99982, 99981])
>>> stats.beta.ppf(q, a, b)
array([0.00014976, 0.00017332, 0.00019478, 0.00021492, 0.0002341 ,
       0.00025256, 0.00027046, 0.00028788, 0.00030491, 0.00032161,
       0.00033802, 0.00035417, 0.00037009, 0.0003858 , 0.00040134,
       0.0004167 , 0.00043192, 0.00044699, 0.00046193])

Just make sure to activate the environment before installing additional packages.

Also, seems to not have any error with scipy=1.7.3 either.

SciPy error: RuntimeWarning: overflow encountered in _beta_ppf

2 Answers2

Native Conda Environment