why does math.log accepts big integer values?

Question

from math import log,sqrt
import sys
n = 760 ** 890
print(log(n))

I get a valid result.

Now change log by sqrt and you get (as expected):

OverflowError: int too large to convert to float

So I suppose there's a trick for integer arguments in log function, using integer logarithms but I didn't find that in the documentation. There's just this:

math.log(x[, base])

With one argument, return the natural logarithm of x (to base e).

With two arguments, return the logarithm of x to the given base, calculated as log(x)/log(base).

Where is that documented?

*"**CPython implementation detail:** The math module consists mostly of thin wrappers around the platform C math library functions."* - I think that includes `log`, so this is potentially platform-dependent and not necessarily a Python question. — jonrsharpe, Dec 31 '17 at 13:30
Do you really get a valid result for `760 ** 890`?. On my machine (osx) it is `5903.65340562` but wolfram alpha says: `451.131905409` — user1767754, Dec 31 '17 at 13:34
@user1767754 Try 890 * log(760) to confirm that the result is correct. Not sure, what you found on wolfram alpha. — Mr. T, Dec 31 '17 at 13:38
@jonrsharpe in C you don't have big numbers (only `long long` which range usually to 2**64 so not really enough), and integers are converted to `double` by the prototype so in C this question is irrelevant. — Jean-François Fabre, Dec 31 '17 at 15:54
@user1767754 what is wolfram alpha? because it doesn't seem to compute logs properly. — Jean-François Fabre, Dec 31 '17 at 15:55
@Jean-FrançoisFabre: Wolfram Alpha is a well known website: https://www.wolframalpha.com/ . It is great for math: for "760**890" it generates a 2546 decimal digit answer (as a picture, so not so easy to copy). So it knows quite well. — Rudy Velthuis, Dec 31 '17 at 19:49
@Jean-FrançoisFabre: log(760*890) returns "5903.653405619535318946719985244828084080638106597086949503....". in Wolfram Alpha. That seems to be correct (tested it with my own Delphi implementation of BigInteger). — Rudy Velthuis, Dec 31 '17 at 20:19
@user1767754: Wolfram Alpha returns 5903.etc for me. Where did you get the 451.etc value? — Rudy Velthuis, Dec 31 '17 at 20:23

score 4 · Accepted Answer · edited Oct 24 '21 at 21:36

I finally dug into python math lib source code and found this:

/* A decent logarithm is easy to compute even for huge ints, but libm can't
   do that by itself -- loghelper can.  func is log or log10, and name is
   "log" or "log10".  Note that overflow of the result isn't possible: an int
   can contain no more than INT_MAX * SHIFT bits, so has value certainly less
   than 2**(2**64 * 2**16) == 2**2**80, and log2 of that is 2**80, which is
   small enough to fit in an IEEE single.  log and log10 are even smaller.
   However, intermediate overflow is possible for an int if the number of bits
   in that int is larger than PY_SSIZE_T_MAX. */

static PyObject*
loghelper(PyObject* arg, double (*func)(double), const char *funcname)
{
    /* If it is int, do it ourselves. */
    if (PyLong_Check(arg)) {
        double x, result;
        Py_ssize_t e;

        ...

I'll spare you the rest of the source (check the link), but what I understand from it is that Python checks if the passed argument is integer, and if it is, don't use math lib (If it is int, do it ourselves.) comment. Also: A decent logarithm is easy to compute even for huge ints, but libm can't do that by itself -- loghelper can

If it's a double, then call native math library.

From the source comments, we see that Python tries the hardest to provide the result even in case of the overflow (Here the conversion to double overflowed, but it's possible to compute the log anyway. Clear the exception and continue)

So thanks to the python wrapping of the log function, Python is able to compute logarithm of huge integers (which is specific to some functions, since some others like sqrt cannot do it), and it's documented, but only in the source code, probably making it an implementation detail as Jon hinted.

score 1 · Answer 2 · edited Oct 24 '21 at 21:38

I think this thread is useful since python now uses long ints, the trick to avoid overflow is the use of _PyLong_Frexp function see here and an alternative formula to compute the log function even after an OverflowError is raised when trying to convert a long int to a Double, check loghelper at this module.

_PyLong_Frexp returns an approximation to the initial long int arg, given inside loghelper with the help of a double x and an exponent e (arg~x*2**e) and the log is calculated as log~log(x*2**e)=log(x)+log(2)*e. I am missing the specifics of the approximation using x,e but you can find it in the implementation of _PyLong_Frexp in the link provided.

why does math.log accepts big integer values?

2 Answers2