Python: handling large numbers

Question

I need to count perplexity and I try to do it with

def get_perplexity(test_set, model):
    perplexity = 1
    n = 0
    for word in test_set:
        n += 1
        perplexity = perplexity * 1 / get_prob(model, word)
    perplexity = pow(perplexity, 1/float(n))
    return perplexity

And after some steps my perplexity is equal to infinity. I need to get number and as last step to do pow(perplexity, 1/float(n))

Is any to multiply numbers like and get result?

3.887311155784627e+243
8.311806360146177e+250
1.7707049372801292e+263
1.690802669602979e+271
3.843294667766984e+278
5.954424789834101e+290
8.859529887856071e+295
7.649470766862909e+306

You could try taking the `log` of all of the numbers, and then add, rather than multiply. — baum, Dec 16 '18 at 18:03
Perhaps related: https://stackoverflow.com/questions/538551/handling-very-large-numbers-in-python — Jens, Dec 16 '18 at 18:10

alexgolec · Answer 1 · 2018-12-16T18:21:41.853

1

The repeated multiplication is going to cause some tricky numerical instability as the results of your multiplications require more and more bits to represent. I propose you translate this into log-space and use summation rather than multiplication:

import math

def get_perplexity(test_set, model):
    log_perplexity = 0
    n = 0
    for word in test_set:
        n += 1
        log_perplexity -= math.log(get_prob(model, word))
    log_perplexity /= float(n)
    return math.exp(log_perplexity)

This way all your logarithms can be represented in the standard number of bits, and you don't get numerical blowups and loss of precision. Also, you can introduce an arbitrary degree of precision by using the decimal module:

import decimal

def get_perplexity(test_set, model):
    with decimal.localcontext() as ctx:
        ctx.prec = 100  # set as appropriate
        log_perplexity = decimal.Decimal(0)
        n = 0
        for word in test_set:
            n += 1
            log_perplexity -= decimal.Decimal(get_prob(model, word))).ln()
        log_perplexity /= float(n)
        return log_perplexity.exp()

edited Dec 16 '18 at 18:21

answered Dec 16 '18 at 18:15

alexgolec

26,898
33
107
159

Could your explain, why it should return math.exp(log_perplexity), not square of log_perplexity? – Petr Petrov Dec 16 '18 at 18:42
That doesn't make algebraic sense. math.exp is the inverse of math.log. Squaring it would be appropriate if I had taken the square root. – alexgolec Dec 16 '18 at 18:57
And why you use `-=` in `log_perplexity -= math.log(get_prob(model, word))`? – Petr Petrov Dec 16 '18 at 19:03
Unfortunately these comments aren't the best place to write this out because you need some notation. Basically, you're computing perplexity as "the n-th root of the product of one over the probability of each word in the test set." If you take the logarithm of both sides and perform some manipulation you get that the log of the perplexity is "negative one over n, times the sum of the logarithms of the prob of each word in the test set." e to the power of that gives you the perplexity. – alexgolec Dec 16 '18 at 19:09

Ryabchenko Alexander · Answer 2 · 2018-12-16T18:30:48.493

since e+306 is just 10^306 you can make class of two parts

class BigPowerFloat:
    POWER_STEP = 10**100
    def __init__(self, input_value):
        self.value = float(input_value)
        self.power = 0

    def _move_to_power(self):
        while self.value > self.POWER_STEP:
            self.value = self.value / self.POWER_STEP
            self.power += self.POWER_STEP
        # you can add similar for negative values           


    def __mul__(self, other):
        self.value *= other
        self._move_to_power()

    # TODO other __calls for /, +, - ...

    def __str__(self):
        pass
        # make your cust to str

Python: handling large numbers

2 Answers2