11

I see a lot of questions like this one for R, but I couldn't find one specifically for Python, preferably using numpy.

Let's say I have an array of observations stored in x. I can get the value that accumulates q * 100 per cent of the population.

# Import numpy
import numpy as np

# Get 75th percentile
np.quantile(a=x, q=0.75)

However, I was wondering if there's a function that does the inverse. That is, a numpy function that takes a value as an input and returns q.

To further expand on this, scipy distribution objects have a ppf method that allows me to do this. I'm looking for something similar in numpy. Does it exist?

Mad Physicist
  • 107,652
  • 25
  • 181
  • 264
Arturo Sbr
  • 5,567
  • 4
  • 38
  • 76

4 Answers4

15

Not a ready-made function but a compact and reasonably fast snippet:

(a<value).mean()

You can (at least on my machine) squeeze out a few percent better performance by using np.count_nonzero

np.count_nonzero(a<value) / a.size

but tbh I wouldn't even bother.

loopy walt
  • 868
  • 2
  • 6
  • 1
    This incidentally is what the `strict` mode of `scipy.stats.percentileofscore` does in its source code: `np.count_nonzero(a < score) / n * 100`. The `weak` mode just does `<=` instead of `<`. I did a quick `%timeit` test and it appears this is faster than the `(a – BatWannaBe Sep 06 '21 at 05:19
  • Thank you; an answer to this useful question was weirdly hard to find. – Markus Nov 10 '21 at 05:31
  • This is great! @loopywalt care to explain your thinking process in coming up with this? It works of course, but I struggle to abstractly understand why. Thank you! – Tim Aug 03 '22 at 14:30
7

There's a convenience function that does this. Note that it's not an exact inverse because the quantile/percentile functions are not exact. Given a finite array of observations, the percentiles will have discrete values; in other words, you may be specifying a q that falls between those values and the functions find the closest one.

from scipy import stats
import numpy as np

stats.percentileofscore(np.arange(0,1,0.12), .65, 'weak') / 100
BatWannaBe
  • 4,330
  • 1
  • 14
  • 23
2

If x is sorted, the value at index i is the i / len(x) percentile (or so, depending on how you want to treat boundary conditions). If x is not sorted, you can obtain the same value by substituting x.argsort().argsort()[i] for i (or just sorting x first). Since argsort is its own inverse, the double argsort tells you where each element of the original would fall in the sorted array.

If you want to find the result for arbitrary values not necessarily in x, you can apply np.searchsorted to a sorted version of x and interpolating on the result. You can use a more complicated method, like fitting a spline to the sorted data or something similar.

Mad Physicist
  • 107,652
  • 25
  • 181
  • 264
0

While vals = x.argsort().argsort()/(x.size-1) works in arrays with fully unique values, it fails if you have repeated values. Identical values should have the same quantile value, but for example, if the array x had 200 values of zeros and 800 values larger than zero, then this method would give 200 different quantile values to those zero values. Safer to use vals = np.array([np.count_nonzero(x<x_i)/(x.size-1) for x_i in x]), since identical values get identical quantile positions then.

import numpy as np

def get_quant(x):
  " for each value in x, return which quantile it corresponds to "
  return np.array([np.count_nonzero(x<x_i)/(len(x)-1) for x_i in x])

Note: the (x.size-1) denominators ensure the quantile values range from 0 to 1 inclusive. Leaving out the -1 means the 100% quantile is never reached.

William Black
  • 53
  • 4