3

I have some code in matlab, that I would like to rewrite into python. It's simple program, that computes some distribution and plot it in double-log scale.

The problem I occured is with computing cdf. Here is matlab code:

for D = 1:10
    delta = D / 10;
    for k = 1:n
        N_delta = poissrnd(delta^-alpha,1);
        Y_k_delta = ( (1 - randn(N_delta)) / (delta.^alpha) ).^(-1/alpha);
        Y_k_delta = Y_k_delta(Y_k_delta > delta);
        X(k) = sum(Y_k_delta);
        %disp(X(k))

    end
    [f,x] = ecdf(X);

    plot(log(x), log(1-f))
    hold on
end

In matlab one I can simply use:

[f,x] = ecdf(X);

to get cdf (f) at points x. Here is documentation for it.
In python it is more complicated:

import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
from statsmodels.distributions.empirical_distribution import ECDF

alpha = 1.5
n = 1000
X = []
for delta in range(1,5):
    delta = delta/10.0
    for k in range(1,n + 1):
        N_delta = np.random.poisson(delta**(-alpha), 1)
        Y_k_delta = ( (1 - np.random.random(N_delta)) / (delta**alpha) )**(-1/alpha)
        Y_k_delta = [i for i in Y_k_delta if i > delta]
        X.append(np.sum(Y_k_delta))

    ecdf = ECDF(X)

    x = np.linspace(min(X), max(X))
    f = ecdf(x)
    plt.plot(np.log(f), np.log(1-f))

plt.show()

It makes my plot look very strange, definetly not smooth as matlab's one.
I think the problem is that I do not understand ECDF function or it works differently than in matlab.
I implemented this solution (the most points one) for my python code, but it looks like it doesn't work correctly.

Community
  • 1
  • 1
Photon Light
  • 757
  • 14
  • 26
  • 1
    1. I suggest adding figures to your questions in these cases, it helps a lot. 2. are you familiar with matlab's `loglog` and `plt.loglog`? – Andras Deak -- Слава Україні Oct 26 '15 at 12:34
  • 1. Yea, I know, I need one time to learn how to attach those... 2. No,I have never used them. – Photon Light Oct 26 '15 at 13:28
  • 1. just click the little landscape icon in the editor's menu, you can then include an image from your computer or from the web through a link. 2. I suggest you check those out: they work like the respective `plot`s, with a bit more, and they take care of the `log`s for you. It's usually even better: they keep the *log scale* while preserving your original variables: if your `x` goes from 1 to 1000 then the final `xticklabel`s will not write `log(1000)`, but `1000`, and the ticks will be chosen logarithmically. This is not the same as plotting `log` vs `log`, so it's up to your needs. – Andras Deak -- Слава Україні Oct 26 '15 at 14:24

1 Answers1

5

Once you have your sample, you can easily compute the ECDF using a combination of np.unique* and np.cumsum:

import numpy as np

def ecdf(sample):

    # convert sample to a numpy array, if it isn't already
    sample = np.atleast_1d(sample)

    # find the unique values and their corresponding counts
    quantiles, counts = np.unique(sample, return_counts=True)

    # take the cumulative sum of the counts and divide by the sample size to
    # get the cumulative probabilities between 0 and 1
    cumprob = np.cumsum(counts).astype(np.double) / sample.size

    return quantiles, cumprob

For example:

from scipy import stats
from matplotlib import pyplot as plt

# a normal distribution with a mean of 0 and standard deviation of 1
n = stats.norm(loc=0, scale=1)

# draw some random samples from it
sample = n.rvs(100)

# compute the ECDF of the samples
qe, pe = ecdf(sample)

# evaluate the theoretical CDF over the same range
q = np.linspace(qe[0], qe[-1], 1000)
p = n.cdf(q)

# plot
fig, ax = plt.subplots(1, 1)
ax.hold(True)
ax.plot(q, p, '-k', lw=2, label='Theoretical CDF')
ax.plot(qe, pe, '-r', lw=2, label='Empirical CDF')
ax.set_xlabel('Quantile')
ax.set_ylabel('Cumulative probability')
ax.legend(fancybox=True, loc='right')

plt.show()

enter image description here

* If you're using a version of numpy older than 1.9.0 then np.unique won't accept the return_counts keyword argument, and you'll get a TypeError:

TypeError: unique() got an unexpected keyword argument 'return_counts'

In that case, a workaround would be to get the set of "inverse" indices and use np.bincount to count the occurrences:

quantiles, idx = np.unique(sample, return_inverse=True)
counts = np.bincount(idx)
ali_m
  • 71,714
  • 23
  • 223
  • 298
  • 1
    It looks quite good, thank you! I only changed 'return_counts=True' for just 'True', since I've got error in this version. BTW good luck with your thesis! – Photon Light Oct 26 '15 at 13:27
  • If you got the error `unexpected keyword argument 'return_counts'` then you must be running an older version of numpy that doesn't support the `return_counts` argument to `np.unique` (this was added in v1.9). In that case, calling `np.unique(..., True)` would correspond to `np.unique(..., return_index=True)` which will give you the indices of the unique values rather than their counts as the second return variable, and therefore your CDF will be incorrect. See my update for a workaround. – ali_m Oct 26 '15 at 13:48