plot a huge amount of data points

Question

I have encountered a strange problem: when I store a huge amount of data points from a nonlinear equation to 3 arrays (x, y ,and z) and then tried to plot them in a 2D graph (theta-phi plot, hence its 2D).

I tried to eliminate points needed to be plotted by sampling points from every 20 data points, since the z-data is approximately periodic. I picked those points with z value just above zero to make sure I picked one point for every period.

The problem arises when I tried to do the above. I got only a very limited number of points on the graph, approximately 152 points, regardless of how I changed my initial number of data points (as long as it surpassed a certain number of course).

I suspect that it might be some command I use wrongly or the capacity of array is smaller then I expected (seems unlikely), could anyone help me find out where is the problem?

def drawstaticplot(m,n, d_n, n_o):
    counter=0
    for i in range(0,m):
        n=vector.rungekutta1(n, d_n)
        d_n=vector.rungekutta2(n, d_n, i)
        x1 = n[0]    
        y1 = n[1]
        z1 = n[2]
        if i%20==0:
            xarray.append(x1)
            yarray.append(y1)
            zarray.append(z1)
    for j in range(0,(m/20)-20):
        if (((zarray[j]-n_o)>0) and ((zarray[j+1]-n_o)<0)):
           counter= counter +1
           print zarray[j]-n_o,counter
           plotthetaphi(xarray[j],yarray[j],zarray[j])

def plotthetaphi(x,y,z):
    phi= math.acos(z/math.sqrt(x**2+y**2+z**2))
    theta = math.acos(x/math.sqrt(x**2 + y**2))
    plot(theta, phi,'.',color='red')

Besides, I tried to apply the code in the following SO question to my code, I want a very similar result except that my data points are not randomly generated.

Answer 1

Shiuan,

I am still investigating your problem, how ever a few notes:

Instead of looping and appending to an array you could do:

select every nth element:

# inside IPython console:
[2]: a=np.arange(0,10)

In [3]: a[::2] # here we select every 2nd element.
Out[3]: array([0, 2, 4, 6, 8])

so instead of calcultating runga-kutta on all elements of m:

new_m = m[::20] # select every element of m.

now call your function like this:

def drawstaticplot(new_m,n, d_n, n_o):
    n=vector.rungekutta1(n, d_n)
    d_n=vector.rungekutta2(n, d_n, i)
    x1 = n[0]    
    y1 = n[1]
    z1 = n[2]
    xarray.append(x1)
    yarray.append(y1)
    zarray.append(z1)
    ...

about appending, and iterating over large data sets:

append in general is slow, because it copies the whole array and then stacks the new element. Instead, you already know the size of n, so you could do:

def drawstaticplot(new_m,n, d_n, n_o):
    # create the storage based on n,
    # notice i assumed that rungekutta, returns n the size of new_m, 
    # but you can change it.
    x,y,z = np.zeros(n.shape[0]),np.zeros(n.shape[0]), np.zeros(n.shape[0])

for idx, itme in enumerate(new_m): # notice the function enumerate, make it your friend!
    n=vector.rungekutta1(n, d_n)
    d_n=vector.rungekutta2(n, d_n, ite,)
    x1 = n[0]    
    y1 = n[1]
    z1 = n[2]
    #if i%20==0: # we don't need to check for the 20th element, m is already filtered...
    xarray[idx] = n[0]
    yarray[idx] = n[1]
    zarray[idx] = n[2]
    # is the second loop necessary?
    if (((zarray[idx]-n_o)>0) and ((zarray[j+1]-n_o)<0)): 
       print zarray[idx]-n_o,counter
       plotthetaphi(xarray[idx],yarray[idx],zarray[idx])

    

Answer 2

You can use the approach suggested here: Efficiently create a density plot for high-density regions, points for sparse regions e.g. histogram where you have too many points and points where the density is low. Or also you can use rasterized flag for matplotlib, which speeds up matplotlib.