Plotting a heat map from three lists: X, Y, Intensity

Question

I don't get how to create a heatmap (or contour plot) when I have x, y, intensity. I have a file which looks like this:

0,1,6
0,2,10
....

So far:

with open('eye_.txt', 'r') as f:
        for line in f:
                for word in line.split():
                        l = word.strip().split(',')
                        x.append(l[0])
                        y.append(l[1])
                        z.append(l[2])

Tried using pcolormesh but it wants a shape object and I'm unsure how to convert these lists into a NumPy array.

I tried:

i,j = np.meshgrid(x,y)
arr = np.array(z)
plt.pcolormesh(i,j,arr)
plt.show()

It tells me that:

IndexError: too many indices

Can someone stop me from bashing my head against a keyboard, please?

score 9 · Accepted Answer · answered Nov 26 '15 at 16:27

OK, there's a few steps to this.

First, a much simpler way to read your data file is with numpy.genfromtxt. You can set the delimiter to be a comma with the delimiter argument.

Next, we want to make a 2D mesh of x and y, so we need to just store the unique values from those to arrays to feed to numpy.meshgrid.

Finally, we can use the length of those two arrays to reshape our z array.

(NOTE: This method assumes you have a regular grid, with an x, y and z for every point on the grid).

For example:

import matplotlib.pyplot as plt
import numpy as np

data = np.genfromtxt('eye_.txt',delimiter=',')

x=data[:,0]
y=data[:,1]
z=data[:,2]

## Equivalently, we could do that all in one line with:
# x,y,z = np.genfromtxt('eye_.txt', delimiter=',', usecols=(0,1,2))

x=np.unique(x)
y=np.unique(y)
X,Y = np.meshgrid(x,y)

Z=z.reshape(len(y),len(x))

plt.pcolormesh(X,Y,Z)

plt.show()

score 7 · Answer 2 · answered Jun 16 '17 at 22:11

In case you don't have a regular grid (i.e. a value of z for each meshgrid value of x and y), you can use a more general method based on pandas data frames:

import pandas as pd
import seaborn as sns
import matplotlib.pypot as plt

data = pd.DataFrame(data={'x':x, 'y':y, 'z':z})
data = data.pivot(index='x', columns='y', values='z')
sns.heatmap(data)
plt.show()

The pivot method will use unique values from index and columns to construct a table with missing measurments set to NaN. The table can then be plotted as a heatmap.

I have the same problem which is mentioned above. I have 16,000 entries and 16 GB ram, because of the large data, table does not fit. I am not sure why this is happening, is there any other way to do this? — newbie, Nov 04 '19 at 17:44

thomas · Answer 3 · 2015-11-26T17:14:02.033

The index error arises from the fact that pcolormesh expects a 2D array while your arr is a 1D vector. Also if I understand correctly, your input file has the form

0,1,z
0,2,z
 ...
0,ymax,z
 ...
1,1,z
1,2,z
 ...
xmax,ymax,z

In that case meshgrid(x,y) will not work as it expects something like meshgrid(range(xmax),range(ymax)) i.e. vectors without repeated values.

In your case you need to find out how many distinct x and y values there are and then simply reshape your vectors into 2D arrays.

shape = np.unique(x).shape[0],np.unique(y).shape[0]
x_arr = x.reshape(shape)
y_arr = y.reshape(shape)
z_arr = z.reshape(shape)
plt.pcolormesh(x_arr,y_arr,z_arr)

score 0 · Answer 4 · answered Nov 26 '15 at 16:28

To convert a list into a numpy array you can use np.asarray.

Here's an easy way to get a heatmap, you should be able to adapt this example to your problem.

import matplotlib.pyplot as plt
import numpy as np
a = [[1,2,3], [3,4,5], [5,6,7], [7, 8, 9]]
b = np.asarray(a)
plt.pcolor(b)
plt.show()

To read the data in like a list of lists you can do:

a = []
for line in file.readlines():
    a.append( map( int, line.split(',') ) )

in short. In a longer version it's the equivalent of:

a = []
for line in file.readlines():
    tmp = line.split(',')
    inttmp = [int(x) for x in a]
    a.append(inttmp)

Plotting a heat map from three lists: X, Y, Intensity

4 Answers4

Linked