Pearson correlation matrix in Python, ValueError

Question

I'm trying to get a matrix, (lat, lon) size, with the Pearson Coefficient value for every grid point, for

x : a 3D DataArray (time, lat, lon) (time size is 30)

y : a DataArray column vector with a 30 values series inside

So i would like to calculate the pearson coefficient for every (lat,lon) for a column vector of 30 elements for x.

I tried:

corrmap = xr.DataArray(x2)
for i in range(len(corrmap['lat']))
     for j in range(len(corrmap['lon']))
          corrmap[i, j], p_value = pearsonr(x[:, i, j], y)

but i get this error:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

that i cannot perfectly understand in the meaning. Is my method uncorrect? Should i use another type of code to solve my problem?

Any help would be greatly appreciated.

What is `y.shape`? (I'm assuming that a `DataArray` object has a `shape` attribute.) — Warren Weckesser, Apr 09 '19 at 16:36
Is `pearsonr` imported from `scipy.stats`? If not, where is it defined? — Warren Weckesser, Apr 09 '19 at 16:38
sorry I forgot to say that y.shape is (30, 1), so a column vector. And yes I did, i correctly imported pearsonr from scipy.stats — kadau92, Apr 09 '19 at 17:18
*"... y.shape is (30, 1), so a column vector."* That's the problem. See @JulianGiles' answer. — Warren Weckesser, Apr 09 '19 at 20:31

score 2 · Answer 1 · answered Apr 09 '19 at 19:30

The problem is you are using y and it has two dimensions, pearsonr can't handle that. Specify y[:,0] and it works. That is:

corrmap = np.zeros(((len(corrmap['lat']), len(corrmap['lon'])))

for i in range(len(corrmap['lat'])):
     for j in range(len(corrmap['lon'])):
          corrmap[i, j], p_value = pearsonr(x[:, i, j], y[:,0])

Also I would just use a numpy array for the coefficients instead of an xarray, at least to get the values, then you can convert it to xarray.

score 0 · Answer 2 · answered Apr 09 '19 at 17:06

0

Assuming that you are using scipy.stats.pearsonr. The arguments to PearsonR should be one-dimensional arrays.

So give a go to:

corrmap[i, j], p_value = pearsonr(x[:, i, j].ravel(), y)

What ravel() does is return contiguous flattened array, Numpy ravel()

Here is also a bit of context around the error that you are getting, evaluate array in boolean context. In other words, the extra dimension is probably causing some operations to be applied to an array instead of a scalar. This issue is of the same nature as yours: SO: PearsonR ValueError

answered Apr 09 '19 at 17:06

Konstantin Grigorov

1,356
12
20

Thank you very much for your answer. Yes i correctly imported pearsonr from scipy.stats. Anyway, making the correction you suggest (that i understand and approve) i get an AttributeError, because of the fact that "'DataArray object' has no attribute 'ravel'. Is it a mistake to inizialize corrmap at the beginning as a xr.DataArray with a (lat, lon) shape? – kadau92 Apr 09 '19 at 17:24

score 0 · Answer 3 · answered Feb 13 '21 at 00:59

0

You can calutate the Pearson correlation coefficient using:

import numpy
numpy.corrcoef(list1, list2)[0, 1]

answered Feb 13 '21 at 00:59

Bendy Latortue

391
5
6

Pearson correlation matrix in Python, ValueError

3 Answers3