In Python, how can I calculate correlation and statistical significance between two arrays of data?

Question

I have sets of data with two equally long arrays of data, or I can make an array of two-item entries, and I would like to calculate the correlation and statistical significance represented by the data (which may be tightly correlated, or may have no statistically significant correlation).

I am programming in Python and have scipy and numpy installed. I looked and found Calculating Pearson correlation and significance in Python, but that seems to want the data to be manipulated so it falls into a specified range.

What is the proper way to, I assume, ask scipy or numpy to give me the correlation and statistical significance of two arrays?

Why do you think that `scipy.stats.pearsonr` wants the data to be manipulated so it falls into a specified range? — ecatmur, Jun 20 '12 at 14:29
The _correlation coefficient_ falls between -1 and +1. That's the output, not the input. — Janne Karila, Jun 20 '12 at 14:50

score 7 · Answer 1 · answered Jun 20 '12 at 14:55

If you want to calculate the Pearson Correlation Coefficient, then scipy.stats.pearsonr is the way to go; although, the significance is only meaningful for larger data sets. This function does not require the data to be manipulated to fall into a specified range. The value for the correlation falls in the interval [-1,1], perhaps that was the confusion?

If the significance is not terribly important, you can use numpy.corrcoef().

The Mahalanobis distance does take into account the correlation between two arrays, but it provides a distance measure, not a correlation. (Mathematically, the Mahalanobis distance is not a true distance function; nevertheless, it can be used as such in certain contexts to great advantage.)

score 2 · Answer 2 · edited May 23 '17 at 12:23

2

You can use the Mahalanobis distance between these two arrays, which takes into account the correlation between them.

The function is in the scipy package: scipy.spatial.distance.mahalanobis

There's a nice example here

edited May 23 '17 at 12:23

Community

1
1

answered Jun 20 '12 at 14:37

Oriol Nieto

5,409
6
32
38

score 0 · Answer 3 · answered Jun 20 '12 at 14:55

scipy.spatial.distance.euclidean()

This gives euclidean distance between 2 points, 2 np arrays, 2 lists, etc

import scipy.spatial.distance as spsd
spsd.euclidean(nparray1, nparray2)

You can find more info here http://docs.scipy.org/doc/scipy/reference/spatial.distance.html

In Python, how can I calculate correlation and statistical significance between two arrays of data?

3 Answers3

Linked