When trying to calculate numpy corrcoef
cc = np.corrcoef(y, y2)
where shapes of y and y2 are
<class 'tuple'>: (32383, 1)
My computer hangs. Not only interpreter but entire computer.
How can this be and how to fix?
Asked
Active
Viewed 638 times
0
Dims
- 47,675
- 117
- 331
- 600
-
What are your computer specs? – DarkCygnus Jul 07 '17 at 16:02
-
Probably you get the dimension wrong and it's returning 32k x 32k matrix, instead of a scalar – jf328 Jul 07 '17 at 16:02
-
1Have you taken a look to [this](https://stackoverflow.com/questions/30143417/computing-the-correlation-coefficient-between-two-multi-dimensional-arrays) question? – DarkCygnus Jul 07 '17 at 16:05
-
@GrayCygnus 64 bit, 32 Gb RAM, GTX 1070; I don't understand why it is hang totally – Dims Jul 07 '17 at 17:56
-
@jf328 should it hang it the case or complain OOM? – Dims Jul 07 '17 at 17:57
2 Answers
0
According to documentation, by default "each row represents a variable, with observations in the columns". Having shape (32383, 1) means 32383 variables with 1 observation for each, which is totally meaningless for correlation purposes. So, the computer behavior aside, the way to proceed is
cc = np.corrcoef(y, y2, rowvar=False)
indicating that your columns correspond to variables.
0
Computing the correlation coefficient between two shape (32383, 1) arrays will result in a (32383, 32383) array. Such an array will have size (32383 * 32383 * 8) / (1024 ** 3) = 7.8 Gb. Considering that you likely have some overhead in computing the result, you likely need in the order of tens of Gb's to compute this.
What is probably happening is not that your computer hangs, but that the computation overflows your RAM and starts to perform paging, that is, uses the hard-drive to store the partial results. This is exceedingly slow and it will thus seem like your computer has hung. Given enough time, it should give a result.
If you instead want to compute the row-wise cross correlation (which has shape (1, 1)), you can do:
cc = np.corrcoef(y, y2, rowvar=False)
Jonas Adler
- 10,365
- 5
- 46
- 73
-
When computer overloads paging it doesn't turn to complete brick. In this case even mouse stops moving. How can it be? – Dims Jul 09 '17 at 15:35