10

Quick question: Is there a way to use 'dropna' with the Pearson's r function in scipy? I'm using it in conjunction with pandas, and some of my data has holes in it. I know you used to be able suppress 'nan' with Spearman's r in older versions of scipy, but that functionality is now missing.

To my mind, this seems like a disimprovement, so I wonder if I'm missing something obvious.

My code: 

for i in range(len(frame3.columns)):    
    correlation.append(sp.pearsonr(frame3.iloc[ :,i], control['CONTROL']))
Lodore66
  • 1,125
  • 4
  • 16
  • 34
  • Yes, you can use `dropna` for that. What's your question, exactly? – Ami Tavory Aug 11 '16 at 11:05
  • Really? Every time I append it I get an index error. I've added my code above; where's the appropriate place to put it? – Lodore66 Aug 11 '16 at 11:19
  • 2
    *"...that functionality is now missing."* Are you referring to the `nan_policy` argument? That is still in `spearmanr`. In fact, the link that you referred to as "older versions" is the documentation for the most recent release, 0.18.0. What version are you using? Check by running `import scipy; print(scipy.__version__)` – Warren Weckesser Aug 11 '16 at 17:43
  • @WarrenWeckesser I think he might have confused spearman's with pearson's. There is no `nan_policy` for scipy.stats.pearsonr – rovyko Oct 18 '18 at 19:31

2 Answers2

20

You can use np.isnan like this:

for i in range(len(frame3.columns)):    
    x, y = frame3.iloc[ :,i].values, control['CONTROL'].values
    nas = np.logical_or(x.isnan(), y.isnan())
    corr = sp.pearsonr(x[~nas], y[~nas])
    correlation.append(corr)
Ami Tavory
  • 74,578
  • 11
  • 141
  • 185
1

You can also try creating temporary dataframe, and used pandas built-in method for computing pearson correlation, or use the .dropna method in the temporary dataframe to drup null values before using sp.pearsonr

for col in frame3.columns:    
     correlation.append(frame3[col].to_frame(name='3').join(control['CONTROL']).corr()['3']['CONTROL'])
Daniel Gibson
  • 924
  • 8
  • 8