2

The goal is to create a new column in a pandas column that stores the value of a KS D-statistic, df['ks']. The KS statistic is generated between two groups of columns in that dataframe, grp1 and grp2:

# sample dataframe
import pandas as pd
import numpy as np

dic = {'gene': ['x','y','z','n'],
        'cell_a': [1, 5, 8,9],
        'cell_b': [8, 5, 4,9],
        'cell_c': [8, 6, 1,1],
        'cell_d': [1, 2, 7,1],
        'cell_e': [5, 7, 9,1],
       }
df = pd.DataFrame(dic)
df.set_index('gene', inplace=True)
df['ks'] = np.nan

# sample groups
grp1 = ['cell_a','cell_b']
grp2 = ['cell_d','cell_e']

So the D-statistic for gene x would be stats.ks_2samp([1,5], [1,6])[0], gene y would be stats.ks_2samp([5,2], [1,7])[0], etc. Attempt is below:

# attempt 1 to fill in KS stat
for idx, row in df.iterrows():
    df.ix[idx, 'ks'] = stats.ks_2samp(df[grp1], df[grp2])[0]

However, when I attempt to fill the ks series, I get the following error:

ValueError: object too deep for desired array

My question has two parts: 1) What does it mean for an object to be "too deep for an array", and 2) how can I accomplish the same thing without iteration?

Thomas Matthew
  • 2,826
  • 4
  • 34
  • 58
  • 1
    Change that line to - `df.loc[idx, 'ks'] = stats.ks_2samp(row[grp1], row[grp2])[0]` since you're operating per row basis and not the entire `DF` object per se. Also, it's better to use `.loc` for the assignment purposes as accessing via `.ix` would soon be deprecated in the upcoming versions. – Nickil Maveli Mar 16 '17 at 07:57
  • Still too deep for the array. Still don't know what that means or how to fix it... – Thomas Matthew Mar 16 '17 at 15:39

1 Answers1

1

The KS calculation in the loop was getting a "too deep" error because I needed to pass it a 1-D array for each distribution to test: 

for idx, row in df.iterrows():
    df.loc[idx, 'ks'] = stats.ks_2samp(df.loc[idx, grp1], (df.loc[idx, grp2]))[0]

My previous attempt used a 2-D array instead. That is what was causing it to be "too deep"

Thomas Matthew
  • 2,826
  • 4
  • 34
  • 58