0

I have a function that takes dataframe as an argument and while processing this dataframe it calls another function passing a slice of the same dataframe as an argument to the secondary function.

All changes are done in place so nothing is returned (because of the size the dataframe).

But, this secondary function raises SettingWithCopyWarning since it does not deal with the original dataframe anymore.

Here is an example:

import pandas as pd
import numpy as np
df = pd.DataFrame(np.arange(9).reshape(3,3), columns=list('abc'))
print df

def a(df):
    if df.is_copy:
        print 'a got a copy'
    df['a'] = 'a'

def b(df):
    if df.is_copy:
        print 'b got a copy'
        print df.is_copy
    df.loc[:,'b'] = 'b'

def c(df):
    a(df)
    b(df.loc[0:1,:])
    if df.is_copy:
        print 'c got a copy'
    df.loc[0:1,'c'] = 'c'

def d(df):
    new_df = df.loc[0:1].copy(deep=True)
    b(new_df)
    df.update(new_df)
    del new_df

c(df)
df

Results in:

b got a copy
<weakref at 000000000C1DE778; to 'DataFrame' at 000000000C1B9DA0>

   a  b  c
0  a  1  c
1  a  4  c
2  a  7  8

I understand that one option is to create a new dataframe from the slice of the original and to pass it to b and then df.update(new_df) and d shows that it works:

d(df)
df

Produces the desired output:

   a  b  c
0  a  b  c
1  a  b  c
2  a  7  8

But is there a way to deal with this without creating new dataframe and raising SettingWithCopyWarning.

Another complication is that call to b from within c sometimes might be just simple b(df), so slicing is optional.

Thank you.

Primer
  • 10,092
  • 5
  • 43
  • 55

1 Answers1

0

If you want to modify things it is much better to simply pass the frame and a mask around.

def b(df, row_mask):
    df.loc[row_mask,'b'] = 'foo'

Though usually I wouldn't modify things like this especially if its a big frame. These modifications trigger a copy when you are changing dtypes (e.g. putting 'b' in a column with all numbers is normally not something you should do, dtypes are column based).

So a better workflow is to do this:

def b(df):
    sliced = df.loc[0:1].copy()
    sliced.loc[:,'b'] = 'foo'
    return sliced

Then you can simply concetanate at the end:

result = pd.concat([b(df), df[1:]])

then produce a chain of these and concat all at once. Will be much more efficient then modifying in-place (though if you are only modifying a small number of values then my 1st method might work better). YMMV.

Jeff
  • 125,376
  • 21
  • 220
  • 187
  • Thank you, Jeff. This was very helpful. Actually no dtype change is intended - it was perhaps wrong example to use. However I am a bit confused with your second option. 1) If `df.memory_usage()` shows about 4 GB (out of available 8GB), doesn't `df.loc[0:1].copy()` increase overall memory footprint? 2) If I have to pass a mask to your second `b` (in case the slicing index is defined outside of `b`) do I understand correctly that second `b` needs to be modified according to the first `b` (i.e. mask needs to be passed along with `df`)? Thanks again. – Primer Nov 20 '14 at 14:45
  • it depends on the problem of j have a giant frame usually much better to work on it in pieces – Jeff Nov 20 '14 at 23:02