Pandas, set on a copy of a slice from DataFrame problem

Question

I've been struggling with a Pandas warning for a while now.

I have this pretty simple piece of code:

def get_delta(df): 
    df['delta_current_day'] = df['close'] - df['open']
    df = df[pd.notnull(df['delta_current_day'])]

    df['delta_next_day'] = df['delta_current_day'].shift(-1)

    return df

Every time I get this error:

A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

I've read a lot of question about it here on stackoverflow but none works for me. I've tried somethings like this:

 df.loc[:, 'delta_next_day'] = df['delta_current_day'].shift(-1)

# OR 
list = df['delta_current_day'].tolist()
df.loc[:, 'delta_next_day'] = list[:1]

but I still get the same error.

What am I doing wrong?

I'm afraid that by upgrading pandas I'll break more +1000 lines of code working — Steve, Feb 05 '20 at 17:06
Are you sure? 0.24 to 0.25 had more method not many changes, you can always suppress the warning with `warning` — Kenan, Feb 05 '20 at 17:07
Which line exactly is causing the issue, the first (`df['delta_current_day'] = ...`) or the last (`df['delta_next_day'] = ...`? Try commenting out the last line two lines to see if you still get the error from the first assignment. — Alexander, Feb 05 '20 at 17:09
The error is on ```df['delta_next_day'] = df['delta_current_day'].shift(-1)``` — Steve, Feb 05 '20 at 17:10

ansev · Accepted Answer · 2020-02-05T17:28:03.837

1

I think you need DataFrame.copy:

df = df[pd.notnull(df['delta_current_day'])].copy()
#df = df.loc[pd.notnull(df['delta_current_day'])] #or this
df['delta_next_day'] = df['delta_current_day'].shift(-1)

this is a slice:

df = df[pd.notnull(df['delta_current_day'])]

A (df['delta_current_day'].shift(-1)) value is trying to be set on a (df[pd.notnull(df['delta_current_day'])]) copy of a slice from a DataFrame

edited Feb 05 '20 at 17:28

answered Feb 05 '20 at 17:09

ansev

30,322
5
17
31

The error is on ```df['delta_next_day'] = df['delta_current_day'].shift(-1)``` – Steve Feb 05 '20 at 17:10
have you tryed it? – ansev Feb 05 '20 at 17:11
If you could give me the explanation in your answer, I'd really appreciate it. – Steve Feb 05 '20 at 17:16
you have to create a new object, `df[pd.notnull(df['delta_current_day'])]` is not – ansev Feb 05 '20 at 17:17

score 1 · Answer 2 · answered Feb 05 '20 at 17:22

You could also try this:

# Sample data.
df = pd.DataFrame({
    'open': [100, 102, np.nan, 102],
    'close': [101.5, 102.5, np.nan, 104]
})

def get_delta(df):
    df = df.dropna().assign(delta_current_day=df.eval('close - open'))
    return df.assign(delta_next_day=df['delta_current_day'].shift(-1))

>>> get_delta(df)
    open  close  delta_current_day  delta_next_day
0  100.0  101.5                1.5             0.5
1  102.0  102.5                0.5             2.0
3  102.0  104.0                2.0             NaN

Of course, this introduces look ahead bias. Why are you using tomorrow's delta today?

Pandas, set on a copy of a slice from DataFrame problem

2 Answers2