0

Following advice I received on StackOverflow, I thought I had solved my `CopyWithWarning problems when using Pandas dataframes, but they are back to bite me. I'm trying to create a new column in a dataframe that is the first difference of another column:

df[upper_sys_hrt] = df[idx_upper] - df[idx_upper].shift(1)
df[upper_sys_hrt].iloc[0] = d[upper_sys_hrt].iloc[1]

This sometimes (not always) gives me a CopyWithWarning error. I tried changing this to

df[ : , df.get_loc(upper_sys_hrt)] =  df[ : , df.columns.get_loc(idx_upper)] - df[: , df.columns.get_loc(idx_upper)].shift(1)

but all i get is an error message:

TypeError: '(slice(1, None, None), 3)' is an invalid key

I have tried (hunt and peck) changes to the syntax as well as Googling and gotten nowhere. What am I doing wrong?

cs95
  • 379,657
  • 97
  • 704
  • 746
Thomas Philips
  • 935
  • 2
  • 11
  • 22

1 Answers1

1

You have the column's position, not the label. To get the label, index back into df.columns:

col = df.columns[df.get_loc(upper_sys_hrt)]
df[col] = df[col] - df[col].shift()

Or just 

df[col] = df[col].diff()

If you're rather work with the index, you can use iloc:

idx = df.get_loc(upper_sys_hrt)
df.iloc[:,idx] = df.iloc[:,idx].diff()
cs95
  • 379,657
  • 97
  • 704
  • 746
  • Thanks for this. To tell you the truth, it's doubly mysterious because df[col] appears functionally identical to df[upper_sys_hrt], which is what i started with, but it's clear that your way does not induce the confusing CopyWithWarnings error, while mine does.. Is there a difference under the hood that's not obvious on the outside? – Thomas Philips Jun 10 '20 at 08:58
  • @ThomasPhilips Can't say as I don't have all your code. But as a rule of thumb prefer `(i)loc` and `(i)at` are superior methods when it comes to assignment, for more info please take a look at my writeup here: [How to deal with SettingWithCopyWarning in Pandas?](https://stackoverflow.com/a/53954986/4909087). – cs95 Jun 10 '20 at 09:07