6

Coming from R, I naively tried

dfE_fitted['E_after'] = dfE_fitted['E_before']

That gave me

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

Fair enough, I'll try that then:

dfE_fitted.loc[:,'E_after'] = dfE_fitted['E_before']

This gives me

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py:337: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
self.obj[key] = _infer_fill_value(value)
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py:517: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item] = s

What am I doing horribly wrong?

komodovaran_
  • 1,940
  • 16
  • 44
  • Before: `Dem-Dexc Aem-Dexc Aem-Aexc S E E_before transition_frame filename state` After: `Dem-Dexc Aem-Dexc Aem-Aexc S E E_before transition_frame filename state E_after` – komodovaran_ Sep 21 '17 at 15:26

1 Answers1

8

It isn't how you are doing your assignment. It's how you constructed dfE_fitted. You constructed it in such a way so as to be a "copy" of something else. The recommendation to use .loc is intended to be used in the construction of dfE_fitted. Even then, using .loc isn't a guarantee. You can still have a "is_copy" flag after using .loc

However, I can't see how you did that. Best I can do is recommend that you do

dfE_fitted = dfE_fitted.copy()

This will detangle the "copy" relationship and you can proceed to assign as you were.

dfE_fitted['E_after'] = dfE_fitted['E_before']

Working example 

df = pd.DataFrame(dict(A=[1, 2], E_before=[3, 4], E_after=[5, 6]))

# Notice how I constructed this
dfE_fitted = df[['E_before']]

Then I try

dfE_fitted['E_after'] = dfE_fitted['E_before']

I get

enter image description here

However, had I constructed dfE_fitted like this:

# Notice I used `.loc` like what was recommended
dfE_fitted = df.loc[:, ['E_before']]

I can do the following with no warning:

dfE_fitted['E_after'] = dfE_fitted['E_before']

You can tell if a dataframe is a "copy" by looking at its is_copy attribute. If it is a copy, it will return a 

<weakref at 0x1188d94f8; to 'DataFrame' at 0x1184bf898>

Which evaluates to True

bool(dfE_fitted.is_copy)

True

Otherwise, if it isn't a "copy" it is None and evaluates to False

piRSquared
  • 285,575
  • 57
  • 475
  • 624
  • By "construct", do you mean that perhaps my initial definition in my code `dfE_fitted = df_all` is causing problems? – komodovaran_ Sep 21 '17 at 15:37
  • 1
    If that was indeed how you created the name `dfE_fitted`, I'd do `dfE_fitted = df_all.copy()` – piRSquared Sep 21 '17 at 15:39
  • Pandas tracks to see if your current dataframe is a view or copy of another dataframe. If it is, it's going to warn you when you do something that might alter that relationship. It tracks that relationship with the `is_copy` attribute. Most often this warning is encountered when people didn't expect it. They had no idea there dataframe was a view or copy of another. The easiest way to fix that is to explicitly make the dataframe you are working independent of any other dataframe by setting it to be a copy of itself. – piRSquared Sep 21 '17 at 15:47
  • Deleted my old comment, as I just stumbled upon https://stackoverflow.com/questions/3975376/understanding-dict-copy-shallow-or-deep and now I'm in doubt. Wouldn't `deepcopy` be preferable then? – komodovaran_ Sep 21 '17 at 15:49
  • 1
    [**copy**](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.copy.html) is the documentation. You'll notice there is a `deep` parameter which is set to `True` by default. Also notice that you're linked Q&A is for `dict.copy`. Pandas handles copy itself with `pd.DataFrame.copy` – piRSquared Sep 21 '17 at 15:52
  • 1
    My my my, would you look at that. It all makes sense now. Thanks. – komodovaran_ Sep 21 '17 at 15:53