-1

I need to be able to iterate over rows and perform some manipulations. Below is the start of some code.

for i, row in df.iterrows():
    df.loc[i, 'Object'] = row

The performance is ridiculously slow and I get the following output:

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

What am I doing wrong?

nyedidikeke
  • 6,899
  • 7
  • 44
  • 59
RMichalowski
  • 330
  • 1
  • 4
  • 11
  • 1
    Why not just do this? `vector = [row for i, row in df.iterrows()]; df['Object'] = vector`? Performance-wise, you might see some improvements, but you'll also avoid doing exactly what that warning suggests you shouldn't. – blacksite May 25 '17 at 12:51
  • 1
    It is actually not an error, it is a warning and it has nothing to do with your performance issues. See this [question](https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas) for more information on this warning. – Jan Trienes May 25 '17 at 12:52
  • A lot of the times you don't need to iterate through the rows of a dataframe, if you can vectorize your function/operation, you will see great performance improvements. – Scott Boston May 25 '17 at 13:07

2 Answers2

0

If you iterate through the whole data frame and try using loc on each row it is going to be slow. Instead try this:

df.ix[np.in1d(df['Object'], value)
Amit Bhattarai
  • 1
  • 2
0

If you really find that you need to loop, use itertuples, in most cases it's many multiples faster.

df = pd.DataFrame({"Object": [1,2,3], "Lifeishard_butunfair": [2,3,4]})

objectCol = df.columns.get_loc("Object")+1
for row in df.itertuples():
    someVar = row[objectCol]
misantroop
  • 2,276
  • 1
  • 16
  • 24
  • Have you found any documentation that explains when to use iterrows vs itertuples? I've seen posts that state that the itertuples is faster yet most of the tutorials I've found are based on iterrows. Thank you for your advice! – RMichalowski May 30 '17 at 13:08
  • More information here: https://stackoverflow.com/questions/24870953/does-iterrows-have-performance-issues – misantroop May 31 '17 at 04:27