Vectorized alternative to iterrows

Question

I am enhancing the speed of my script and saw the following answer: Iterrows Performance Issues. Here, the answer says that it is rarely needed to use iterrows.

In my code, I make use of iterrows since it is very straightforward and intuitive to use, however also very very slow. So I would like to vectorize the pieces of code I use iterrows. Here come two examples, where i cannot manage to find a solution. In both examples the values in the columns are all datetime values that have the following format: %Y-%m-%d %H:%M:%S

for index, row in df.iterrows():
    df.loc[index, 'Time_Between']= row['Time_Begin'] + timedelta(seconds=row['Some_Integer_Seconds_In_A_Column'])
    df.loc[index, 'Time_Required']= row['Time_End'] - timedelta(seconds=SomeIntegerSecondsAsAVariable)
    df.loc[index, 'Tota_Time']= ((row['Time_Begin'] - row['Time_First']).total_seconds())/60


for index, row in df.iterrows():
    if row['Time_Required'] > row['Time_Between']:
        df.loc[index, 'Check']= 0
    else:
        df.loc[index, 'Check']= 1

How can I vectorized this? I tried masking and apply but I cannot get anything working. Most of the time I get: TypeError: Cannot change data-type for object array.Something I dont get with the iterrows...

score 2 · Accepted Answer · answered May 31 '16 at 06:12

I think you can use:

import pandas as pd
import numpy as np

df = pd.DataFrame({'Time_End': {0: pd.Timestamp('2015-11-15 00:00:00'), 1: pd.Timestamp('2015-10-18 00:00:00'), 2: pd.Timestamp('2015-10-17 00:00:00'), 3: pd.Timestamp('2015-10-16 00:00:00')}, 'Int_Sec': {0: 4, 1: 2, 2: 7, 3: 10}, 'Time_First': {0: pd.Timestamp('2015-10-15 00:00:00'), 1: pd.Timestamp('2015-10-15 00:00:00'), 2: pd.Timestamp('2015-12-15 00:00:00'), 3: pd.Timestamp('2015-12-15 00:00:00')}, 'Time_Begin': {0: pd.Timestamp('2015-10-15 10:00:00'), 1: pd.Timestamp('2015-10-15 12:00:00'), 2: pd.Timestamp('2015-12-15 10:00:00'), 3: pd.Timestamp('2015-12-15 10:00:00')}})
print (df)
   Int_Sec          Time_Begin   Time_End Time_First
0        4 2015-10-15 10:00:00 2015-11-15 2015-10-15
1        2 2015-10-15 12:00:00 2015-10-18 2015-10-15
2        7 2015-12-15 10:00:00 2015-10-17 2015-12-15
3       10 2015-12-15 10:00:00 2015-10-16 2015-12-15

Sec_Var = 20
df['Time_Between'] = df['Time_Begin'] + pd.to_timedelta(df['Int_Sec'], unit='s')
df['Time_Required'] = df['Time_End'] - pd.to_timedelta(Sec_Var, unit='s')
df['Tota_Time'] = ((df['Time_Begin'] - df['Time_First']).dt.total_seconds()) / 60

df['Check'] = np.where(df['Time_Required'] > df['Time_Between'], 0, 1)

print (df)

   Int_Sec          Time_Begin   Time_End Time_First        Time_Between  \
0        4 2015-10-15 10:00:00 2015-11-15 2015-10-15 2015-10-15 10:00:04   
1        2 2015-10-15 12:00:00 2015-10-18 2015-10-15 2015-10-15 12:00:02   
2        7 2015-12-15 10:00:00 2015-10-17 2015-12-15 2015-12-15 10:00:07   
3       10 2015-12-15 10:00:00 2015-10-16 2015-12-15 2015-12-15 10:00:10   

        Time_Required  Tota_Time  Check  
0 2015-11-14 23:59:40      600.0      0  
1 2015-10-17 23:59:40      720.0      0  
2 2015-10-16 23:59:40      600.0      1  
3 2015-10-15 23:59:40      600.0      1

many thanks! i didn't know you could use np.where and pd.timedelta this way. This saves me 10 minutes run time again:) — Uis234, May 31 '16 at 08:48

Merlin · Answer 2 · 2016-05-31T00:54:22.053

For second set:

for index, row in df.iterrows():
    if row['Time_Required'] > row['Time_Between']:
        df.loc[index, 'Check']= 0
    else:
       df.loc[index, 'Check']= 1

np.where(row['Time_Required'] > row['Time_Between'], df['Check']= 0, df['Check']= 1)

For first set:

df['Time_Between']  = row['Time_Begin'] + timedelta(seconds=row['Some_Integer_Seconds_In_A_Column'])
df['Time_Required'] = row['Time_End'] - timedelta(seconds=SomeIntegerSecondsAsAVariable)
df['Tota_Time']     = ((row['Time_Begin'] - row['Time_First']).total_seconds())/60

Vectorized alternative to iterrows

2 Answers2

Linked