Finding time difference between two columns in DataFrame

Question

I am trying to find the time difference between two columns of the following frame:

Test Date | Test Type | First Use Date

---

I used the following function definition to get the difference:

def days_between(d1, d2):
    d1 = datetime.strptime(d1, "%Y-%m-%d")
    d2 = datetime.strptime(d2, "%Y-%m-%d")
    return abs((d2 - d1).days)

And it works fine, however it does not take a series as an input. So I had to construct a for loop that loops over indices:

age_veh = []
for i in range(0, len(data_manufacturer)-1):
    age_veh[i].append(days_between(data_manufacturer.iloc[i,0], data_manufacturer.iloc[i,4]))

However, it does return an error: IndexError: list index out of range

I don't know whether it's the right way of doing and what am I doing wrong or an alternative solution will be much appreciated. Please also bear in mind that I have around 2 mil rows.

Answer 1

Convert the columns using to_datetime then you can subtract the columns to produce a timedelta on the abs values, then you can call dt.days to get the total number of days, example:

In [119]:
import io
import pandas as pd
t="""Test Date,Test Type,First Use Date
2011-02-05,A,2010-01-05
2012-02-05,A,2010-03-05
2013-02-05,A,2010-06-05
2014-02-05,A,2010-08-05"""
df = pd.read_csv(io.StringIO(t))
df
Out[119]:
    Test Date Test Type First Use Date
0  2011-02-05         A     2010-01-05
1  2012-02-05         A     2010-03-05
2  2013-02-05         A     2010-06-05
3  2014-02-05         A     2010-08-05

In [121]:    
df['Test Date'] = pd.to_datetime(df['Test Date'])
df['First Use Date'] = pd.to_datetime(df['First Use Date'])
df.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 4 entries, 0 to 3
Data columns (total 3 columns):
Test Date         4 non-null datetime64[ns]
Test Type         4 non-null object
First Use Date    4 non-null datetime64[ns]
dtypes: datetime64[ns](2), object(1)
memory usage: 128.0+ bytes

In [122]:
df['days'] = (df['Test Date'] - df['First Use Date']).abs().dt.days
df

Out[122]:
   Test Date Test Type First Use Date  days
0 2011-02-05         A     2010-01-05   396
1 2012-02-05         A     2010-03-05   702
2 2013-02-05         A     2010-06-05   976
3 2014-02-05         A     2010-08-05  1280

Answer 2

IIUC you can first convert columns to_datetime, use abs and then convert timedelta to days:

print df
  id  value      date1      date2  sum
0  A    150 2014-04-08 2014-03-08  NaN
1  B    100 2014-05-08 2014-02-08  NaN
2  B    200 2014-01-08 2014-07-08  100
3  A    200 2014-04-08 2014-03-08  NaN
4  A    300 2014-06-08 2014-04-08  350

df['date1'] =  pd.to_datetime(df['date1'])
df['date2'] =  pd.to_datetime(df['date2'])

df['diff'] = (df['date1'] - df['date2']).abs()  / np.timedelta64(1, 'D')
print df
  id  value      date1      date2  sum  diff
0  A    150 2014-04-08 2014-03-08  NaN    31
1  B    100 2014-05-08 2014-02-08  NaN    89
2  B    200 2014-01-08 2014-07-08  100   181
3  A    200 2014-04-08 2014-03-08  NaN    31
4  A    300 2014-06-08 2014-04-08  350    61

EDIT:

I think better is use for converting np.timedelta64(1, 'D') to days in larger DataFrames, because it is faster:

I use EdChum sample, only len(df) = 4k:

import io
import pandas as pd
import numpy as np

t=u"""Test Date,Test Type,First Use Date
2011-02-05,A,2010-01-05
2012-02-05,A,2010-03-05
2013-02-05,A,2010-06-05
2014-02-05,A,2010-08-05"""

df = pd.read_csv(io.StringIO(t))

df =  pd.concat([df]*1000).reset_index(drop=True)

df['Test Date'] = pd.to_datetime(df['Test Date'])
df['First Use Date'] = pd.to_datetime(df['First Use Date'])

print (df['Test Date'] - df['First Use Date']).abs().dt.days

print (df['Test Date'] - df['First Use Date']).abs()  / np.timedelta64(1, 'D')

Timings:

In [174]: %timeit (df['Test Date'] - df['First Use Date']).abs().dt.days
10 loops, best of 3: 38.8 ms per loop

In [175]: %timeit (df['Test Date'] - df['First Use Date']).abs()  / np.timedelta64(1, 'D')
1000 loops, best of 3: 1.62 ms per loop