In Pandas how do I convert a string of date strings to datetime objects and put them in a DataFrame?

Question

import pandas as pd
date_stngs = ('2008-12-20','2008-12-21','2008-12-22','2008-12-23')

a = pd.Series(range(4),index = (range(4)))

for idx, date in enumerate(date_stngs):
    a[idx]= pd.to_datetime(date)

This code bit produces error:

TypeError:" 'int' object is not iterable"

Can anyone tell me how to get this series of date time strings into a DataFrame as DateTime objects?

For those coming to this answer in 2017+, see the [idiomatic solution below](https://stackoverflow.com/a/47142267/3707607) — Ted Petrou, Nov 06 '17 at 17:19

falsetru · Accepted Answer · 2013-07-17T07:33:06.390

55

>>> import pandas as pd
>>> date_stngs = ('2008-12-20','2008-12-21','2008-12-22','2008-12-23')
>>> a = pd.Series([pd.to_datetime(date) for date in date_stngs])
>>> a
0    2008-12-20 00:00:00
1    2008-12-21 00:00:00
2    2008-12-22 00:00:00
3    2008-12-23 00:00:00

UPDATE

Use pandas.to_datetime(pd.Series(..)). It's concise and much faster than above code.

>>> pd.to_datetime(pd.Series(date_stngs))
0   2008-12-20 00:00:00
1   2008-12-21 00:00:00
2   2008-12-22 00:00:00
3   2008-12-23 00:00:00

edited Jul 17 '13 at 07:33

answered Jul 17 '13 at 03:53

falsetru

357,413
63
732
636

12

Hundred times slower than `pd.to_datetime(pd.Series(date_stngs))`. Strongly not recommend this method. – waitingkuo Jul 17 '13 at 05:45
@DickEshelman, waitingkuo's version is more elegant solution. – falsetru Jul 17 '13 at 06:07
@waitingkuo, I agree that your version is more concise. But your version is not 100x faster than than mine. `timeit.timeit` mine, yours, shows very little difference. – falsetru Jul 17 '13 at 06:14
falsetru, you might try to create a larger date_stngs and give it a try. – waitingkuo Jul 17 '13 at 06:19
@waitingkuo, I did `timeit.timeit` with a tuple with 5000 strings (100 repeats). How many items should I use to produce 100x times difference? – falsetru Jul 17 '13 at 06:21
By your script I still get the same result. What's your pandas version? – waitingkuo Jul 17 '13 at 06:38
@waitingkuo, I use pandas 0.8.0. – falsetru Jul 17 '13 at 06:39
it's better to upgrade to 0.11.0 first. – waitingkuo Jul 17 '13 at 06:42
4

@waitingkuo, I upgraded to 0.11.0. You were right. Shame on me. – falsetru Jul 17 '13 at 06:48
In addition, if you are able specify the format, e.g. pd.to_datetime(pd.Series,format='%Y-%m-%d'), it speeds up by a factor of 10 for ~100k rows – Fermion Portal Dec 18 '14 at 14:15
1

@FermionPortal, It's almost same for me. (Python 2.7.8, pandas 0.14.1, Windows 7): http://pastebin.com/m1u9Prn6 – falsetru Dec 18 '14 at 14:40

waitingkuo · Answer 2 · 2013-07-17T06:25:44.623

38

In [46]: pd.to_datetime(pd.Series(date_stngs))
Out[46]: 
0   2008-12-20 00:00:00
1   2008-12-21 00:00:00
2   2008-12-22 00:00:00
3   2008-12-23 00:00:00
dtype: datetime64[ns]

Update: benchmark

In [43]: dates = [(dt.datetime(1960, 1, 1)+dt.timedelta(days=i)).date().isoformat() for i in range(20000)]

In [44]: timeit pd.Series([pd.to_datetime(date) for date in dates])
1 loops, best of 3: 1.71 s per loop

In [45]: timeit pd.to_datetime(pd.Series(dates))
100 loops, best of 3: 5.71 ms per loop

edited Jul 17 '13 at 06:25

answered Jul 17 '13 at 04:00

waitingkuo

89,478
28
112
118

Why do you benchmark `datetime -> datetime` conversion, not `str -> datetime`? – falsetru Jul 17 '13 at 06:36
Sorry, I missed that. – falsetru Jul 17 '13 at 06:41

Ted Petrou · Answer 3 · 2017-11-06T17:18:55.607

A simple solution involves the Series constructor. You can simply pass the data type to the dtype parameter. Also, the to_datetime function can take a sequence of strings now.

Create Data

date_strings = ('2008-12-20','2008-12-21','2008-12-22','2008-12-23')

All three produce the same thing

pd.Series(date_strings, dtype='datetime64[ns]')
pd.Series(pd.to_datetime(date_strings))
pd.to_datetime(pd.Series(date_strings))

Benchmarks

The benchmarks provided by @waitingkuo are wrong. The first method is a bit slower than the other two, which have the same performance.

import datetime as dt
dates = [(dt.datetime(1960, 1, 1)+dt.timedelta(days=i)).date().isoformat() 
         for i in range(20000)] * 100

%timeit pd.Series(dates, dtype='datetime64[ns]')
730 ms ± 9.06 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


%timeit pd.Series(pd.to_datetime(dates))
426 ms ± 3.45 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit pd.to_datetime(pd.Series(dates))
430 ms ± 5.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In Pandas how do I convert a string of date strings to datetime objects and put them in a DataFrame?

3 Answers3

Update: benchmark

All three produce the same thing

Benchmarks

Linked

Related