Speed up Pandas - speed up changing unix timestamp to date in whole column

Question

I have a DataFrame with a column which has a time represented as a unix timestamp. I would like to parse the whole column to represent time as datetime (year-month-day-hours-minute-second). I did it using the foor loop but it takes a long time for large DataFrame. Can I speed this up or maybe is there some built-in Pandas function which I can use instead?

for idx, row in df.iterrows():
            df.loc[idx, "time"] = pd.to_datetime(row['time'], unit='s')

Just don't loop, do it with `df['time'] = pd.to_datetime(df['time'], unit='s')` — Chris Adams, Apr 09 '19 at 16:20
Possible duplicate of [Convert DataFrame column type from string to datetime](https://stackoverflow.com/questions/17134716/convert-dataframe-column-type-from-string-to-datetime) — w-m, Apr 09 '19 at 16:21

score 2 · Answer 1 · edited Apr 09 '19 at 16:23

2

You can do this by accessing the column directly and specifying the format:

df['time'] = pd.to_datetime(df['time'], format='%Y-%m-%d-%H-%M-%S')

edited Apr 09 '19 at 16:23

ALollz

57,915
7
66
89

answered Apr 09 '19 at 16:22

d_kennetz

5,219
5
21
44

score 1 · Answer 2 · answered Apr 09 '19 at 17:09

d_kennetz's answer is the right one, but it is also useful to know that df.at() should perform a lot better for you than df.loc() in this case. I'd be willing to bet that df.at() would be at least one thousand times faster than df.loc():

for idx, row in df.iterrows():
            df.at[idx, "time"] = pd.to_datetime(row['time'], unit='s')

This isn't the solution I'd recommend in this case, but this information may be useful to you or someone else later on.

Speed up Pandas - speed up changing unix timestamp to date in whole column

2 Answers2