50

I have data:

                             Symbol      bid      ask
Timestamp                                            
2014-01-01 21:55:34.378000  EUR/USD  1.37622  1.37693
2014-01-01 21:55:40.410000  EUR/USD  1.37624  1.37698
2014-01-01 21:55:47.210000  EUR/USD  1.37619  1.37696
2014-01-01 21:55:57.963000  EUR/USD  1.37616  1.37696
2014-01-01 21:56:03.117000  EUR/USD  1.37616  1.37694

The timestamp is in GMT. Is there a way to convert that to Eastern?

Note when I do:

data.index

I get output:

<class 'pandas.tseries.index.DatetimeIndex'>
[2014-01-01 21:55:34.378000, ..., 2014-01-01 21:56:03.117000]
Length: 5, Freq: None, Timezone: None
FObersteiner
  • 22,500
  • 8
  • 42
  • 72
user1234440
  • 22,521
  • 18
  • 61
  • 103

3 Answers3

55

Localize the index (using tz_localize) to UTC (to make the Timestamps timezone-aware) and then convert to Eastern (using tz_convert): 

import pytz
eastern = pytz.timezone('US/Eastern')
df.index = df.index.tz_localize(pytz.utc).tz_convert(eastern)

For example:

import pandas as pd
import pytz

index = pd.date_range('20140101 21:55', freq='15S', periods=5)
df = pd.DataFrame(1, index=index, columns=['X'])
print(df)
#                      X
# 2014-01-01 21:55:00  1
# 2014-01-01 21:55:15  1
# 2014-01-01 21:55:30  1
# 2014-01-01 21:55:45  1
# 2014-01-01 21:56:00  1

# [5 rows x 1 columns]
print(df.index)
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 21:55:00, ..., 2014-01-01 21:56:00]
# Length: 5, Freq: 15S, Timezone: None

eastern = pytz.timezone('US/Eastern')
df.index = df.index.tz_localize(pytz.utc).tz_convert(eastern)
print(df)
#                            X
# 2014-01-01 16:55:00-05:00  1
# 2014-01-01 16:55:15-05:00  1
# 2014-01-01 16:55:30-05:00  1
# 2014-01-01 16:55:45-05:00  1
# 2014-01-01 16:56:00-05:00  1

# [5 rows x 1 columns]

print(df.index)
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2014-01-01 16:55:00-05:00, ..., 2014-01-01 16:56:00-05:00]
# Length: 5, Freq: 15S, Timezone: US/Eastern
A Merii
  • 574
  • 9
  • 21
unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
  • 3
    My datetimes are in the column called "date", so I'm doing `df.date = df['date'].pytz_localize(pytz.utc).pytz_convert(pytz.timezone('Europe/Amsterdam'))`, but I get `TypeError: index is not a valid DatetimeIndex or PeriodIndex`. Why does the error mention "index" when I do not? Does this only work if the datetime is in the df index? – AstroFloyd Jan 31 '21 at 19:53
  • 4
    @AstroFloyd --- see the thread below for your answer (I had same problem): This also works similarly for datetime columns, but you need dt after accessing the column: df['column'] = df['column'].dt.tz_convert('America/New_York') – lb_so Feb 01 '21 at 10:54
  • If `pandas` is to drop pytz at some point in the future, better omit that part in the code *now*. Just e.g. `df.index.tz_localize("UTC").tz_convert("America/New_York")` – FObersteiner May 23 '22 at 06:55
48

The simplest way is to use to_datetime with utc=True:

df = pd.DataFrame({'Symbol': ['EUR/USD'] * 5,
                  'bid': [1.37622, 1.37624, 1.37619, 1.37616, 1.37616],
                  'ask': [1.37693, 1.37698, 1.37696, 1.37696, 1.37694]})

df.index = pd.to_datetime(['2014-01-01 21:55:34.378000',
                          '2014-01-01 21:55:40.410000',
                          '2014-01-01 21:55:47.210000',
                          '2014-01-01 21:55:57.963000',
                          '2014-01-01 21:56:03.117000'],
                           utc=True)

For more flexibility, you can convert timezones with tz_convert(). If your data column/index is not timezone-aware, you will get a warning, and should first make the data timezone-aware with tz_localize.

df = pd.DataFrame({'Symbol': ['EUR/USD'] * 5,
                  'bid': [1.37622, 1.37624, 1.37619, 1.37616, 1.37616],
                  'ask': [1.37693, 1.37698, 1.37696, 1.37696, 1.37694]})

df.index = pd.to_datetime(['2014-01-01 21:55:34.378000',
                          '2014-01-01 21:55:40.410000',
                          '2014-01-01 21:55:47.210000',
                          '2014-01-01 21:55:57.963000',
                          '2014-01-01 21:56:03.117000'])

df.index = df.index.tz_localize('GMT')
df.index = df.index.tz_convert('America/New_York')

This also works similarly for datetime columns, but you need dt after accessing the column:

df['column'] = df['column'].dt.tz_convert('America/New_York')
wordsforthewise
  • 13,746
  • 5
  • 87
  • 117
  • 18
    and if your column is not an index, you can do: `df['time_local'] = df['time'].dt.tz_localize('GMT').dt.tz_convert('America/New_York')` – Ufos Oct 09 '18 at 16:43
  • For me simply writing `...dt.tz_convert('Asia/Singapore')` didn't work, I got strange times. I had to do `...dt.tz_convert(pytz.timezone('Asia/Singapore'))` – Saaru Lindestøkke Aug 07 '20 at 15:45
  • 3
    no need to localize to `GMT`; just parse the timestamps with kwarg `utc=True` ([docs](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html#pandas.to_datetime)). – FObersteiner Dec 30 '20 at 14:47
  • 1
    Does this procedure (tc_localize & tc_convert) Also take into account the jungle that is DST? – Psychotechnopath Mar 03 '21 at 10:37
  • Pretty sure yes but someone should confirm – wordsforthewise Mar 03 '21 at 19:20
  • 1
    In response to the comment-question by @Psychotechnopath, I just tested it out for 2020 and 2021 and it correctly captured the DST changes for EST. – BioData41 Oct 01 '21 at 19:25
  • @Psychotechnopath It depends on the timezone used. If you use GMT, no as GMT is a fixed timezone, but if you use one of the location based labels then it s figured out as appropriate, eg 'Europe/London' includes daylight savings for all years after 1911 (or whenever British Summer Time was introduced) – ciaran haines Aug 30 '23 at 08:19
5

To convert EST time into Asia tz

df.index = data.index.tz_localize('EST')
df.index = data.index.tz_convert('Asia/Kolkata')

Pandas has now inbuilt tz conversion ability.

Johannes Kuhn
  • 14,778
  • 4
  • 49
  • 73
Shivendra Rajawat
  • 131
  • 2
  • 4