Pandas Dataframe - Droping Certain Hours of the Day from 20 Years of Historical Data

Question

I have stock market data for a single security going back 20 years. The data is currently in an Pandas DataFrame, in the following format:

The problem is, I do not want any "after hours" trading data in my DataFrame. The market in question is open from 9:30AM to 4PM (09:30 to 16:00 on each trading day). I would like to drop all rows of data that are not within this time frame.

My instinct is to use a Pandas mask, which I know how to do if I wanted certain hours in a single day:

mask = (df['date'] > '2015-07-06 09:30:0') & (df['date'] <= '2015-07-06 16:00:0')
sub = df.loc[mask]

However, I have no idea how to use one on a revolving basis to remove the data for certain times of day over a 20 year period.

what is the datatype of column `date`. Could you run this command `print(df['date'].map(type))` and post its output to the question? — Andy L., Mar 30 '20 at 00:12

Bhavesh Ghodasara · Accepted Answer · 2020-03-30T10:09:12.403

9

Problem here is how you are importing data. There is no indicator whether 04:00 is am or pm? but based on your comments we need to assume it is PM. However input is showing it as AM.

To solve this we need to include two conditions with OR clause.

9:30-11:59
0:00-4:00

Input:

df = pd.DataFrame({'date':   {880551: '2015-07-06 04:00:00', 880552: '2015-07-06 04:02:00',880553: '2015-07-06 04:03:00', 880554: '2015-07-06 04:04:00', 880555: '2015-07-06 04:05:00'},
                   'open':   {880551: 125.00, 880552: 125.36,880553: 125.34, 880554: 125.08, 880555: 125.12},
                   'high':   {880551: 125.00, 880552: 125.36,880553: 125.34, 880554: 125.11, 880555: 125.12},
                   'low':    {880551: 125.00, 880552: 125.32,880553: 125.21, 880554: 125.05, 880555: 125.12},
                   'close':  {880551: 125.00, 880552: 125.32,880553: 125.21, 880554: 125.05, 880555: 125.12},
                   'volume': {880551: 141, 880552: 200,880553: 750, 880554: 17451, 880555: 1000},
                   },
                   )


df.head()

    date    open    high    low close   volume
880551  2015-07-06 04:00:00 125.00  125.00  125.00  125.00  141
880552  2015-07-06 04:02:00 125.36  125.36  125.32  125.32  200
880553  2015-07-06 04:03:00 125.34  125.34  125.21  125.21  750
880554  2015-07-06 04:04:00 125.08  125.11  125.05  125.05  17451
880555  2015-07-06 04:05:00 125.12  125.12  125.12  125.12  1000

from datetime import time

start_first = time(9, 30)
end_first = time(11, 59)
start_second = time(0, 00)
end_second = time(4,00)
df['date'] = pd.to_datetime(df['date'])
df= df[(df['date'].dt.time.between(start_first, end_first)) | (df['date'].dt.time.between(start_second, end_second))]
df
date    open    high    low close   volume
880551  2015-07-06 04:00:00 125.0   125.0   125.0   125.0   141

Above is not good practice, and I strongly discourage to use this kind of ambiguous data. long time solution is to correctly populate data with am/pm.

We can achieve it in two way in case of correct data format:

1) using datetime

from datetime import time

start = time(9, 30)
end = time(16)
df['date'] = pd.to_datetime(df['date'])
df= df[df['date'].dt.time.between(start, end)]

2) using between time, which only works with datetime index

df['date'] = pd.to_datetime(df['date'])

df = (df.set_index('date')
          .between_time('09:30', '16:00')
          .reset_index())

If you still face error, edit your question with line by line approach and exact error.

edited Mar 30 '20 at 10:09

answered Mar 27 '20 at 23:03

Bhavesh Ghodasara

1,981
2
15
29

That results in the following error ```TypeError: Index must be DatetimeIndex``` – HMLDude Mar 28 '20 at 06:37
edited my answer, df['date'] = pd.to_datetime(df['date']) – Bhavesh Ghodasara Mar 28 '20 at 15:11
From this [SO post](https://stackoverflow.com/q/24576786/1422451), looks like `between_time` requires data frame to be datetime index. OP can try at DataFrame level: `day_df = df.set_index('date').between_time('9:30', '16:00')`. – Parfait Mar 28 '20 at 15:32
BhaveshGhodasara I tried what you suggested in your latest edits and the result is the same ```TypeError: Index must be DatetimeIndex```. – HMLDude Mar 28 '20 at 21:09
@Parfait I tried your suggestion as well and once again the error message was: ```TypeError: Index must be DatetimeIndex```. – HMLDude Mar 28 '20 at 21:11
@HMLDude - I have again edited my answer. can you please check now? – Bhavesh Ghodasara Mar 28 '20 at 22:17
@BhaveshGhodasara, your most recent edit results in the following: ```df = df[df['date'].dt.time.between(start, end)] ^ SyntaxError: invalid syntax``` – HMLDude Mar 29 '20 at 23:39
Is your date in mixed format i.e has am/pm and in other places 24hour? – wwnde Mar 30 '20 at 00:28
@wwnde The ```date``` column of my DataFrame is in the format listed in my question, i.e. ```2000-01-03 09:31:00```. – HMLDude Mar 30 '20 at 01:40
@HMLDude i have not got any error. I created data frame in your format and showing it with an example. Problem here is input format is ambiguous. – Bhavesh Ghodasara Mar 30 '20 at 10:10
Hi @HMLDude, I am facing the same error actually (working with financial time series, but less data). I tried Bhavesh's solution and it worked for me...I added an extra record to his example, namely 880556: '2015-07-06 10:05:00' as a test and both solutions worked for me. If you are receiving a syntax error, are you missing a bracket or parenthesis perhaps? Hope this helps. – cmp Apr 02 '20 at 12:33

score 5 · Answer 2 · answered Apr 02 '20 at 07:55

I think the answer is already in the comments (@Parfait's .between_time) but that it got lost in debugging issues. It appears your df['date'] column is not of type Datetime yet.

This should be enough to fix that and get the required result:

df['date'] = pd.to_datetime(df['date'])
df = df.set_index('date')
df = df.between_time('9:30', '16:00')

score 1 · Answer 3 · answered Apr 05 '20 at 19:51

This example code consolidates the answers provided by Bhavesh Ghodasara, Parfait and jorijnsmit into one complete, commented example:

import pandas as pd

# example dataframe containing 6 records: 2 days of 3 records each in which all cases are covered:
# each day has one record before trading hours, one record during trading hours and one recrod after trading hours
df = pd.DataFrame({'date':   {0: '2015-07-06 08:00:00', 1: '2015-07-06 13:00:00', 2: '2015-07-06 18:00:00', 
                              3: '2015-07-07 08:00:00', 4: '2015-07-07 13:00:00', 5: '2015-07-07 18:00:00'},
                   'open':   {0: 125.00, 1: 125.36, 2: 125.34, 3: 125.08, 4: 125.12, 5: 125.37},
                   'high':   {0: 125.00, 1: 125.36, 2: 125.34, 3: 125.08, 4: 125.12, 5: 125.37},
                   'low':    {0: 125.00, 1: 125.36, 2: 125.34, 3: 125.08, 4: 125.12, 5: 125.37},
                   'close':  {0: 125.00, 1: 125.36, 2: 125.34, 3: 125.08, 4: 125.12, 5: 125.37},
                   'volume': {0: 141, 1: 200, 2: 750, 3: 17451, 4: 1000, 5: 38234},
                   },
                   )

# inspect the example data set
df.head(6)

# first, ensure that the 'date' column is of the correct data type: MAKE IT SO!
df['date'] = pd.to_datetime(df['date'])

# inspect the data types: date column should be of type 'datetime64[ns]'
print(df.dtypes)

# set the index of the dataframe to the datetime-type column 'data'
df = df.set_index('date')

# inspect the index: it should be a DatetimeIndex of dtype 'datetime64[ns]'
print(df.index)

# filter the data set
df_filtered = df.between_time('9:30', '16:00')

# inspect the filtered data set: Voilà! No more outside trading hours records.
df_filtered.head()

Crispy Holiday · Answer 4 · 2022-11-23T13:06:24.870

All of the previous answers are ignoring one important fact - Daylight saving.

Assuming your data is in UTC time zone, the opening and closing hours NYSE are different depending on DST.

Just filtering your data with df.between_time("09:30","16:30") is wrong. You should be aware of the NYSE's schedule on any given day.

Fortunately, The pip package pandas_market_calendars is making this much easier to handle.

import pandas_market_calendars as mcal

nyse = mcal.get_calendar('NYSE')
nyse.schedule(start_date='2022-03-10', end_date='2022-03-20')

This will result in

2022-03-10  2022-03-10 14:30:00+00:00   2022-03-10 21:00:00+00:00
2022-03-11  2022-03-11 14:30:00+00:00   2022-03-11 21:00:00+00:00
2022-03-14  2022-03-14 13:30:00+00:00   2022-03-14 20:00:00+00:00
2022-03-15  2022-03-15 13:30:00+00:00   2022-03-15 20:00:00+00:00
2022-03-16  2022-03-16 13:30:00+00:00   2022-03-16 20:00:00+00:00
2022-03-17  2022-03-17 13:30:00+00:00   2022-03-17 20:00:00+00:00
2022-03-18  2022-03-18 13:30:00+00:00   2022-03-18 20:00:00+00:00

You can use this output to create one index that contains all minutes between market_open and market_close of each day.

Note: This piece of code for sure can be done better, but it still runs pretty fast.

hours = []
for i, row in nyse_scehdule.iterrows():
    hours.append(pd.date_range(start=row['market_open'], end=row['market_close'], tz="UTC", freq="1min").to_series())
hours_index = pd.concat(hours).index

Now you can just reindex your original dataframe by this new index:

data.reindex(hours_index)

Hope this helps.

Pandas Dataframe - Droping Certain Hours of the Day from 20 Years of Historical Data

4 Answers4