Converting imported dates from .txt to date format python

Question

I have issues with converting dates in an imported .txt file and I wonder what I'm doing wrong.

I import the data by:

df_TradingMonthlyDates = pd.read_csv(TradingMonthlyDates, dtype=str, sep=',') # header=True,

and it looks like the following table (dates represents start/end of month and have a header Date):

           Date
0    2008-12-30
1    2008-12-31
2    2009-01-01
3    2009-01-02
4    2009-01-29
..          ...

557  2020-06-29
558  2020-06-30
559  2020-07-01
560  2020-07-02
561  2020-07-30
..          ...

624  2021-11-30
625  2021-12-01
626  2021-12-02
627  2021-12-30
628  2021-12-31

[629 rows x 1 columns]
<class 'pandas.core.frame.DataFrame'>

I then calculate today's date:

df_EndDate = datetime.now().date()

I'm trying to apply the data above in this function to get the closest date before a given date (given date = today's date in my case):

# https://stackoverflow.com/questions/32237862/find-the-closest-date-to-a-given-date
def nearest(items, pivot):
    return min([i for i in items if i < pivot], key=lambda x: abs(x - pivot))

date_output = nearest(df_TradingMonthlyDates, df_EndDate)
# date_output should be = 2020-07-02 given today's date of 2020-07-12

The error messages I receive is that the df_TradingMonthlyDates is not in date format. So I have tried to convert the dataframe to datetime format but can't make it work.

What I have tried to convert the data to date format:

# df_TradingMonthlyDates["Date"] = pd.to_datetime(df_TradingMonthlyDates["Date"], format="%Y-%m-%d")
# df_TradingMonthlyDates = datetime.strptime(df_TradingMonthlyDates, "%Y-%m-%d").date()
# df_TradingMonthlyDates['Date'] = df_TradingMonthlyDates['Date'].apply(lambda x: pd.to_datetime(x[0], format="%Y-%m-%d"))
# df_TradingMonthlyDates = df_TradingMonthlyDates.iloc[1:]
# print(df_TradingMonthlyDates)
# df_TradingMonthlyDates = datetime.strptime(str(df_TradingMonthlyDates), "%Y-%m-%d").date()
# for line in split_source[1:]: # skip the first line

Code:

import pandas as pd
from datetime import datetime
 
# Version 1

TradingMonthlyDates = "G:/MonthlyDates.txt"

# Import file where all the first/end month date exists
df_TradingMonthlyDates = pd.read_csv(TradingMonthlyDates, dtype=str, sep=',') # header=True,
print(df_TradingMonthlyDates)

# https://community.dataquest.io/t/datetime-and-conversion/213425
# df_TradingMonthlyDates["Date"] = pd.to_datetime(df_TradingMonthlyDates["Date"], format="%Y-%m-%d")
# df_TradingMonthlyDates = datetime.strptime(df_TradingMonthlyDates, "%Y-%m-%d").date()
# df_TradingMonthlyDates['Date'] = df_TradingMonthlyDates['Date'].apply(lambda x: pd.to_datetime(x[0], format="%Y-%m-%d"))
# df_TradingMonthlyDates = df_TradingMonthlyDates.iloc[1:]
# print(df_TradingMonthlyDates)
# df_TradingMonthlyDates = datetime.strptime(str(df_TradingMonthlyDates), "%Y-%m-%d").date()
# for line in split_source[1:]: # skip the first line # maybe header is the problem
print(type(df_TradingMonthlyDates))
df_TradingMonthlyDates = df_TradingMonthlyDates.datetime.strptime(df_TradingMonthlyDates, "%Y-%m-%d")
df_TradingMonthlyDates = df_TradingMonthlyDates.time()


print(df_TradingMonthlyDates)

df_EndDate = datetime.now().date()
print(type(df_EndDate))



# https://stackoverflow.com/questions/32237862/find-the-closest-date-to-a-given-date
def nearest(items, pivot):
    return min([i for i in items if i < pivot], key=lambda x: abs(x - pivot))

date_output = nearest(df_TradingMonthlyDates, df_EndDate)

---

Error messages are different depending on how I tried to convert data type, but I interpret that they all notice that my date format is not successful :

df_TradingMonthlyDates = df_TradingMonthlyDates.datetime.strptime(df_TradingMonthlyDates, "%Y-%m-%d")

Traceback (most recent call last):
  File "g:/till2.py", line 25, in <module>
    df_TradingMonthlyDates = df_TradingMonthlyDates.datetime.strptime(df_TradingMonthlyDates, "%Y-%m-%d")
  File "C:\Users\ID\AppData\Roaming\Python\Python38\site-packages\pandas\core\generic.py", line 5274, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'datetime'

df_TradingMonthlyDates["Date"] = pd.to_datetime(df_TradingMonthlyDates["Date"], format="%Y-%m-%d")

Traceback (most recent call last):
  File "g:/till2.py", line 40, in <module>
    date_output = nearest(df_TradingMonthlyDates, df_EndDate)
  File "g:/till2.py", line 38, in nearest
    return min([i for i in items if i < pivot], key=lambda x: abs(x - pivot))
  File "g:/till2.py", line 38, in <listcomp>
    return min([i for i in items if i < pivot], key=lambda x: abs(x - pivot))
TypeError: '<' not supported between instances of 'str' and 'datetime.date'

Answer 1

Edit: Added Method 3, which might be the easiest with.loc and then .iloc

You could take a slightly different approach (with Method #1 or Method #2 below) by taking the absolute minimum of the difference between today's date and the data, but a key thing you weren't doing was wrapping pd.to_datetime() around the datetime.date object df_EndDate in order to transform it into a DatetimeArray so that it could be compared against your Date column. They both have to be in the same format of DatetimeArray in order to be compared.

Method 1:

import pandas as pd
import datetime as dt
df_TradingMonthlyDates = pd.DataFrame({'Date': {'0': '2008-12-30',
  '1': '2008-12-31',
  '2': '2009-01-01',
  '3': '2009-01-02',
  '4': '2009-01-29',
  '557': '2020-06-29',
  '558': '2020-06-30',
  '559': '2020-07-01',
  '560': '2020-07-02',
  '561': '2020-07-30',
  '624': '2021-11-30',
  '625': '2021-12-01',
  '626': '2021-12-02',
  '627': '2021-12-30',
  '628': '2021-12-31'}})

df_TradingMonthlyDates['Date'] = pd.to_datetime(df_TradingMonthlyDates['Date'])
df_TradingMonthlyDates['EndDate'] = pd.to_datetime(dt.datetime.now().date())
df_TradingMonthlyDates['diff'] = (df_TradingMonthlyDates['Date'] - df_TradingMonthlyDates['EndDate'])
a=min(abs(df_TradingMonthlyDates['diff']))
df_TradingMonthlyDates = df_TradingMonthlyDates.loc[(df_TradingMonthlyDates['diff'] == a)
                                                    | (df_TradingMonthlyDates['diff'] == -a)]
df_TradingMonthlyDates

output 1:

    Date        EndDate     diff
560 2020-07-02  2020-07-11  -9 days

If you don't want the extra columns and just the date, then assign variables to create series rather than new columns:

Method 2:

d = pd.to_datetime(df_TradingMonthlyDates['Date'])
t = pd.to_datetime(dt.datetime.now().date())
e = (d-t)
a=min(abs(e))
df_TradingMonthlyDates = df_TradingMonthlyDates.loc[(e == a) | (e == -a)]
df_TradingMonthlyDates

output 2:

    Date
560 2020-07-02

Method 3:

df_TradingMonthlyDates['Date'] = pd.to_datetime(df_TradingMonthlyDates['Date'])
date_output = df_TradingMonthlyDates.sort_values('Date') \
              .loc[df_TradingMonthlyDates['Date'] <= 
pd.to_datetime(dt.datetime.now().date())] \
              .iloc[-1,:]
date_output

output 3:

Date   2020-07-02
Name: 560, dtype: datetime64[ns]