5

I encountered a problem with converting the airflow macros reference 'ts' into a datetime object. The problem is with the tz at the end of the string.

from datetime import datetime
timetoday = kwargs['ts']
t = datetime.strptime(timetoday, '%Y-%m-%dT%H:%M:%S%z')

The code didn't execute successfully, instead it throws an error message:

time data '2019-08-24T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%S%z'

If I used the same code with slightly different format, and try with 'ds' macros, and I managed to convert the string into datetime object.

from datetime import datetime
timetoday = kwargs['ds']
t = datetime.strptime(timetoday, '%Y-%m-%d')

Update: Reading the notes from http://strftime.org/ and understand that %z is UTC offset in the form +HHMM or -HHMM (empty string if the the object is naive). This led me wondering whether the additional ':' in between +00:00 (returned string of kwargs['ts'] is the root cause of the error. However, I'm still unsure how to parse it correctly, as there were no other options in strptime.

Therefore, I change my macro to kwargs['ts_nodash'] that doesn't return the timezone, and proceed with my code for now. If anyone knows how to do it, I'm still interested to learn how that can be done correctly! Thanks!

timetoday = kwargs['ts_nodash']
t = datetime.strptime(timetoday, '%Y%m%dT%H%M%S')
Alex Ho
  • 418
  • 1
  • 4
  • 8
  • 1
    Possible duplicate of [Convert timestamps with offset to datetime obj using strptime](https://stackoverflow.com/questions/12281975/convert-timestamps-with-offset-to-datetime-obj-using-strptime) – drum Aug 27 '19 at 02:53
  • @drum yes, you are right... I didn't manage to find that. Anyway, the solution provided there doesn't work for me. – Alex Ho Aug 27 '19 at 03:03

2 Answers2

5

Looking at the macros reference and then your use-case, I'd suggest you to use execution_date macro instead of ts macro

{{ ts }}            : same as execution_date.isoformat(). Example: 2018-01-01T00:00:00+00:00

{{ execution_date }}: the execution_date (pendulum.Pendulum)

If your'e OK with that, it all boils down to converting the execution_date which is a pendulum object into a Python datetime object, which is as simple as datetime.fromtimestamp(execution_date.timestamp())

Python snippet (conversion) demo:

import pendulum
from pendulum import Pendulum
from datetime import datetime
..
execution_date: Pendulum = pendulum.now()
execution_date_as_datetime: datetime = datetime.fromtimestamp(execution_date.timestamp())

References

y2k-shubham
  • 10,183
  • 11
  • 55
  • 131
  • 1
    How to convert if we pass the date as a runtime argument? – Tula Oct 05 '21 at 05:08
  • You cannot deal normally with Airflow variables because they are runtime arguments. You have to use templating macros for it. You can use datetime functions inside the template string: '{{ execution_date.strftime("%d-%m-%Y") }}' – Madhava Carrillo Dec 16 '22 at 15:55
  • Can we increase the timestamp like {{ execution_date | ts }} +00050000, I need to get the timestamp starting after 5 am – Code run Apr 12 '23 at 09:59
1

You can just do it the easy way in your dag if needed.

Example airflow output object {{ ts }} from dag

2022-04-27T01:00:00+00:00

Example code in your dag or script

from datetime import datetime as dt 
launch_dt = dt.fromisoformat(kwargs['ts'])
print(launch_dt)

2022-04-27 01:00:00+00:00

Gabe Church
  • 331
  • 3
  • 7