Converting timezone-aware date string to UTC and back in Python

Question

I'm parsing the National Weather Service alerts feed into a web application. I'd like to purge the alerts when they hit their expiration time. I'd also like to display the expiration time in the local time format for the geographic area they pertain to.

The alerts cover the whole US, so I think the best approach is to store and compare the times in UTC timestamps. The expiration time arrives in the feed as a string like this: 2011-09-09T22:12:00-04:00.

I'm using the Labix dateutils package to parse the string in a timezone-aware manner:

>>> from dateutil.parser import parse
>>> d = parse("2011-09-18T15:52:00-04:00")
>>> d
datetime.datetime(2011, 9, 18, 15, 52, tzinfo=tzoffset(None, -14400))

I'm also able to capture the UTC offset in hours:

>>> offset_hours = (d.utcoffset().days * 86400 + d.utcoffset().seconds) / 3600
>>> offset_hours
-4

Using the datetime.utctimetuple() and time.mktime() methods, I'm able to convert the parsed date to a UTC timestamp:

>>> import time
>>> expiration_utc_ts = time.mktime(d.utctimetuple())
>>> expiration_utc_ts
1316393520.0

At this point, I feel pretty good that I'm able to convert the raw strings into a timestamp representing the expiration time in UTC. I'm able to compare the current time as a UTC timestamp to the expiration and determine if it needs to be purged:

>>> now_utc_ts = time.mktime(time.gmtime())
>>> now_utc_ts
1316398744.0
>>> now_utc_ts >= expiration_tc_ts
True

The difficulty I'm having is trying to convert my stored UTC timestamp back to the original localized format. I have the offset hours stored from the original conversion and a string I parsed to store the timezone label:

>>> print offset_hours
-4
>>> print timezone
EDT

I'd like to convert the UTC timestamp back to a locally formatted time, but converting it back to a datetime doesn't seem to be working:

>>> import datetime
>>> datetime.datetime.fromtimestamp(expiration_utc_ts) + datetime.timedelta(hours=offset_hours)
datetime.datetime(2011, 9, 18, 16, 52) # The hour is 16 but it should be 15

It looks like it's off by an hour. I'm not sure where the error was introduced? I put together another test and got similar results:

>>> # Running this at 21:29pm EDT
>>> utc_now = datetime.datetime.utcnow()
>>> utc_now_ts = time.mktime(right_now.utctimetuple())
>>> datetime.datetime.fromtimestamp(utc_now_ts)
datetime.datetime(2011, 9, 18, 22, 29, 47) # Off by 1 hour

Can someone help me find my mistake? I'm not sure if it's a daylight savings issue? I came across some stuff that leads me to believe it might be trying to localize my dates and times but at this point I'm pretty stumped. I was hoping to do all of these calculations/comparisons in a timezone-agnostic manner.

Answer 1

The problem is that Daylight Savings time is being applied twice.

A trivial example:

>>> time_tuple = datetime(2011,3,13,2,1,1).utctimetuple()
time.struct_time(tm_year=2011, tm_mon=3, tm_mday=13, tm_hour=2, tm_min=1, tm_sec=1, tm_wday=6, tm_yday=72, tm_isdst=0)
>>> datetime.fromtimestamp(time.mktime(time_tuple))
datetime.datetime(2011, 3, 13, 3, 1, 1)

I am fairly certain that the fault lies within time.mktime(). As it says in its documentation:

This is the inverse function of localtime(). Its argument is the struct_time or full 9-tuple (since the dst flag is needed; use -1 as the dst flag if it is unknown) which expresses the time in local time, not UTC. It returns a floating point number, for compatibility with time(). If the input value cannot be represented as a valid time, either OverflowError or ValueError will be raised (which depends on whether the invalid value is caught by Python or the underlying C libraries). The earliest date for which it can generate a time is platform-dependent.

When you pass a time tuple to time.mktime(), it expects a flag on whether the time is in daylight savings time or not. As you can see above, utctimetuple() returns a tuple with that flag marked 0, as it says it will do in its documentation:

If datetime instance d is naive, this is the same as d.timetuple() except that tm_isdst is forced to 0 regardless of what d.dst() returns. DST is never in effect for a UTC time.

If d is aware, d is normalized to UTC time, by subtracting d.utcoffset(), and a time.struct_time for the normalized time is returned. tm_isdst is forced to 0. Note that the result’s tm_year member may be MINYEAR-1 or MAXYEAR+1, if d.year was MINYEAR or MAXYEAR and UTC adjustment spills over a year boundary.

Since you have told time.mktime() that your time is not DST, and its job is to convert all times into local time, and it is currently daylight savings time in your area, it adds an hour to make it daylight savings time. Hence the result.

---

While I don't have the post handy, I came across a method a couple of days ago to convert timezone-aware datetimes into naive ones in your local time. This might work much better for your application than what you are currently doing (uses the excellent pytz module):

import pytz
def convert_to_local_time(dt_aware):
    tz = pytz.timezone('America/Los_Angeles') # Replace this with your time zone string
    dt_my_tz = dt_aware.astimezone(tz)
    dt_naive = dt_my_tz.replace(tzinfo=None)
    return dt_naive

Replace 'America/LosAngeles' with your own timezone string, which you can find somewhere in pytz.all_timezones.

Answer 2

datetime.fromtimestamp() is the correct method to get local time from POSIX timestamp. The issue in your question is that you convert an aware datetime object to POSIX timestamp using time.mktime() that is incorrect. Here's one of correct ways to do it:

expiration_utc_ts = (d - datetime(1970, 1, 1, tzinfo=utc)).total_seconds()
local_dt = datetime.fromtimestamp(expiration_utc_ts)