When subtracting a datetime that was converted from string, total_seconds() is wrong

Question

The total_seconds() is incorrect when I do this:

from datetime import timedelta, datetime
from pytz import timezone

timezone = timezone('Australia/Sydney')
startDate = datetime.now(timezone)
dateStr = '2020-05-18 20:12:30' # In our brain we know this time is in Sydney Time
endDate = datetime.strptime(dateStr, '%Y-%m-%d %H:%M:%S').replace(tzinfo=timezone)

diff = endDate - startDate
print(diff.total_seconds()) # incorrect answer

When both datetime objects are datetime objects originally, and you substract them, they are right

from datetime import timedelta, datetime
from pytz import timezone

timezone = timezone('Australia/Sydney')
startDate = datetime.now(timezone)
endDate = datetime.now(timezone) + timedelta(hours=2, seconds=32)

diff = endDate - startDate
print(diff.total_seconds()) # correct answer

How can I fix my issue ?

@TinNguyen looks like it might be `from pytz import timezone` — Chris Doyle, May 18 '20 at 08:11
They dont seem to have the same offset. if you print the start and end dates, the start has an offset of `+10:00` and the end has an offset of `+10:05`. Is your total seconds thats incorrect different by about 300 seconds? — Chris Doyle, May 18 '20 at 08:14
Id say be consistent in your date creation. change your startdate line to `datetime.now().replace(tzinfo=timezone)` So that its creating a date consistnely like your end date. This will ensure they are both on the same offset. As to why the offset is different in your example i am still looking — Chris Doyle, May 18 '20 at 08:15
Also yes @ChrisDoyle You right. The difference is always about 5 mins — Jackie, May 18 '20 at 08:24

score 2 · Accepted Answer · edited May 18 '20 at 08:57

So it seems like all things in the horrible world of date-times, timezones and offsets this is one of these weird and wonderful things. the issue seems to stem from the fact that pytz.timezone will return a timezone object with several timezones.

{
(datetime.timedelta(seconds=36300), datetime.timedelta(0), 'LMT'): <DstTzInfo 'Australia/Sydney' LMT+10:05:00 STD>, 
(datetime.timedelta(seconds=36000), datetime.timedelta(0), 'AEST'): <DstTzInfo 'Australia/Sydney' AEST+10:00:00 STD>, 
(datetime.timedelta(seconds=39600), datetime.timedelta(seconds=3600), 'AEDT'): <DstTzInfo 'Australia/Sydney' AEDT+11:00:00 DST>
}

It seems when you are passing the timezone to the now method it's picking the timezone from your choice of 3 based on probably some local TZINFO in your setup. However, when passing the timezone to replace, it's just picking the LMT which is different by 300. A quick mention about LMT:

Local Mean Time Today: While Local Mean Time does not directly determine civil time these days, it is still used to make sure our clocks follow the Sun as closely as possible. UT1, a version of Universal Time is the Local Mean Time at the prime meridian in Greenwich, London. It is one of the components used to calculate Coordinated Universal Time (UTC), the time scale used to determine local times worldwide.

LMT is also used by astronomers around the world to time their observations.

Essentially your issue spans from datetime.now() acting on the local timezone and datetime.replace() acting on the LMT timezone. So as I mentioned in my post create your dates consistently either create them both via replace (although you will still be off by 5 mins in terms of actual time, the difference will be correct.)

UPDATE

If you want both datetime objects to be in local Sydney time then you can create your stardate as you did before using datetime.now(). But you should create your end date from your timezone objects asking it to localize it for you like.

from datetime import datetime
from pytz import timezone

timezone = timezone('Australia/Sydney')
startDate = datetime.now(timezone)
dateStr = '2020-05-18 18:52:30' # In our brain we know this time is in Sydney Time
endDate = timezone.localize(datetime.strptime(dateStr, '%Y-%m-%d %H:%M:%S'))
print(startDate, endDate, sep="\n")
diff = endDate - startDate
print(diff.total_seconds())

OUTPUT

2020-05-18 18:51:24.722614+10:00
2020-05-18 18:52:30+10:00
65.277386

So how do I actually fix it in terms of the code I provided? — Jackie, May 18 '20 at 08:44
I don't want my time to be in Local Mean Time. I want it to be in AEST time. How can I change my unaware string to aware sydney timezone? — Jackie, May 18 '20 at 08:46
I have updates the answer with a code example how to get your unaware string in to local syndey timeobject — Chris Doyle, May 18 '20 at 08:53
Sorry one more question. Say we have a `datetime` object which is `unaware` of the timezone. How can we convert an this unaware datetime object to an aware AEST Sydney timezone datetime object and not face this problem again — Jackie, May 18 '20 at 08:59
exactly the same way as you do for the string object, pass it to `timezone.localize()` cause if you look at my example thats exactly what happens here. We dont pass the string to timezone.localize, we first create a datetime object form it, which is an unaware object, we then give this to timezone to localize for us — Chris Doyle, May 18 '20 at 09:07
@ChrisDoyle: this is an interesting finding! Basically, using `replace` gives an invalid result, right? Do you know of any resources when it it is ***safe*** to use `replace`? For example, I think it is fine if you use it to add UTC tzinfo to a dt obj ( `datetime.timezone.utc`). — FObersteiner, May 18 '20 at 10:12
Honestly @MrFuppes am not sure, i had never seen it before so done a little digging, but datetimes can get complex very quickly so i am not sure on the rules or implementation under the hood that decides to use LMT or AEST in this example — Chris Doyle, May 18 '20 at 10:16
@ChrisDoyle: I think the answer to my comment above is: whenever you stay within the Python time zone model. `pytz` doesn't do that, so you have to `loaclize()`. I've added a link to a blog article in my answer that explains the issue a little bit, if you're interested. — FObersteiner, May 19 '20 at 10:24

When subtracting a datetime that was converted from string, total_seconds() is wrong

1 Answers1