0

Working with historical data in pandas, I encountered this peculiar timezone change, which I want to understand, to handle it correctly.

s = pd.Series(["1901-12-12 10:00:00", "1901-12-13 10:00:00", "1901-12-14 10:00:00", "1901-12-16 10:00:00"],
 dtype= "datetime64[ns, America/New_York]")
s
>>>
 
0   1901-12-12 10:00:00-04:56
1   1901-12-13 10:00:00-04:56   ###  4 minute shift
2   1901-12-14 10:00:00-05:00
3   1901-12-16 10:00:00-05:00
dtype: datetime64[ns, America/New_York]

Researching this, it seems to not be a bug but to have to do with Local Mean Time, which is also represented here:

from pytz import timezone
timezone("America/New_York")
>>> 
<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>  ## LMT and 4 minute oddity

Here, you can read that LMT is a system implying a 4 minute shift per degree longitude, which was replaced by standard time, which was replaced by UTC in the 60s.

I am having a hard time finding any real dates relating to the change between LMT and standard time and I am definitely not finding the date 1901-12-14 anywhere.

I understand that different regions may have adopted systems at different times and that historical data can be inconsistent, but any explanation for these system changes and/or why certain dates have been used by pandas, would be greatly appreciated.

My sub-questions:

  • Is the timezone shift on 1901-12-14 correct?
  • Why was there a timezone shift?
  • Are there important pitfalls when handling data around this date in pandas?

Thank you.

Stryder
  • 848
  • 6
  • 9
  • There was a similar question that got a huge amount of upvotes as it showcased the importance of UTC. USE UTC – chess_lover_6 Sep 24 '21 at 16:10
  • 1
    Maybe you can gain some insights from https://github.com/eggert/tz - the comments are quite comprehensive – FObersteiner Sep 24 '21 at 16:27
  • Thank you, @MrFuppes, I will definitely have a look at it. – Stryder Sep 24 '21 at 16:29
  • And I know, @chess_lover_6, but sometimes it's just not that simple and a better understanding can go a long way :) – Stryder Sep 24 '21 at 16:30
  • 1
    @Stryder I was hopping someone would find that question because it had 3 amazing answers. – chess_lover_6 Sep 24 '21 at 16:38
  • the closest I could find here on SO is [this answer](https://stackoverflow.com/a/24370601/10197418) by Matt "the time lord" Johnson-Pint. But it actually doesn't go beyond my comment above I think, "check out tz database". There also was a very interesting talk by [eumiro](https://stackoverflow.com/users/449449/eumiro?tab=profile) related to this matter, but only in German unfortunately - [Ein Tag hat nur 24±1 Stunden](https://media.ccc.de/v/gpn19-15-ein-tag-hat-nur-24-1-stunden). – FObersteiner Sep 25 '21 at 09:54
  • 1
    @MrFuppes, You gave me some very interesting stuff to look through and if I find more on this date, I will post it as an answer. Also, this is a nice coincidence, because my german is fluent, vielen Dank für Ihren Einsatz! – Stryder Sep 25 '21 at 14:11

0 Answers0