15

I would like to convert string date format to timestamp with microseconds I try the following but not giving expected result:

"""input string date -> 2014-08-01 04:41:52,117
expected result -> 1410748201.117"""

import time
import datetime

myDate = "2014-08-01 04:41:52,117"
timestamp = time.mktime(datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple())

print timestamp
> 1410748201.0

Where did the milliseconds go?

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Krilin
  • 153
  • 1
  • 1
  • 5
  • the input date (2014-08-01) doesn't correspond to the timestamp (2014-09-15 02:30:01.117Z) – jfs Oct 02 '14 at 13:57

3 Answers3

21

There is no slot for the microseconds component in a time tuple:

>>> import time
>>> import datetime
>>> myDate = "2014-08-01 04:41:52,117"
>>> datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple()
time.struct_time(tm_year=2014, tm_mon=8, tm_mday=1, tm_hour=4, tm_min=41, tm_sec=52, tm_wday=4, tm_yday=213, tm_isdst=-1)

You'll have to add those manually:

>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> time.mktime(dt.timetuple()) + (dt.microsecond / 1000000.0)
1406864512.117

The other method you could follow is to produce a timedelta() object relative to the epoch, then get the timestamp with the timedelta.total_seconds() method:

epoch = datetime.datetime.fromtimestamp(0)
(dt - epoch).total_seconds()

The use of a local time epoch is quite deliberate since you have a naive (not timezone-aware) datetime value. This method can be inaccurate based on the history of your local timezone however, see J.F. Sebastian's comment. You'd have to convert the naive datetime value to a timezone-aware datetime value first using your local timezone before subtracting a timezone-aware epoch.

As such, it is easier to stick to the timetuple() + microseconds approach.

Demo:

>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> epoch = datetime.datetime.fromtimestamp(0)
>>> (dt - epoch).total_seconds()
1406864512.117
Community
  • 1
  • 1
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • 1
    **1** `(dt - epoch)` is incorrect if `dt` is a naive local time. Local time may be set back (DST transition in the fall in the northern hemisphere). Timestamp should **not** be set back in this case. **2** Also "local time epoch" might be misleading. The epoch is a *single* time instance (1970 for Python) -- it is the same around the world. If C library supports a historical timezone database or the UTC for the local timezone had the same value at the time then `datetime.fromtimestamp(0)` returns local time corresponding to the epoch. – jfs Oct 02 '14 at 13:58
  • @J.F.Sebastian: the `epoch` value uses `fromtimestamp()`, not `utcfromtimestamp()`, to handle the DST difference, precisely because the `dt` value is naive. – Martijn Pieters Oct 02 '14 at 14:01
  • @J.F.Sebastian: Had I used `utcfromtimestamp()` instead, I'd have to first make `dt` timezone aware and thus allow Python to compensate for the DST issues. – Martijn Pieters Oct 02 '14 at 14:02
  • you misunderstood. The issue is that the formula uses local time. Local time may be ambiguous (there is nothing you can do if it is the input and no other info is available). And the utc offset for the local timezone may be different in the past (you need the tz database to find it). The correct formula: `ts = utc_time - epoch_in_utc = local time - utc offset(now) - epoch in local time + utc offset (then)`. Your formula is equivalent to using UTC offset as it is now i.e., it assumes `utc offset(now) == utc offset (then)`. – jfs Oct 02 '14 at 14:09
  • @J.F.Sebastian: ah, good point. The formula would be easier with a proper timezone database. As an *approximation*, ignoring historical issues, is it good enough? Because `time.time()` doesn't use historical values *either*. – Martijn Pieters Oct 02 '14 at 14:11
  • `time.time()` doesn't depend on the local timezone (if we ignore systems with `'right'` timezones (it is a different time scale)). It doesn't need the historical timezone database. – jfs Oct 02 '14 at 14:12
5

In Python 3.4 and later you can use

timestamp = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timestamp()

This doesn't require importing the time module. It also uses less steps so it should be faster. For older versions of python the other provided answers are probably your best option.

However, the resulting timestamp will interpret myDate in local time, rather than UTC, which may cause issues if myDate was given in UTC

Ally
  • 1
  • 3
user144153
  • 829
  • 1
  • 12
  • 28
2

Where did the milliseconds go?

It is the easy part. .timetuple() call drops them. You could add them back using .microsecond attribute. The datetime.timestamp() method from the standard library works that way for naive datetime objects:

def timestamp(self):
    "Return POSIX timestamp as float"
    if self._tzinfo is None:
        return _time.mktime((self.year, self.month, self.day,
                             self.hour, self.minute, self.second,
                             -1, -1, -1)) + self.microsecond / 1e6
    else:
        return (self - _EPOCH).total_seconds()

It is enough if possible ~1 hour errors could be ignored in your case. I assume that you want microseconds and therefore you can't ignore ~1 hour time errors silently.

To convert the local time given as a string to the POSIX timestamp correctly is a complex task in general. You could convert the local time to UTC and then get the timestamp from UTC time.

There are two main issues:

Both can be solved using the tz database (pytz module in Python):

from datetime import datetime
import pytz # $ pip install pytz
from tzlocal import get_localzone # $ pip install tzlocal

tz = get_localzone() # get pytz timezone corresponding to the local timezone

naive_d = datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
# a) raise exception for non-existent or ambiguous times
d = tz.localize(naive_d, is_dst=None)
## b) assume standard time, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=False))
## c) assume DST is in effect, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=True))
timestamp = d - datetime(1970, 1, 1, tzinfo=pytz.utc)

The result is timestamp -- a timedelta object, you can convert it to seconds, milliseconds, etc.

Also different systems may behave differently around/during leap seconds. Most application can ignore that they exist.

In general, it might be simpler to store POSIX timestamps in addition to the local time instead of trying to guess it from the local time.

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • Note: `datetime.timestamp` is only available in Python 3; the OP is using Python 2 (judging by the `print` statements). – Martijn Pieters Oct 02 '14 at 16:33
  • @MartijnPieters: I have *not* suggested to use `.timestamp()` method. I just use it as an example (flawed) implementation. – jfs Oct 02 '14 at 16:58
  • I didn't mean to suggest you did. I am only pointing out that the method doesn't exist in Python 2. – Martijn Pieters Oct 02 '14 at 17:04
  • 2
    @MartijnPieters: I understand *what* you said. I don't understand *why* you said it. It doesn't matter in this case that the code happens to be included in Python 3 stdlib `+ d.microsecond / 1e6` works the same way on Python 2 and 3. – jfs Oct 03 '14 at 10:29