Parsing datetime strings containing nanoseconds

Question

I have some log files with times in the format HH:MM::SS.nano_seconds (e.g. 01:02:03.123456789). I would like to create a datetime in python so I can neatly do math on the time (e.g. take time differences). strptime works well for microseconds using %f. Do the Python datetime and time modules really not support nanoseconds?

Answer 1

You can see from the source that datetime objects don't support anything more fine than microseconds. As pointed out by Mike Pennington in the comments, this is likely because computer hardware clocks aren't nearly that precise. Wikipedia says that HPET has frequency "at least 10 MHz," which means one tick per 100 nanoseconds.

If you can live with throwing out the last three digits (which probably aren't too meaningful anyway), you could parse this by just slicing the input string to have only six digits after the decimal point and parsing with %f. Otherwise, it looks like you'll have to implement the subtraction yourself.

---

Much later update: numpy and pandas now each have (somewhat different) support for timestamps that includes the possibility of tracking nanoseconds, which are often good solutions. See the other answers for how.

Python 3.7+ also has time.time_ns and related functions in time (PEP 564), but still no support for nanoseconds in datetime.

Answer 2

This is an old thread, but still...

You can use Pandas functionality to achieve this. I had timestamps like '2019-03-22T14:00:01.700311864Z' which I converted to a Timestamp by:

    firstStamp = pd.to_datetime(firstStampString, format='%Y-%m-%dT%H:%M:%S.%fZ')
    lastStamp = pd.to_datetime(lastStampString, format='%Y-%m-%dT%H:%M:%S.%fZ')

    deltaTime = lastStamp - firstStamp

This works fine.

Answer 3

You can quite naturally use nanoseconds and even more precise time units (ps, fs, as) with numpy. Numpy has its own Datetimes and Timedeltas implementation, so you can try np.datetime64:

import numpy as np
def str_to_ns(time_str):
     """
     input: time in a format `hh:mm:ss.up_to_9_digits`
     """
     h, m, s = time_str.split(":")
     int_s, ns = s.split(".")
     ns = map(lambda t, unit: np.timedelta64(t, unit),
              [h,m,int_s,ns.ljust(9, '0')],['h','m','s','ns'])
     return sum(ns)

Then you can use this function in a following way:

>>> src = "1:2:34.123456789"
>>> out = str_to_ns(src)
>>> print(out)
3754123456789 nanoseconds
>>> out / np.timedelta64(1,'h')
1.0428120713302778
>>> out / np.timedelta64(1,'m')
62.568724279816664
>>> out / np.timedelta64(1,'s')
3754.123456789

Arithmetic is also possible:

>>> t1, t2 = str_to_ns("1:0:12.12345678"), str_to_ns("1:0:12.12")
>>> t1 - t2
numpy.timedelta64(3456780,'ns')

I agree that it's not that natural, but in this manner you can achieve arbitrary high time precision with just numpy.

Answer 4

If you don't actually care about the nanoseconds, but you still want to be able to parse datetimes that have >6 decimal places in the seconds, you can use the python-dateutils library.

For example, trying to use standard lib datetime package:

>>> from datetime import datetime
>>> datetime.strptime('2021-02-14T02:27:57.96119078Z', '%Y-%m-%dT%H:%M:%S.%fZ')
ValueError: time data '2021-02-14T02:27:57.96119078Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'

But with python-dateutils, it actually parses it without throwing an error:

>>> from dateutil.parser import isoparse
>>> isoparse('2021-02-14T02:27:57.96119078Z')
datetime.datetime(2021, 2, 14, 2, 27, 57, 961190, tzinfo=tzutc())

Note that it doesn't preserve the nanoseconds (nor does it round correctly - it just chops off after 6 decimal places), but it at least won't break parsing >6 decimal places.

Answer 5

def parse_nanodate(s):
  """
  parse date, ignore nanoseconds
  sample input: 2020-12-31T16:20:00.000000123Z
  --> 123ns will be ignored
  """
  if s[-1] == 'Z':
    # add explicit UTC timezone, to make strptime happy
    s += '+0000'
  return datetime.datetime.strptime(
    s[0:26]+s[29:], '%Y-%m-%dT%H:%M:%S.%fZ%z')

Answer 6

I could remove any digits after the 6th by means of regular expression substitution:

def parse_nanosecond_ts(ts):
ts = re.sub(
    r"^([^ ]+ [0-9]+:[0-9]+:[0-9]+\.[0-9]{0,6})[0-9]*( .*)$",
    "\\1\\2",
    ts,
)
return datetime.datetime.strptime(ts, 
     "%Y-%m-%d %H:%M:%S.%f %z %Z")