python: converting unix/posix epoch timestamp to datetime (after monogdb, date_util json export)

Question

So I received data that was stored in mongoDB then exported as a JSON via the bson.json_util.dumps() function. When I try to reconvert it back into datetime objects, none of the usual parsers work. Here's a sample:

{
u'_group': 0,
u'_range': u'',
u'canon': 0,
u'comment_id': 0,
u'created': {u'$date': 1491468607000L},
u'description': u''
}

and all the dates look like this:

[{u'$date': 1491457629000L}, {u'$date': 1491458993000L}, {u'$date': 1491457072000L}, {u'$date': 1491457035000L}, {u'$date': 1491457330000L}, {u'$date': 1491458323000L}, {u'$date': 1491458147000L}, {u'$date': 1491458277000L}, {u'$date': 1491459839000L}, {u'$date': 1491466340000L}, {u'$date': 1491463804000L}, {u'$date': 1491464304000L}, {u'$date': 1491465145000L}, {u'$date': 1492073749000L}, {u'$date': 1492073750000L}, {u'$date': 1492075415000L}, {u'$date': 1492155813000L}, {u'$date': 1492608582000L}, {u'$date': 1492671698000L}, {u'$date': 1493001047000L}, {u'$date': 1493714117000L}]

My tests:

y = {u'$date': 1491457629000L}
>>> y['$date']
1491457629000L

And various failed attempts:

print(
    datetime.datetime.fromtimestamp(
        y['$date']
    ).strftime('%Y-%m-%d %H:%M:%S')
)

ValueError: timestamp out of range for platform localtime()/gmtime() function

from dateutil import parser
parser.parse(str(y['$date']))

OverflowError: Python int too large to convert to C long

utc_time = datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=y['$date'])

OverflowError: date value out of range

It seems like any unix timestamp converter should be able to read this. Why isn't it working? I also only need the year and month, so timezone specificity is outside the scope of my demands.

Note (to the downvoter): This is not a duplicate of converting epoch time with milliseconds to datetime because the solution here was to use a different JSON parser that mirrors the way it was converted out of mongodb, instead of trying to reformat it with an unrelated timestamp parser. I had already researched the other link and found it didn't solve my problem.

Possible duplicate of [converting epoch time with milliseconds to datetime](https://stackoverflow.com/questions/21787496/converting-epoch-time-with-milliseconds-to-datetime) — , Jul 28 '17 at 13:40
Timezones are important, btw. Example: if the timestamp value corresponds to `2017-12-31T23:00Z`, the year and month can be either December 2017 or January 2018, depending on the timezone. Actually, any date in the first or last day of a month will have a different value for the month, depending on the timezone. Of course you can assume that it will always use the system's default timezone (or any specific one). Even though, that's something you must consider. — , Jul 28 '17 at 14:05

score 0 · Answer 1 · answered Jul 28 '17 at 13:47

0

Have you tried removing the last 3 zeroes (dividing by 1000)?

Error seems to be coming from conversion, your value probably is in milliseconds, python datetime.datetime.fromtimestamp expects seconds it seems.

answered Jul 28 '17 at 13:47

Marc Z

116
5

score 0 · Accepted Answer · answered Jul 28 '17 at 15:51

This worked. I don't know why this particular timestamp format won't convert with other tools,

{u'$date': 1491457629000L}

but reversing the json_util.dumps() method with its own corresponding json_util.loads() method works. It have to do that on the requested data instead of using standard .json() method in python's requests module:

from bson import json_util  
import requests
raw = requests.get('someurlhere')
data = json_util.loads(raw.text)
   {...
   u'created': datetime.datetime(2017, 4, 6, 5, 47, 9, tzinfo=<bson.tz_util.FixedOffset object at 0x0000000002FAAC18>),
...}

That yields a datetime object and I can work with that.

python: converting unix/posix epoch timestamp to datetime (after monogdb, date_util json export)

2 Answers2