How to remove unconverted data from a Python datetime object

Question

I have a database of mostly correct datetimes but a few are broke like so: Sat Dec 22 12:34:08 PST 20102015

Without the invalid year, this was working for me:

end_date = soup('tr')[4].contents[1].renderContents()
end_date = time.strptime(end_date,"%a %b %d %H:%M:%S %Z %Y")
end_date = datetime.fromtimestamp(time.mktime(end_date))

But once I hit an object with a invalid year I get ValueError: unconverted data remains: 2, which is great but im not sure how best to strip the bad characters out of the year. They range from 2 to 6 unconverted characters.

Any pointers? I would just slice end_date but im hoping there is a datetime-safe strategy.

score 34 · Answer 1 · edited Nov 16 '16 at 23:47

Unless you want to rewrite strptime (a very bad idea), the only real option you have is to slice end_date and chop off the extra characters at the end, assuming that this will give you the correct result you intend.

For example, you can catch the ValueError, slice, and try again:

def parse_prefix(line, fmt):
    try:
        t = time.strptime(line, fmt)
    except ValueError as v:
        if len(v.args) > 0 and v.args[0].startswith('unconverted data remains: '):
            line = line[:-(len(v.args[0]) - 26)]
            t = time.strptime(line, fmt)
        else:
            raise
    return t

For example:

parse_prefix(
    '2015-10-15 11:33:20.738 45162 INFO core.api.wsgi yadda yadda.',
    '%Y-%m-%d %H:%M:%S'
) # -> time.struct_time(tm_year=2015, tm_mon=10, tm_mday=15, tm_hour=11, tm_min=33, ...

Nice to remove exactly is left over without too many assumptions. — S.Lott, Feb 18 '11 at 19:09
Could also replace `time.strptime` with `datetime.datetime.strptime` for a datetime object, as requested. — Nick T, Nov 16 '16 at 23:49

score 25 · Accepted Answer · answered Feb 18 '11 at 18:57

25

Yeah, I'd just chop off the extra numbers. Assuming they are always appended to the datestring, then something like this would work:

end_date = end_date.split(" ")
end_date[-1] = end_date[-1][:4]
end_date = " ".join(end_date)

I was going to try to get the number of excess digits from the exception, but on my installed versions of Python (2.6.6 and 3.1.2) that information isn't actually there; it just says that the data does not match the format. Of course, you could just continue lopping off digits one at a time and re-parsing until you don't get an exception.

You could also write a regex that will match only valid dates, including the right number of digits in the year, but that seems like overkill.

answered Feb 18 '11 at 18:57

kindall

178,883
35
278
309

3

+1, chopping off the excess is the best approach. Here is a fairly simple one-liner to do the same thing: `end_date = end_date[:end_date.rindex(" ")+5]` – Andrew Clark Feb 18 '11 at 19:02
@Andrew: Yeah, that's a much better way. – kindall Feb 18 '11 at 19:06
Doesn't work for '2021-05-13T23:30:52.000Z'. Kindly don't post anything as a generic solution without verifying. – saurabh gupta May 15 '21 at 21:08

score 9 · Answer 3 · answered Sep 06 '18 at 21:20

9

Here's an even simpler one-liner I use:

end_date = end_date[:-4]

answered Sep 06 '18 at 21:20

skeller88

4,276
1
32
34

eyquem · Answer 4 · 2011-02-18T19:52:28.340

Improving (i hope) the code of Adam Rosenfield:

import time

for end_date in ( 'Fri Feb 18 20:41:47 Paris, Madrid 2011',
                  'Fri Feb 18 20:41:47 Paris, Madrid 20112015'):

    print end_date

    fmt = "%a %b %d %H:%M:%S %Z %Y"
    try:
        end_date = time.strptime(end_date, fmt)
    except ValueError, v:
        ulr = len(v.args[0].partition('unconverted data remains: ')[2])
        if ulr:
            end_date = time.strptime(end_date[:-ulr], fmt)
        else:
            raise v

    print end_date,'\n'

score 0 · Answer 5 · answered Feb 18 '11 at 18:59

0

strptime() really expects to see a correctly formatted date, so you probably need to do some munging on the end_date string before you call it.

This is one way to chop the last item in the end_date to 4 chars:

chop = len(end_date.split()[-1]) - 4
end_date = end_date[:-chop]

answered Feb 18 '11 at 18:59

payne

13,833
5
42
49

This is pretty good too but I probably wont remember it as easily as kindall's answer. – Ben Keating Feb 18 '11 at 19:08

score 0 · Answer 6 · answered Jul 31 '23 at 10:40

0

from datetime import datetime
ReportingDate = struct[7][1:-1] # 6/21/2022 5:00
dt = ReportingDate[:-5] # 6/21/2022
ReportingDate1 = datetime.strptime(dt, "%m/%d/%Y").strftime("%Y-%m-%d")

answered Jul 31 '23 at 10:40

Vong Vilong

1

3

Answer needs supporting information Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](https://stackoverflow.com/help/how-to-answer). – moken Aug 01 '23 at 12:34

How to remove unconverted data from a Python datetime object

6 Answers6

Linked

Related