time.strptime() - argument 0 must be str, not bytes

Question

Obviously I'm aware already that strftime and strptime doesn't like byte strings as parameters, however i'm in a pickle here because I sort of need to read a file content which has different character encodings saved in it and i need to handle them all, and send the time portion of each line in this text-file to strptime().

A quick fix would be to split the string, making sure the time simply contains numbers and dashes, but is it possible to somehow pass the byte object without trying to figure out the encoding to strptime()?

with open('file.txt', 'rb') as fh:
    for line in fh:
        time.strptime(line, '%Y-%m-%d ...')

This would obviously fail. I thought of doing repr(line) but that causes the string to look like b'2014-01-07 ...', which i could strip..

You need to `.decode()`, but without knowing the encoding, this isn't going to be easy. — Wooble, Jan 14 '14 at 15:33
@Wooble: Well, it fits a date-time pattern, it almost certainly will fit ASCII. — Martijn Pieters, Jan 14 '14 at 15:34
"has different character encodings saved in it and i need to handle them all" — Wooble, Jan 14 '14 at 15:34
@Wooble: If there are any lines in there that don't use ASCII, they won't be holding a date that `%Y-%m-%d` can parse. — Martijn Pieters, Jan 14 '14 at 15:35

Martijn Pieters · Accepted Answer · 2014-01-14T15:39:55.060

line is a bytestring, because you opened the file in binary mode. You'll need to decode the string; if it is a date string matching the pattern, you can simply use ASCII:

 time.strptime(line.decode('ascii'), '%Y-%m-%d ...')

You can add a 'ignore' argument to ignore anything non-ASCII, but chances are the line won't fit your date format then anyway.

Note that you cannot pass a value that contains more than the parsed format in it; a line with other text on it not explicitly covered by the strptime() pattern will not work, whatever codec you used.

And if your input really varies that widely in codecs, you'll need to catch exceptions one way or another anyway.

Aside from UTF-16 or UTF-32, I would not expect you to encounter any codecs that use different bytes for the arabic numerals. If your input really mixes multi-byte and single-byte codecs in one file, you have a bigger problem on your hand, not in the least because newline handling will be majorly messed up.

`´ignore´` could work, since it's a matter of not knowing what's behind the time/date part of the string that's causing the issue, most likely a log-entry from a client with unusual language settings compared to the standard English language most of the logs contain. And since i'm mostly interested in the time/date i could just do `´ignore´` and let the string be "corrupt" at the end, since the date will be intact, thx! This most likely solves it. — Torxed, Jan 15 '14 at 08:14

score 0 · Answer 2 · answered Jan 14 '14 at 15:53

0

You should decode the data when you're reading the file:

import codecs
with codecs.open('file.txt', encoding='utf8') as fh:
    for line in fh:
        time.strptime(line, '%Y-%m-%d ...')

It's always better to decode your content as soon as possible.

Also check http://docs.python.org/2/library/codecs.html#codecs.open

answered Jan 14 '14 at 15:53

gawel

2,038
14
16

This might be a good idea but i'm afraid that utf-8 will crash on certain lines due to certain byte-data can not be converted using the utf-8 table. Don't ask me what character it is because i'm not quite sure yet because causing this issue takes about half a day of parsing through files to find it.. I didn't have debug output when i noticed this issue (stopid of me) :) – Torxed Jan 15 '14 at 08:15
you can handle errors with errors=(replace|ignore) as explained in the doc – gawel Jan 15 '14 at 10:40

time.strptime() - argument 0 must be str, not bytes

2 Answers2

Linked