How to find floating point numbers in binary file with Python?

Question

I have a binary file mixed with ASCII in which there are some floating point numbers I want to find. The file contains some lines like this:

1,1,'11.2','11.3';1,1,'100.4';

In my favorite regex tester I found that the correct regex should be ([0-9]+\.{1}[0-9]+).

Here's the code:

import re

data = open('C:\\Users\\Me\\file.bin', 'rb')
pat = re.compile(b'([0-9]+\.{1}[0-9]+)')
print(pat.match(data.read()))

I do not get a single match, why is that? I'm on Python 3.5.1.

What is a *binary file mixed with ASCII* and how can you identify the ASCII and binary parts? Without knowing that I cannot tell you how you can safely read the file. BTW, you regex fails to match `1.`, `.5`, not speaking of `1` (but for this last part it may be desirable to fail) — Serge Ballesta, Apr 14 '16 at 09:09
@SergeBallesta Sorry I meant ASCII and non-ASCII characters. — JohnnyFromBF, Apr 14 '16 at 09:11

Adem Öztaş · Accepted Answer · 2016-04-14T09:10:20.533

2

You can try like this,

import re
with open('C:\\Users\\Me\\file.bin', 'rb') as f:
    data = f.read()

re.findall("\d+\.\d+", data)

Output:

['11.2', '11.3', '100.4']

re.findall returns string list. If you want to convert to float you can do like this

>>> list(map(float, re.findall("\d+\.\d+", data)))
[11.2, 11.3, 100.4]

edited Apr 14 '16 at 09:10

answered Apr 14 '16 at 09:01

Adem Öztaş

20,457
4
34
42

Not working, tells me `TypeError: expected string or bytes-like object`. It contains binary stuff. – JohnnyFromBF Apr 14 '16 at 09:07
@JohnnyFromBF you should try, `re.findall("\d+\.\d+", data.read())` – Adem Öztaş Apr 14 '16 at 09:08
I'm afraid it still says `TypeError: cannot use a string pattern on a bytes-like object`. – JohnnyFromBF Apr 14 '16 at 09:12
@JohnnyFromBF, why are you opening in binary mode? – Padraic Cunningham Apr 14 '16 at 09:14
@PadraicCunningham If I don't it says `UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 313: character maps to `. – JohnnyFromBF Apr 14 '16 at 09:15
1

@JohnnyFromBF, are you sure it is not a latin-1 encoded file? What does the actual content look like? Try `encoding="latin-1"` – Padraic Cunningham Apr 14 '16 at 09:16
2

@PadraicCunningham Turns out, the file was indeed in a wrong encoding m( – JohnnyFromBF Apr 14 '16 at 09:50
@AdemÖztaş: the question has Python 3 tag. `str` pattern won't work with `bytes` data. – jfs Apr 14 '16 at 10:14
@JohnnyFromBF .. It was better for the sake of good interpretation of the source of your issue to post that traceback message instead...This is called an [X-Y problems](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem) – Iron Fist Apr 14 '16 at 14:12

score 2 · Answer 2 · edited May 23 '17 at 12:07

2

How to find floating point numbers in binary file with Python?

float_re = br"[+-]? *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?"
for m in generate_tokens(r'C:\Users\Me\file.bin', float_re):
    print(float(m.group()))

where float_re is from this answer and generate_tokens() is defined here.

pat.match() tries to match at the very start of the input string and your string does not start with a float and therefore you "do not get a single match".

re.findall("\d+\.\d+", data) produces TypeError because the pattern is Unicode (str) but data is a bytes object in your case. Pass the pattern as bytes:
re.findall(b"\d+\.\d+", data)

edited May 23 '17 at 12:07

Community

1
1

answered Apr 14 '16 at 10:13

jfs

399,953
195
994
1,670

Your pattern allows for a space after the leading sign character, which I believe is an error. For example, "- 2.2" is not a floating point number, but "-2.2" is. – Steve Hollasch Apr 15 '16 at 00:57
Also, if you want to be really pedantic, you could limit the number of allowable exponent digits, if you knew whether this was a 32- or 64-bit floating-point value. – Steve Hollasch Apr 15 '16 at 00:58
@SteveHollasch What is or is not a float is a matter of a definition. Follow the link that explains the origin of the regex. – jfs Apr 15 '16 at 05:17

How to find floating point numbers in binary file with Python?

2 Answers2