How to read binary files in Python using NumPy?

Question

I know how to read binary files in Python using NumPy's np.fromfile() function. The issue I'm faced with is that when I do so, the array has exceedingly large numbers of the order of 10^100 or so, with random nan and inf values.

I need to apply machine learning algorithms to this dataset and I cannot work with this data. I cannot normalise the dataset because of the nan values.

I've tried np.nan_to_num() but that doesn't seem to work. After doing so, my min and max values range from 3e-38 and 3e+38 respectively, so I could not normalize it.

Is there any way to scale this data down? If not, how should I deal with this?

Thank you.

EDIT:

Some context. I'm working on a malware classification problem. My dataset consists of live malware binaries. They are files of the type .exe, .apk etc. My idea is store these binaries as a numpy array, convert to a grayscale image and then perform pattern analysis on it.

what do you mean "binary file"? can you add a sample of the file to your post? — maxymoo, Sep 29 '16 at 05:32
What kinds of numbers do you expect? Have you tried specifying the `dtype` parameter? It's using the default `float`. — hpaulj, Sep 29 '16 at 05:58
OK, the values don't represent real numbers; no wonder the float version looks odd. Try a dtype like `np.uint8`, which represents each byte as number between 0 and 255. — hpaulj, Sep 29 '16 at 06:08

score 32 · Accepted Answer · answered Sep 29 '16 at 06:15

If you want to make an image out of a binary file, you need to read it in as integer, not float. Currently, the most common format for images is unsigned 8-bit integers.

As an example, let's make an image out of the first 10,000 bytes of /bin/bash:

>>> import numpy as np
>>> import cv2
>>> xbash = np.fromfile('/bin/bash', dtype='uint8')
>>> xbash.shape
(1086744,)
>>> cv2.imwrite('bash1.png', xbash[:10000].reshape(100,100))

In the above, we used the OpenCV library to write the integers to a PNG file. Any of several other imaging libraries could have been used.

This what the first 10,000 bytes of bash "looks" like:

This worked! I was reading the binary files as floats, which caused the error. I read it as uint8 and it worked fine. Thanks — Suyash Shetty, Sep 29 '16 at 06:21

score 1 · Answer 2 · edited May 23 '17 at 12:33

EDIT 2

Refer this answer: https://stackoverflow.com/a/11548224/6633975

It states: NaN can't be stored in an integer array. This is a known limitation of pandas at the moment; I have been waiting for progress to be made with NA values in NumPy (similar to NAs in R), but it will be at least 6 months to a year before NumPy gets these features, it seems:

source: http://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na

Numpy integer nan
Accepted answer states:NaN can't be stored in an integer array. A nan is a special value for float arrays only. There are talks about introducing a special bit that would allow non-float arrays to store what in practice would correspond to a nan, but so far (2012/10), it's only talks. In the meantime, you may want to consider the numpy.ma package: instead of picking an invalid integer like -99999, you could use the special numpy.ma.masked value to represent an invalid value.

a = np.ma.array([1,2,3,4,5], dtype=int)
a[1] = np.ma.masked
masked_array(data = [1 -- 3 4 5],
             mask = [False  True False False False],
       fill_value = 999999)

EDIT 1

To read binary file:

Read the binary file content like this:

with open(fileName, mode='rb') as file: # b is important -> binary
    fileContent = file.read()

After that you can "unpack" binary data using struct.unpack

If you are using np.fromfile() function:

numpy.fromfile, which can read data from both text and binary files. You would first construct a data type, which represents your file format, using numpy.dtype, and then read this type from file using numpy.fromfile.

How to read binary files in Python using NumPy?

2 Answers2