Call a function on whole array instead of on every element of it

Question

Let's say I have a list of tuples in a form of:

 data = [(1,2), (1,2), ..., (1,2)]

and a method data_to_bytes that accepts tuple does something to it and returns bytes. Now I want to call this method on every element from data and save output to the file. This is something I already have:

def create_data_file(data_file, data):
    with data_file.open('wb') as _file:
        for i in data:
            _file.write(data_to_bytes(i))
    return data_file

But this is terribly slow and I would like to improve it. Maybe it is possible to get rid of the loop inside with. I was thinking about using numpy somehow and maybe call data_to_bytes() on whole array instead of every element. Is this possible somehow?

The speed of numpy applications mainly comes from the fact that numpy uses C under the hood. Applying your own function on an array is usually slow. [This thread](https://stackoverflow.com/questions/35215161/most-efficient-way-to-map-function-over-numpy-array) covers some ways to map functions over an array. — Timo, Jul 18 '22 at 09:16
You can create a numpy array of an intended datatype, apply some mathematical operation on it and write it to a file using https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tofile.html — max9111, Jul 18 '22 at 12:49

score 0 · Answer 1 · answered Jul 18 '22 at 08:51

Numpy is only fast if you can replace your "data_to_bytes" by numpy functions that run on the whole array. Numpy does not vectorize arbitrary functions.

If you don't think that's possible, other ways of increasing performance might be:

Caching results of "data_to_bytes" or
Improving the performance of "data_to_bytes" itself

score 0 · Answer 2 · answered Jul 18 '22 at 08:52

0

Take a look at numpy.apply_along_axis. This takes the input array and applies the given function along the axis returning a new array with the computed data.

answered Jul 18 '22 at 08:52

MangoNrFive

1,541
1
10
23

score 0 · Answer 3 · answered Jul 18 '22 at 08:55

Try this code:

data = [(1, 2), (1, 2), (1, 2)]
def create_data_file(data_file, data):
    with open(data_file, 'wb') as _file:
        for i in data:
            _file.write(bytearray(i))
            print(type(bytearray(i)))
    return data_file
create_data_file('test.txt', data)

Call a function on whole array instead of on every element of it

3 Answers3