save time with lambda function + map with 2D array

Question

I was convinced to save computation time in using lambda function, but it's not that clear. look at this example:

import numpy as np
import timeit

def f_with_lambda():
    a = np.array(range(5))
    b = np.array(range(5))
    A,B = np.meshgrid(a,b)

    rst = list(map(lambda x,y : x+y , A, B))

    return np.array(rst)

def f_with_for():
    a = range(5)
    b = np.array(range(5))

    rst = [b+x for x in a]

    return np.array(rst)

lambda_rst = f_with_lambda()
for_rst = f_with_for()

if __name__ == '__main__':
    print(timeit.timeit("f_with_lambda()",setup = "from __main__ import f_with_lambda",number = 10000))
    print(timeit.timeit("f_with_for()",setup = "from __main__ import f_with_for",number = 10000))

result is simple: -lambda function result with time it is 0.3514268280014221 s - with for loop : 0.10633227700236603 s

How do I write my lambda function to be competitive ? I noticed the list function to get results from de map object is not good in time. Any other way to proceed ? the mesgrid function is certainly not the best as well...

every tip is welcome!

yes, for example with 150 instead of 5: 2.696938256998692s vs 1.9488381640003354s — lelorrain7, Jun 07 '19 at 13:46
Where have you read that with `lambda` it should be faster? I understand the use of `map`. But using `lambda` should not make it faster. Isn't better to define a function like `add(a,b)` and then pass that to `map()`? — alec_djinn, Jun 07 '19 at 13:57
Look at my edit, it becomes faster and competitive without the list. — politinsa, Jun 07 '19 at 14:01

score 2 · Answer 1 · answered Jun 07 '19 at 14:16

Considering the remark about the list:

import numpy as np
import timeit

def f_with_lambda():
    A,B = np.meshgrid(range(150),range(150))
    return np.array(map(lambda x,y : x+y , A, B))


def f_with_for():
    return np.array([np.array(range(150))+x for x in range(150)])


if __name__ == '__main__':
    print(timeit.timeit("f_with_lambda()",setup = "from __main__ import f_with_lambda",number = 10000))
    print(timeit.timeit("f_with_for()",setup = "from __main__ import f_with_for",number = 10000))

it is changing a lot of things. This time (lambda vs for)

for 5: 0.30227499100146815 vs 0.2510572589999356 (quite similar)

for 150: 0.6687559890015109 vs 20.31807473200024 ( :) :) :) ) !! great job! thank you!

politinsa · Accepted Answer · 2019-06-07T14:04:44.950

Memory allocation is taking time (it should call an OS procedure, it might be delayed).
In the lambda version, you allocated a, b, meshgrid, rst (list and array versions) + the return array.
In the for version, you allocated b and rst + the return array. a is a generator so it takes no time to create and load it in memory.

This is why your function using lambda is slower.

Plus, don't use list to handle result of np-array operations to cast it back to np-array.
Just by removing the list() it become faster (from 0.9 to 0.4).

def f_with_lambda():
    a = np.array(range(SIZE))
    b = np.array(range(SIZE))
    A,B = np.meshgrid(a,b)

    rst = map(lambda x,y : x+y , A, B)

    return np.array(rst)

See https://stackoverflow.com/a/46470401/9453926 for speed comparison.

score 0 · Answer 3 · answered Jun 07 '19 at 14:07

I compacted the code:

import numpy as np
import timeit

def f_with_lambda():
    A,B = np.meshgrid(range(150),range(150))
    return np.array(list(map(lambda x,y : x+y , A, B)))


def f_with_for():
    return np.array([np.array(range(150))+x for x in range(150)])


if __name__ == '__main__':
    print(timeit.timeit("f_with_lambda()",setup = "from __main__ import f_with_lambda",number = 10000))
    print(timeit.timeit("f_with_for()",setup = "from __main__ import f_with_for",number = 10000))

This time, for a 5x5, the result is

Lambda vs for

0.38113487999726203 vs 0.24913009200099623

and with 150 it's better:

2.680842614001449 vs 20.176408246999927

But I found no way to integrate the mesgrid inside the lambda function. and the list conversion before the array is sad as well.

You are still using a list [] and converting it to array. Forget about lists. Use numpy array. — politinsa, Jun 07 '19 at 14:21

score 0 · Answer 4 · edited Dec 11 '21 at 02:16

I took time to integrate the last remark from politinsa:

import numpy as np
import timeit

def f_with_lambda():
    A,B = np.meshgrid(range(150),range(150))
    return np.array(list(map(lambda x,y : x+y , A, B)))


def f_with_for():
    return np.array([np.array(range(150))+x for x in range(150)])

def f_with_lambda_nolist():
    A,B = np.meshgrid(range(150),range(150))
    return np.array(map(lambda x,y : x+y , A, B))


if __name__ == '__main__':
    print(timeit.timeit("f_with_lambda()",setup = "from __main__ import f_with_lambda",number = 10000))
    print(timeit.timeit("f_with_for()",setup = "from __main__ import f_with_for",number = 10000))
    print(timeit.timeit("f_with_lambda_nolist()",setup = "from __main__ import f_with_lambda_nolist",number = 10000))

results are: 2.4421722999977646 s 18.75847979998798 s 0.6800016999914078 s -> list conversion has (as explained) a real impact on memory allocation

save time with lambda function + map with 2D array

4 Answers4