Why is accessing elements using `tolist` faster than accessing it directly through the pandas series?

Question

I have a dataframe and I wanted to apply a certain function on a set of columns. Something like:

data[["A","B","C","D","E"]].apply(some_func, axis=1)

In the some_func function, the first step is extracting out all the column values into separate variables.

def some_func(x):
    a,b,c,d,e = x # or x.tolist()
    #Some more processing

To reproduce, the result, use

x = pd.Series([1,2,3,4,5], index=["A","B","C","D","E"])

Now, my question is, why does

%%timeit 
a,b,c,d,e = x.tolist()

Output:

538 ns ± 2.82 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

perform better than

%%timeit 
a,b,c,d,e = x

Output:

1.61 µs ± 15.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Just a guess. Pandas `tolist` uses specialized operations to move the whole data from numpy array to python list at once. Your `a, b, c = x` calls `tuple(x)` which is a generic python function. This function iterates and gets one numpy element (and puts it to python list) at a time — phi, Jun 12 '22 at 13:57
You may want to use `apply` with `raw=True` to avoid creating a pandas Series for each row. Your `x` should be a tuple with `raw=True`. The code will get much faster. — phi, Jun 12 '22 at 14:02

SultanOrazbayev · Accepted Answer · 2022-06-13T05:54:28.207

Let's define two functions and inspect them with dis:

from dis import dis
from pandas import Series

x = Series([1,2,3,4,5], index=["A","B","C","D","E"])

def a():
   a, b, c, d, e = x.tolist()

def b():
   a, b, c, d, e = x

dis(a)
dis(b)

Executing the above will yield:

# dis(a)
  7           0 LOAD_GLOBAL              0 (x)
              2 LOAD_METHOD              1 (tolist)
              4 CALL_METHOD              0
              6 UNPACK_SEQUENCE          5
              8 STORE_FAST               0 (a)
             10 STORE_FAST               1 (b)
             12 STORE_FAST               2 (c)
             14 STORE_FAST               3 (d)
             16 STORE_FAST               4 (e)
             18 LOAD_CONST               0 (None)
             20 RETURN_VALUE

# dis(b)
 10           0 LOAD_GLOBAL              0 (x)
              2 UNPACK_SEQUENCE          5
              4 STORE_FAST               0 (a)
              6 STORE_FAST               1 (b)
              8 STORE_FAST               2 (c)
             10 STORE_FAST               3 (d)
             12 STORE_FAST               4 (e)
             14 LOAD_CONST               0 (None)
             16 RETURN_VALUE

From the above, it seems that, if anything, function (a) has more instructions. So why is it faster?

As explained in this answer, looking at the contents of UNPACK_SEQUENCE, one can see that there are some special-cases, such as when the number of left-hand side variables is equal to the length of the right-hand side object.

So, x.tolist() under the hood uses numpy method to create a list from the array data, which allows making use of the optimization for this special case (you can check the deterioration in performance by changing the number of arguments on the left-hand side, e.g. a, *b = range(3), will work, but will be slower than a, b, c = range(3)).

When the right-hand side object is not a Python tuple or a list, then Python iterates over the contents of the object, which appears to be less efficient.

For practical reasons, if you really want best performance (with the current versions of the modules), you can swap x.tolist() with x._values.tolist(), which should give about 10-15% boost in performance (you're just removing one layer of pandas to numpy call, and doing it directly here). The caveat is that these types of optimizations are sensitive to what's happening in lower-level code, so there is no guarantee that performance gains will be there in future Python/library combinations.

What do you exactly mean by "changing the number of arguments on the left-hand side"? I mean if I have an unequal number of variables, then I would get some "too many tuples to unwrap" kind of error right? — paradocslover, Jun 13 '22 at 02:33
Not quite, remember you can do something like `a, b, c, *d = ...` or `a, *b, e = ...`. — SultanOrazbayev, Jun 13 '22 at 02:52
For example, `a, *b = range(3)`, will work, but will be slower than `a, b, c = range(3)`. — SultanOrazbayev, Jun 13 '22 at 02:55

max · Answer 2 · 2022-06-14T19:43:14.293

First, let us check the significance again to verify the timings:

from timeit import timeit

setup = """
import pandas as pd
import numpy as np

def row_to_list(x):
    a, b, c, d, e = x
    return [a, b, c, d, e]

df1 = pd.DataFrame(np.random.rand(2000, 5))
"""
num = 10000

codes = ['lambda x: x.tolist()',
        'pd.Series.tolist',
        'row_to_list']

for code in codes:
    fnc_str = f'df1.apply({code}, axis=1)'
    t = timeit(fnc_str, setup=setup, number=num)
    print(f'{fnc_str}: {t}')

output:

df1.apply(lambda x: x.tolist(), axis=1): 111.2637004610151
df1.apply(pd.Series.tolist, axis=1): 108.36258125200402
df1.apply(row_to_list, axis=1): 141.3846389260143

OK, indeed there is a clear difference! So let's profile the function calls:

import cProfile
import pandas as pd
import numpy as np

def row_to_list(x):
    a, b, c, d, e = x
    return [a, b, c, d, e]

df1 = pd.DataFrame(np.random.rand(2000, 5))

codes = ['lambda x: x.tolist()',
        'pd.Series.tolist',
        'row_to_list']

for code in codes:
    fnc_str = f'df1.apply({code}, axis=1)'
    cProfile.run(fnc_str)

The output is actually way too long to post it here, but from the first row of the profile alone, you can see that under the hood, python is conducting 44678 (primitive) function calls in the first two cases (calling the method tolist()) and 52678 (+18%!) with unraveling the row to a list "manually".

Well, this is the magic of numpy but I can't pin it down to a more specific reason...

Why is accessing elements using `tolist` faster than accessing it directly through the pandas series?

2 Answers2