apply function takes a long time to run

Question

I'm working with a dataset of about ~ 32.000.000 rows:

RangeIndex: 32084542 entries, 0 to 32084541

df.head()


        time                        device      kpi                                 value
0   2020-10-22 00:04:03+00:00       1-xxxx  chassis.routing-engine.0.cpu-idle   100
1   2020-10-22 00:04:06+00:00       2-yyyy  chassis.routing-engine.0.cpu-idle   97
2   2020-10-22 00:04:07+00:00       3-zzzz  chassis.routing-engine.0.cpu-idle   100
3   2020-10-22 00:04:10+00:00       4-dddd  chassis.routing-engine.0.cpu-idle   93
4   2020-10-22 00:04:10+00:00       5-rrrr  chassis.routing-engine.0.cpu-idle   99

My goal is to create one aditional columns named role, filled with regard a regex

This is my approach

def router_role(row):
    if row["device"].startswith("1"):
        row["role"] = '1'
    if row["device"].startswith("2"):
        row["role"] = '2'
    if row["device"].startswith("3"):
        row["role"] = '3'
    if row["device"].startswith("4"):
        row["role"] = '4'
    return row

then,

df = df.apply(router_role,axis=1)

However it's taking a lot of time ... any idea about other possible approach ?

Thanks

score 4 · Accepted Answer · answered Nov 19 '20 at 18:18

4

Apply is very slow and never very good. Try something like this instead:

df['role'] = df['device'].str[0]

answered Nov 19 '20 at 18:18

teepee

2,620
2
22
47

Thanks a lot , simple and fast – psagrera Nov 19 '20 at 18:26
And like someone pointed out, if you are dealing with more numbers than you showed in your `router_role` function, it would be best to use this: `df['role'] = df['device'].str.extract('^(\d+)')` – teepee Nov 19 '20 at 21:50

score 1 · Answer 2 · answered Nov 19 '20 at 18:24

Using apply is notoriously slow because it doesn't take advantage of multithreading (see, for example, pandas multiprocessing apply). Instead, use built-ins:

>>> import pandas as pd
>>> df = pd.DataFrame([["some-data", "1-xxxx"], ["more-data", "1-yyyy"], ["other-data", "2-xxxx"]])
>>> df
            0       1
0   some-data  1-xxxx
1   more-data  1-yyyy
2  other-data  2-xxxx
>>> df["Derived Column"] = df[1].str.split("-", expand=True)[0]
>>> df
            0       1 Derived Column
0   some-data  1-xxxx              1
1   more-data  1-yyyy              1
2  other-data  2-xxxx              2

Here, I'm assuming that you might have multiple digits before the hyphen (e.g. 42-aaaa), hence the extra work to split the column and get the first value of the split. If you're just getting the first character, do what @teepee did in their answer with just indexing into the string.

score 0 · Answer 3 · answered Nov 19 '20 at 21:27

0

You can trivially convert your code to use np.vectorize().

See here: Performance of Pandas apply vs np.vectorize to create new column from existing columns

answered Nov 19 '20 at 21:27

stackoverflowuser2010

38,621
48
169
217

apply function takes a long time to run

3 Answers3