Concatenating dataframes in a loop very slow

Question

I am building one "master" dataframes by concatenating in a for loop

df = pd.concat([df, pd.DataFrame(listdf)], axis = 1)

listdf is the temp list for preprocessing before it is concatenated to the dataframe. The problem I am facing is that as the number of iterations of the loop increases, process of building the dataframe gets slower and slower.

I have researched here and based on this suggestion, it is much faster to build a list and then to finally concatenate it into a dataframe Why does concatenation of DataFrames get exponentially slower?

However this approach is not working

interimdf.append(listdf)
df = pd.concat(interimdf)

I am getting this error

TypeError: cannot concatenate object of type '<class 'list'>'; only Series and DataFrame objs are valid

How do I resolve this issue. Also there is an alternative way to speed up the process to create one big dataframe in a loop

What is the type of listdf? `type(listdf)` For this to work, listdf must by a pd.Series or pd.DataFrame. — Scott Boston, Sep 13 '20 at 13:48
I not sure if this will work for you, but looking at your post here. Try this instead `interimdf.append(pd.DataFrame(listdf))` — Scott Boston, Sep 13 '20 at 13:50
That worked awesomely. Blazing fast now. Earlier it was taking me 3-4 hrs to run that loop and now it finishes in less than a second. Thanks — Bappa, Sep 13 '20 at 14:07
Awesome. I glad this worked for you. Yeah, I will remember the "quadratic copying" problem. I submitted a solution. — Scott Boston, Sep 13 '20 at 15:03

score 1 · Accepted Answer · answered Sep 13 '20 at 15:00

1

You need to append dataframes to your list and not the data as list.

Try:

interimdf.append(pd.DataFrame(listdf))

Then outside of your loop,

pd.concat(interimdf)

answered Sep 13 '20 at 15:00

Scott Boston

147,308
15
139
187

Concatenating dataframes in a loop very slow

1 Answers1