1

I have a process which I am able to loop through for values held in a list but it overwrites the final dataframe with each loop and I would like to append or concat the result of the loops into one dataframe. For example given below I can see 'dataframe' will populate initially with result of 'blah1', then when process finishes it has the result of 'blah2'

listtoloop = ['blah1','blah2']

for name in listtoloop:


    some process happens here resulting in

        dataframe = result of above process
Evan
  • 59
  • 2
  • 7
  • https://stackoverflow.com/questions/28669482/appending-pandas-dataframes-generated-in-a-for-loop – Evan Feb 03 '22 at 19:12

3 Answers3

6

The typical pattern used for this is to create a list of DataFrames, and only at the end of the loop, concatenate them into a single DataFrame. This is usually much faster than appending new rows to the DataFrame after each step, as you are not constructing a new DataFrame on every iteration.

Something like this should work:

listtoloop = ['blah1','blah2']
dfs = []
for name in listtoloop:
    # some process happens here resulting in
    # dataframe = result of above process
    dfs.append(dataframe)
final = pd.concat(dfs, ignore_index=True)
sjw
  • 6,213
  • 2
  • 24
  • 39
0

Put your results in a list and then append the list to the df making sure the list is in the same order as the df

listtoloop = ['blah1','blah2']
df = pd.DataFrame(columns="A","B")
for name in listtoloop:
    ## processes here

    to_append = [5, 6]
    df_length = len(df)
    df.loc[df_length] = to_append
stefan_aus_hannover
  • 1,777
  • 12
  • 13
0
data_you_need=pd.DataFrame()  
listtoloop = ['blah1','blah2']  
for name in listtoloop:     
    ##some process happens here resulting in          
    ##dataframe = result of above process    
    data_you_need=data_you_need.append(dataframe,ignore_index=True)
Evan
  • 59
  • 2
  • 7
  • The potential issue with this one is it results in a FutureWarning message. frame.append is deprecated and will be removed from pandas in a future version. However, you can append to a list no problem. Additionally, it's less performant. – Sean Richards Oct 25 '22 at 21:07