Appending to an empty data frame in Pandas using a for loop

Question

I am writing because I am having an issue with a for loop which fills a dataframe when it is empty. Unfortunately, the posts Filling empty python dataframe using loops, Appending to an empty data frame in Pandas?, Creating an empty Pandas DataFrame, then filling it? did not help me to solve it.

My attempt aims, first, at finding the empty dataframes in the list "listDataframe" and then, wants to fill them with some chosen columns. I believe my code is clearer than my explanation. What I can't do is to save the new dataframe using its original name. Here my attempt:

for k,j in zip(listOwner,listDataframe):
for y in j:
    if y.empty:
        data = pd.DataFrame({"Event Date": list_test_2, "Site Group Name" : k, "Impressions" : 0})
        y = pd.concat([data,y])
        #y = y.append(data)

where "listOwner", "listDataframe" and "list_test_2" are, respectively, given by:

listOwner = ['OWNER ONE', 'OWNER TWO', 'OWNER THREE', 'OWNER FOUR']
listDataframe = [df_a,df_b,df_c,df_d]

with

df_a = [df_ap_1, df_di_1, df_er_diret_1, df_er_s_1]
df_b = [df_ap_2, df_di_2, df_er_diret_2, df_er_s_2]
df_c = [df_ap_3, df_di_3, df_er_diret_3, df_er_s_3]
df_d = [df_ap_4, df_di_4, df_er_diret_4, df_er_s_4]

and

list_test_2 = []
for i in range(1,8):
    f = (datetime.today() - timedelta(days=i)).date()
    list_test_2.append(datetime.combine(f, datetime.min.time()))

The empty dataframe were df_ap_1 and df_ap_3. After running the above lines (using both concat and append) if I call these two dataframes they are still empty. Any idea why that happens and how to overcome this issue?

UPDATE

In order to avoid both append and concat, I tried to use the coming attempt (again with no success).

for k,j in zip(listOwner,listDataframe):
    for y in j:
        if y.empty:
            y = pd.DataFrame({"Event Date": list_test_2, "Site Group Name" : k, "Impressions" : 0})

The two desired result should be:

where the first dataframe should be called df_ap_1 while the second one df_ap_3.

Thanks in advance.

Drigo

[Never call DataFrame.append or pd.concat inside a for-loop. It leads to quadratic copying.](https://stackoverflow.com/a/36489724/1422451) — Parfait, Feb 18 '19 at 21:42
What is list_test_2? Also, that's a lot of dataframes. Any chance you could combine all 16 of your data frames into one big data frame? Maybe you could do so by adding a column that would have value of 'a', 'b', 'c', or 'd' and then also a column that would have a value of 'ap', 'di', 'er_direct' or 'er_s'. — LetEpsilonBeLessThanZero, Feb 18 '19 at 21:53
Thanks for answering. I have updated my attempt. I had forgotten the definition of list_test_2. Actually they are a lot of dataframes but with only fews rows each. — fdrigo, Feb 18 '19 at 21:58

LetEpsilonBeLessThanZero · Answer 1 · 2019-02-19T00:02:41.470

Here's a way to do it:

import pandas as pd

columns = ['Event Date', 'Site Group Name', 'Impressions']
df_ap_1 = pd.DataFrame(columns=columns) #empty dataframe
df_di_1 = pd.DataFrame(columns=columns) #empty dataframe
df_ap_2 = pd.DataFrame({'Event Date':[1], 'Site Group Name':[2], 'Impressions': [3]}) #non-empty dataframe
df_di_2 = pd.DataFrame(columns=columns) #empty dataframe

df_a = [df_ap_1, df_di_1]
df_b = [df_ap_2, df_di_2]
listDataframe = [df_a,df_b]

list_test_2 = 'foo'
listOwner = ['OWNER ONE', 'OWNER TWO']

def appendOwner(df, owner, list_test_2):
    #appends a row to a dataframe for each row in listOwner
    new_row = {'Event Date': list_test_2,
               'Site Group Name': owner,
               'Impressions': 0,
               }
    df.loc[len(df)] = new_row

for owner, dfList in zip(listOwner, listDataframe):
    for df in dfList:
        if df.empty:
            appendOwner(df, owner, list_test_2)

print(listDataframe)

You can use the appendOwner function to append the rows from listOwner to an empty dataframe.

Hi @LetEpsilonBeLessThanZero, thanks for your effort. I am afraid this is not what I was looking for. Please see the photo I have just attached to this post. I would like the owner to depend on the listDataframe. That's why I used zip. Depending where the empty dataframe is in the listDataframe, it has a different owner. I hope I explained myself. :) — fdrigo, Feb 18 '19 at 22:49

Appending to an empty data frame in Pandas using a for loop

1 Answers1