Append dictionary to pandas dataframe in a loop

Question

I have a requirement to create a dictionary within a loop and append them to a pandas data frame with matching key name of dict and column name of data frame. The key value pairs of dictionary in each iteration could be different. An empty pandas data frame df_podcast have been defined at the beginning with all possible keys in the dictionary.

Below is the sample of a code which is not completed yet

df_podcast=pd.DataFrame(columns=podcast_cols)

podcast_dict={}
for j in range(len(podcast[0])):
    if podcast[0][j].tag=="key":
        podcast_dict[podcast[0][j].text]=podcast[0][j+1].text
### Have to append dict to pandas df ############

I have append podcast_dict to df_podcast. Podcast is actually a list of lists, here I'm just considering only 1st row of the list

Last after loop is possible use `df = pd.DataFrame(podcast_dict)` ? — jezrael, May 13 '17 at 20:03
The answer here is very helpful: https://stackoverflow.com/questions/40973687/create-new-dataframe-in-pandas-with-dynamic-names-also-add-new-column — Sankara Viswanathan, Feb 13 '18 at 18:08

score 16 · Accepted Answer · edited Aug 04 '18 at 07:51

16

You need:

df  = pd.DataFrame([podcast_dict], columns=podcast_dict.keys())
df_podcast = pd.concat([df_podcast, df], axis =0).reset_index()

edited Aug 04 '18 at 07:51

Sociopath

13,068
19
47
75

answered Oct 23 '17 at 01:57

xingpei Pang

1,185
11
15

score 14 · Answer 2 · answered Aug 18 '19 at 13:44

If you want to simply append new data from a created dictionary within a loop to an existening Dataframe:

df = pd.DataFrame()
for i in range(n):
    dict_new = dict(i)
    df = df.append(dict_new, ignore_index=True)
print(df)

NOTE: As long as the keys in your created dictionary are the same, appending it to an existing dataframe shouldn't be cumbersome. Source

score 5 · Answer 3 · answered May 13 '17 at 20:42

5

IIUC:

What you need to do is to build your dictionary with your loop, then at then end of your loop, you can use your dictionary to create a dataframe with:

df1  = pd.DataFrame(podcast_dict)

And append using pd.concat:

df_podcast = pd.concat([df_podcast, df1])

answered May 13 '17 at 20:42

Scott Boston

147,308
15
139
187

Thanks Scott for the answer. But the key names , no of key value pairs etc in my dictionary will be different in each iteration. I want to insert the value to the data frame where key of the dictionary matches with column name of the data frame. for example :- if the first iteration dictionary has only two item { "Track id":100 , "Play count":10 }. I should insert values only in Track id and play count columns of first row of the data frame. – Leo May 15 '17 at 06:15
Did you try it? Pandas does automatic index alignment, and column are an index. I did try witb some dummy data to verify and order doesnt matter. – Scott Boston May 15 '17 at 06:26

Append dictionary to pandas dataframe in a loop

3 Answers3

Linked