Appending multiple dictionaries to Pandas dataframe: Error DataFrame constructor not properly called?

Question

I am trying to get a dictionary to pandas dataframe. I am having trouble with a few things. I tried the following

data = {'applicableMargin': '12.50', 'marketType': 'N', 'totalBuyQuantity': '1,14,514', 'buyPrice1': '1,546.30', 'dayLow': '1,541.20', 'symbol': 'ACC', 'cm_adj_low_dt': '23-MAR-17', 'open': '1,571.50', 'sellPrice2': '1,547.85', 'sellPrice4': '1,547.95', 'cm_ffm': '13,249.84', 'buyPrice3': '1,546.00', 'css_status_desc': 'Listed', 'ndStartDate': '-', 'buyQuantity1': '43', 'totalTradedValue': '1,468.42', 'surv_indicator': '-', 'recordDate': '26-JUL-17', 'secDate': '16MAR2018', 'faceValue': '10.00', 'totalTradedVolume': '94,384', 'pricebandlower': '1,411.20', 'sellQuantity4': '16', 'averagePrice': '1,555.79', 'buyPrice2': '1,546.05', 'totalSellQuantity': '84,873', 'closePrice': '0.00', 'buyPrice4': '1,545.90', 'extremeLossMargin': '5.00', 'isinCode': 'INE012A01025', 'buyQuantity4': '48', 'sellPrice3': '1,547.90', 'bcEndDate': '-', 'buyQuantity5': '27', 'indexVar': '-', 'purpose': 'INTERIM DIVIDEND - RS 11/- PER SHARE', 'sellQuantity5': '286', 'series': 'EQ', 'low52': '1,380.40', 'dayHigh': '1,573.70', 'pricebandupper': '1,724.70', 'basePrice': '1,567.95', 'lastPrice': '1,546.05', 'sellQuantity2': '32', 'deliveryToTradedQuantity': '50.45', 'high52': '1,869.95', 'cm_adj_high_dt': '13-SEP-17', 'sellQuantity1': '67', 'buyQuantity2': '155', 'isExDateFlag': False, 'quantityTraded': '2,53,481', 'previousClose': '1,567.95', 'securityVar': '5.74', 'bcStartDate': '-', 'sellQuantity3': '25', 'ndEndDate': '-', 'buyQuantity3': '31', 'companyName': 'ACC Limited', 'sellPrice1': '1,547.65', 'adhocMargin': '-', 'sellPrice5': '1,548.00', 'change': '-21.90', 'exDate': '25-JUL-17', 'varMargin': '7.50', 'pChange': '-1.40', 'buyPrice5': '1,545.85', 'priceBand': 'No Band'}



pd_cols = []
for i in data:
    pd_cols.append(i)

#fut_data = pd.DataFrame()
#fut_data.columns = pd_cols
fut_data = pd.DataFrame(data.items(), columns=pd_cols)

This gives error:

Traceback (most recent call last): File "", line 1, in File "C:\Python34\lib\site-packages\pandas\core\frame.py", line 345, in >init raise PandasError('DataFrame constructor not properly called!') pandas.core.common.PandasError: DataFrame constructor not properly called!

After this I will have a bunch of more dict which will have the same columns. I want to add them all to the same database.

Thanks in advance.

Try this: `fut_data = pd.DataFrame.from_dict(data, orient='index')` or its transpose `fut_data = pd.DataFrame.from_dict(data, orient='index').T` — jpp, Mar 19 '18 at 10:28
Possible duplicate of https://stackoverflow.com/questions/17839973/construct-pandas-dataframe-from-values-in-variables — swathis, Mar 19 '18 at 10:37

jpp · Accepted Answer · 2018-03-19T10:51:56.527

2

This works for me. Since this errors for you, you may have a copy-paste error.

fut_data = pd.DataFrame.from_dict(data, orient='index').T

print(fut_data)

#   applicableMargin marketType totalBuyQuantity buyPrice1    dayLow symbol  \
# 0            12.50          N         1,14,514  1,546.30  1,541.20    ACC   

#   cm_adj_low_dt      open sellPrice2 sellPrice4    ...     companyName  \
# 0     23-MAR-17  1,571.50   1,547.85   1,547.95    ...     ACC Limited   

#   buyPrice5 priceBand  
# 0  1,545.85   No Band  

# [1 rows x 67 columns]

You can append as follows:

df = pd.DataFrame.from_dict(data, orient='index').T

df = df.append(pd.DataFrame.from_dict(data2, orient='index').T)

Here data2 is another similar dictionary.

edited Mar 19 '18 at 10:51

answered Mar 19 '18 at 10:31

jpp

159,742
34
281
339

Thanks, I guess the error was my mistake some where. If I try to add a different `dict` to the dataframe(having the same columns/keys) it overwrites the exisiting rows, how do I add a new row? – Sid Mar 19 '18 at 10:40
Another quick question though, when I add more rows using dicts. All of them have index set as `0`, how would I change this? Thanks for all the help. – Sid Mar 19 '18 at 11:12
1

After you have added all your dictionaries, you can do `df = df.reset_index(drop=True)`. – jpp Mar 19 '18 at 11:12

user667489 · Answer 2 · 2018-03-19T12:16:17.497

Does this give you the output you want?

import pandas as pd
data = {'applicableMargin': '12.50', 'marketType': 'N', 'totalBuyQuantity': '1,14,514', 'buyPrice1': '1,546.30', 'dayLow': '1,541.20', 'symbol': 'ACC', 'cm_adj_low_dt': '23-MAR-17', 'open': '1,571.50', 'sellPrice2': '1,547.85', 'sellPrice4': '1,547.95', 'cm_ffm': '13,249.84', 'buyPrice3': '1,546.00', 'css_status_desc': 'Listed', 'ndStartDate': '-', 'buyQuantity1': '43', 'totalTradedValue': '1,468.42', 'surv_indicator': '-', 'recordDate': '26-JUL-17', 'secDate': '16MAR2018', 'faceValue': '10.00', 'totalTradedVolume': '94,384', 'pricebandlower': '1,411.20', 'sellQuantity4': '16', 'averagePrice': '1,555.79', 'buyPrice2': '1,546.05', 'totalSellQuantity': '84,873', 'closePrice': '0.00', 'buyPrice4': '1,545.90', 'extremeLossMargin': '5.00', 'isinCode': 'INE012A01025', 'buyQuantity4': '48', 'sellPrice3': '1,547.90', 'bcEndDate': '-', 'buyQuantity5': '27', 'indexVar': '-', 'purpose': 'INTERIM DIVIDEND - RS 11/- PER SHARE', 'sellQuantity5': '286', 'series': 'EQ', 'low52': '1,380.40', 'dayHigh': '1,573.70', 'pricebandupper': '1,724.70', 'basePrice': '1,567.95', 'lastPrice': '1,546.05', 'sellQuantity2': '32', 'deliveryToTradedQuantity': '50.45', 'high52': '1,869.95', 'cm_adj_high_dt': '13-SEP-17', 'sellQuantity1': '67', 'buyQuantity2': '155', 'isExDateFlag': False, 'quantityTraded': '2,53,481', 'previousClose': '1,567.95', 'securityVar': '5.74', 'bcStartDate': '-', 'sellQuantity3': '25', 'ndEndDate': '-', 'buyQuantity3': '31', 'companyName': 'ACC Limited', 'sellPrice1': '1,547.65', 'adhocMargin': '-', 'sellPrice5': '1,548.00', 'change': '-21.90', 'exDate': '25-JUL-17', 'varMargin': '7.50', 'pChange': '-1.40', 'buyPrice5': '1,545.85', 'priceBand': 'No Band'}
df = pd.DataFrame.from_dict([data])

print(df.iloc[:,:5])

When I run the above I get a 1-row dataframe:

  adhocMargin applicableMargin averagePrice basePrice bcEndDate
0           -            12.50     1,555.79  1,567.95         -

If you have multiple similar dicts, place them all in a list like so:

df = pd.DataFrame.from_dict([data1,data2])

That results in a dataframe with one row per dict.

Not sure if `pd.DataFrame.from_dict([data])` would need `data` as a list. It looks like it would work otherwise, though. — Jon, Mar 19 '18 at 10:33
Thanks for the answer. It seems there was some error from my side. The command works now. If I try to add a different dict to the dataframe(having the same columns/keys) it overwrites the exisiting rows, how do I add a new row? — Sid, Mar 19 '18 at 10:41
@Jon - if I don't use a list I get `ValueError: If using all scalar values, you must pass an index` — user667489, Mar 19 '18 at 12:17

shubham · Answer 3 · 2018-03-19T10:48:09.600

0

Although the previous answers are better you can try this absurd solution if it works for you :

    fut_data = pd.DataFrame(data,index=[0])

for adding more rows, you may try:

    fut_data1 = pd.DataFrame(data1,index=[1])
    fut_data.append(fut_data1)

or

    fut_data1 = pd.DataFrame(data1,index=[i]) #where i is a loop variable
    fut_data.append(fut_data1)

edited Mar 19 '18 at 10:48

answered Mar 19 '18 at 10:42

shubham

576
4
7

Appending multiple dictionaries to Pandas dataframe: Error DataFrame constructor not properly called?

3 Answers3