Dataframes from dictionnaries in nested lists - Python

Question

I have for example a nested list of 2 lists containing dictionaries like this : [[{},{},{},{}],[{},{},{},{}]]

[[{'Date': 'Dec 16 2020', 'Open': '37.00', 'High': '37.71', 'Low': '36.65', 'Close': '36.67', 'Adj Close': '36.67', 'Volume': '1840194'}, {'Date': 'Dec 15 2020', 'Open': '35.80', 'High': '36.88', 'Low': '35.63', 'Close': '36.79', 'Adj Close': '36.79', 'Volume': '1683816'}, {'Date': 'Dec 14 2020', 'Open': '35.48', 'High': '36.63', 'Low': '35.48', 'Close': '35.74', 'Adj Close': '35.74', 'Volume': '1808091'}, {'Date': 'Dec 11 2020', 'Open': '36.29', 'High': '36.63', 'Low': '34.96', 'Close': '35.08', 'Adj Close': '35.08', 'Volume': '1947461'}, {'Date': 'Dec 10 2020', 'Open': '36.51', 'High': '36.88', 'Low': '35.47', 'Close': '36.35', 'Adj Close': '36.35', 'Volume': '2207529'}],[{'Date': 'Sep 22 2020', : lose': '22.17', 'VolOpen': '22.00', 'High': '22.87', 'Low': '21.89', 'Close': '22.87', 'Adj Close': '22.87', 'Volume': '1997258'}, {'Date': 'Oct 05 2020', 'Open': '21.60', 'High': '
'Open': '21.01', 'High': '21.85', 'Low': '20.74', 'Close': '21.43', 'Adj Close': '21.43', 'Volume': '2064940'}, {'Date': 'Sep 21'O, 'Volume': '2064940 '20.72', 'Close': '21.19', 'Adj Close': '21.19', 'Volume': '2007302'}, {'Date': 'Oct 01 2020', 'Open': '22.35', 'High': '22.XMRClose': '20.88', 'Adj Close': '20.88', 'Volume': '3544256'}]]

I would like 2 dataframes something like :

          Date  Open  High  Low  Close Adj Close Volume 
   Dec 16 2020    37    37   36         36      1840194 
   Dec 15 2020     '     '    '          '            '
   Dec 14 2020     '     '    '          '            '
   Dec 13 2020     '     '    '          '            '

And I obviously can't use that :

df = pd.DataFrame(data)

with data as a list of dictionaries.

If I try to copy-paste your example data into Python, I get a `SyntaxError: invalid syntax`. Please fix the syntax errors (e.g. closing strings with quotes, use columns between keys and values in the dictionary, separate key-value pairs with commas) so that we can test your code using the provided data. — jfaccioni, Feb 10 '21 at 20:08
it's values that I get after crawling a website. That's also half of the question i'm asking. — LuckyFr, Feb 10 '21 at 20:12
In fact, I know how to create a datafram with : [{],{],{],{}] and with data like the sample I gave you but not like I asked. — LuckyFr, Feb 10 '21 at 20:16
Are you sure you copied the correct result into your question? For example, in one of the lists you have a dictionary which reads `{'Date': 'Sep 22 2020', : lose': '22.17' ...`. That's not valid Python - the second element is not a key-value pair, and the string is broken. it looks like there's a piece of the dictionary missing in there. If this is in fact your actual result after webscraping, then you need to sanitize it prior to parsing it into a dafaframe. — jfaccioni, Feb 10 '21 at 20:20
Doesn't matter how the value are, if I keep only one list and I create a dataframe, it's working like charm. but how to create different dataframe in a loop for every list inside the main list. — LuckyFr, Feb 10 '21 at 20:49
it does matter because we're trying to help you. And it's much harder to do that if you can't provide valid input data for your problem. — jfaccioni, Feb 10 '21 at 20:50
I understand, and I thank you for that, but i'm not asking about the values inside, I'm asking about creating different dataframe through the different lists — LuckyFr, Feb 10 '21 at 20:51
I know how to create a dataframe for dictionnaries in one list then how to do it for many list in one ? — LuckyFr, Feb 10 '21 at 20:52

Nicolò Gasparini · Accepted Answer · 2021-02-10T20:54:11.283

1

The solution is simply to flatten your list of lists, then you can pass it to pandas normally

l = [
  [{'date':1, 'value':'1'}, {'date':2, 'value':'2'}],
  [{'date':3, 'value':'3'}, {'date':4, 'value':'4'}]
]
df = pd.DataFrame([item for sublist in l for item in sublist])

Will get you

   date   value
0     1     '1'
1     2     '2'
2     3     '3'
3     4     '4'

Source

While if you need a list of pandas DataFrames you can simply iterate your main list like so:

df_list = []
for x in l:
     df_list.append(pd.DataFrame(x))

Or simply

df_list = [pd.DataFrame(x) for x in l]

edited Feb 10 '21 at 20:54

answered Feb 10 '21 at 20:17

Nicolò Gasparini

2,228
2
24
53

Hello Nicolo thank you for your answer, i would like it to be 2 dataframes separated could you edit your answer please ? The first list as one and the second list as another one. – LuckyFr Feb 10 '21 at 20:28
Using a loop and not declaring another df – LuckyFr Feb 10 '21 at 20:41
Just what I needed thank you for your patience and sorry if I was not clear at the beginning. – LuckyFr Feb 10 '21 at 21:05

Dataframes from dictionnaries in nested lists - Python

1 Answers1