0

I'm trying to convert a 4 dimensional list to a pandas dataframe. I have a solution that uses a triple nested for loop to accomplish this, but it is highly unoptimised - I feel there must be a faster solution for this. The code I've been using is below:

import pandas as pd

master_df = pd.DataFrame(columns=('a1', 'a2', 'intersection', 'similarity'))

for i in master_list[0:2]:
    for x in i:
        for y in x:        
            t = [y[0], y[1], repr(y[2]), y[3]]
            master_df.loc[-1] = t
            master_df.index = master_df.index + 1
            master_df = master_df.sort_index()

This is a slice of master_list I've been attempting to insert into a dataframe.

master_list = [[[['residential property 42 holywell hill st. albans east of england al1 1bx',
'gnd flr 38 holywell hill st albans herts al1 1bx',
{'1bx', 'al1', 'albans', 'hill', 'holywell'},
0.5809767086589066],
['residential property 42 holywell hill st. albans east of england al1 1bx',
'62 holywell hill st albans herts al1 1bx',
{'1bx', 'al1', 'albans', 'hill', 'holywell'},
0.62250400597525191]]],
[[['aitchisons 2 holywell hill st. albans east of england al1 1bz',
'22 holywell hill st albans herts al1 1bz',
{'1bz', 'al1', 'albans', 'hill', 'holywell'},
0.64696827426453596],
['aitchisons 2 holywell hill st. albans east of england al1 1bz',
'24 holywell hill st albans herts al1 1bz',
{'1bz', 'al1', 'albans', 'hill', 'holywell'},
0.64660269146725069],
['aitchisons 2 holywell hill st. albans east of england al1 1bz',
'26 holywell hill st albans herts al1 1bz',
{'1bz', 'al1', 'albans', 'hill', 'holywell'},
0.64617599950794757],
['aitchisons 2 holywell hill st. albans east of england al1 1bz',
'20 holywell hill st albans herts al1 1bz',
{'1bz', 'al1', 'albans', 'hill', 'holywell'},
0.64798547824947428]]]]

Does anybody have any advice as to convert this 4d list into a pandas dataframe in a more... pythonic way?

Sam

Sam Comber
  • 1,237
  • 1
  • 15
  • 35

1 Answers1

3

Here is a solution:

  • Flatten master_list
  • Use repr for dictionaries (I don't think you really need this...)
  • Reshape the values to have 4 columns

The code:

def flatten(container):
    for i in container:
        if isinstance(i, (list,tuple)):
            for j in flatten(i):
                yield j
        else:
            yield i

def fix_dict(x):
    return repr(x) if isinstance(x, dict) else x

all_values = list(flatten(master_list))
all_values = [fix_dict(val) for val in all_values]

master_df = pd.DataFrame(np.reshape(all_values, (-1, 4)), columns = ['a1', 'a2', 'intersection', 'similarity'])

Which gives the expected output.

FLab
  • 7,136
  • 5
  • 36
  • 69