Iterate over a list of lists of dictionaries

Question

I need to return a specific value from every dictionary. In this case, I need to return the action performed by the antivirus, whose id is either id:5 or id:6. The action is either File Cleaned or Quarantine Successfully. Another problem is that the lists do not have the same length. Most have 6 dictionaries in them, but sometimes you can find a list with 1 dict or even 8 dicts.

Here is the list:

data = [
    [
        {
            "id": 1,
            "type": "detection_name",
            "field": "malName",
            "value": "HKTL_AMMYYADMIN",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 2,
            "type": "file_sha1",
            "field": "fileHash",
            "value": "",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 3,
            "type": "filename",
            "field": "fileName",
            "value": "D:\\PAMH\\DATA\\LABORTATORY\\AMMYY_Admin.exe",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 4,
            "type": "fullpath",
            "field": "fullPath",
            "value": "D:\\PAMH\\DATA\\LABORTATORY\\AMMYY_Admin.exe",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 5,
            "type": "text",
            "field": "actResult",
            "value": "File cleaned",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 6,
            "type": "text",
            "field": "scanType",
            "value": "Real-time Scan",
            "provenance": [
                "Alert"
            ]
        }
    ],
    [
        {
            "id": 1,
            "type": "command_line",
            "field": "objectCmd",
            "value": "c:\\SW\\nmap\\nmap.exe  -sS -P0 92.168.155.83 -p 22",
            "provenance": [
                "Alert"
            ]
        }
    ],
    [
        {
            "id": 1,
            "type": "detection_name",
            "field": "malName",
            "value": "HackTool.Win32.PortScan.SWO",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 2,
            "type": "file_sha1",
            "field": "fileHash",
            "value": "",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 3,
            "type": "filename",
            "field": "fileName",
            "value": "E:\\EXCHANGE\\John Smith\\Documents\\Advanced_Port_Scanner_2.5.3869.exe",
           "provenance": [
                "Alert"
            ]
        },
        {
            "id": 4,
            "type": "filename",
            "field": "fileName",
            "value": "H:\\home\\john.smith\\Documents\\Advanced_Port_Scanner_2.5.3869.exe",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 5,
            "type": "fullpath",
            "field": "fullPath",
            "value": "H:\\home\\john.smith\\Documents\\Advanced_Port_Scanner_2.5.3869.exe",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 6,
            "type": "text",
            "field": "actResult",
            "value": "File cleaned",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 7,
            "type": "text",
            "field": "actResult",
            "value": "File cleaned",
            "provenance": [
                "Alert"
            ]
        },
        {
            "id": 8,
            "type": "text",
            "field": "scanType",
            "value": "Scheduled Scan",
            "provenance": [
                "Alert"
            ]
        }
    ]
]

I'm able to return values from a list of dicts, but so far everything I tried with a list of lists of dicts hasn't worked.

Expected Output:

"File cleaned", "-", "File cleaned"

You could [flatten the list first](https://stackoverflow.com/questions/952914/how-do-i-make-a-flat-list-out-of-a-list-of-lists), and then you have a list of dicts. — larsks, May 01 '23 at 17:16
"I'm able to return values from a list of dicts..." You have the right idea by solving a simpler problem. I recommend that you put that code into a function. Then you can use that function to solve the bigger problem. If you still need help with that, please show the code that you have writen so far. — Code-Apprentice, May 01 '23 at 17:22

score 2 · Answer 1 · answered May 01 '23 at 17:19

2

You have to create a flat list based on an existing 2D list. Here is example using List Comprehension

flatten_list = [item['value'] for sublist in data for item in sublist]
print(flatten_list)

answered May 01 '23 at 17:19

KSs

420
3
9

Thank you. While this works in some way, it still doesn't produce the exact output I need. It does flatten all items, but then it's impossible to tell which ones I need. – OverflowStack May 02 '23 at 05:03

score 1 · Accepted Answer · answered May 01 '23 at 21:04

1

You could make a recursive generator that will drill down nested lists to pick the dictionaries of interest:

def getActions(D,ids):
    if isinstance(D,dict):
        if D["id"] in ids and D["field"] == "actResult":
            yield D["value"]
    else:
        for d in D:
            yield from getActions(d,ids)

print(*getActions(data,{5,6}),sep=" - ")

File cleaned - File cleaned

answered May 01 '23 at 21:04

Alain T.

40,517
4
31
51

Thank you. This works great. It's outside the scope of my knowledge though. I need to take a look at generators more in depth. In the meantime, I need the output to be in form of a list. Despite my tinkering with it, I was not able to do it. How can I have a list in the end? – OverflowStack May 02 '23 at 05:00
`list(getActions(data,{5,6}))` – Alain T. May 02 '23 at 05:20
Hey, so I'm having an issue with the recursion script. When I run in in small controlled example, such as the one I provided in the question, it works perfectly. But when I make a call to the server, I get "RecursionError: maximum recursion depth exceeded while calling a Python object" since there are too many objects. Any way around it? – OverflowStack May 03 '23 at 15:04
You either have a very deeply nested structure (hundreds of levels deep) or there is one of the nested list that references itself or one of its containers at a higher level – Alain T. May 03 '23 at 16:40
Mind if I share the logfile? – OverflowStack May 03 '23 at 21:33

Iterate over a list of lists of dictionaries

2 Answers2