(Python) Recursive Function Not Working on this Dicts of a List

Question

I have been implementing one of the answers of similar questions. The specific reference I used is this thread: How can I search for specific keys in this nested dictionary in Python?

My recursive function and its complimentary function look like following:

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
from selenium.webdriver import Chrome



def recursive_lookup(key, d):
    
    def _lookup(key, d):
        if key in d:
            return True

        for k, v in d.items():
            if isinstance(v, dict):
                result = _lookup(key, v)

                if result:
                    accumulator.append(d)
                else:
                    continue

    accumulator = []
    _lookup(key, d)

    return accumulator

def accumulator_appender(key, lst):
    total_accumulation = list()
    for d in lst:
        total_accumulation.append(recursive_lookup(key, d))
    return total_accumulation


cap = DesiredCapabilities.CHROME
cap['goog:loggingPrefs'] = {'performance': 'ALL'}
driver = Chrome(desired_capabilities=cap)
driver.implicitly_wait(1)
url = "http://www.testing.tradir.io/"
driver.get(url)
log = driver.get_log("performance")
filtered = accumulator_appender("status", log)
file_obj = open("network_log.txt", "w+")
file_obj.write(str(filtered))
file_obj.close()

My objective is to use Selenium to log all the performance-related information, specifically anything that has a status code from the API.

{"status": 200}

So, any dictionary that contains {"status": int} would show up in the network_log.txt file.

I used to just get the devtool opened on Chrome and rummaged through the 'Network' tab. I am trying to automate this process.

The example list of dictionaries look like this:

[
    {'level': 'INFO', 'message': '{
            "message": {
                "method": "Network.requestWillBeSent",
                "params": {
                    "documentURL": "http://www.testing.tradir.io/",
                    "frameId": "A4DE79DC87DD4C983D0328DFB9BACC3C",
                    "hasUserGesture": false,
                    "initiator": {
                        "type": "other"
                    },
                    "loaderId": "F0CFD89AC76F111C08A82A5AAC757B88",
                    "request": {
                        "headers": {
                            "Upgrade-Insecure-Requests": "1",
                            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
                        },
                        "initialPriority": "VeryHigh",
                        "method": "GET",
                        "mixedContentType": "none",
                        "referrerPolicy": "strict-origin-when-cross-origin",
                        "url": "http://www.testing.tradir.io/"
                    },
                    "requestId": "F0CFD89AC76F111C08A82A5AAC757B88",
                    "timestamp": 8536.750974,
                    "type": "Document",
                    "wallTime": 1616730707.544681
                }
            },
            "webview": "A4DE79DC87DD4C983D0328DFB9BACC3C"
        }', 'timestamp': 1616730707544
    },
    {'level': 'INFO', 'message': '{
            "message": {
                "method": "Network.requestWillBeSentExtraInfo",
                "params": {
                    "associatedCookies": [],
                    "headers": {
                        ":authority": "testing.tradir.io",
                        ":method": "GET",
                        ":path": "/",
                        ":scheme": "https",
                        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
                        "accept-encoding": "gzip, deflate, br",
                        "accept-language": "en-US,en;q=0.9",
                        "sec-fetch-dest": "document",
                        "sec-fetch-mode": "navigate",
                        "sec-fetch-site": "none",
                        "sec-fetch-user": "?1",
                        "upgrade-insecure-requests": "1",
                        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
                    },
                    "requestId": "F0CFD89AC76F111C08A82A5AAC757B88"
                }
            },
            "webview": "A4DE79DC87DD4C983D0328DFB9BACC3C"
        }', 'timestamp': 1616730707776
    },
    {'level': 'INFO', 'message': '{
            "message": {
                "method": "Network.responseReceived",
                "params": {
                    "frameId": "A4DE79DC87DD4C983D0328DFB9BACC3C",
                    "loaderId": "F0CFD89AC76F111C08A82A5AAC757B88",
                    "requestId": "6002.4",
                    "response": {
                        "connectionId": 37,
                        "connectionReused": true,
                        "encodedDataLength": 321,
                        "fromDiskCache": false,
                        "fromPrefetchCache": false,
                        "fromServiceWorker": false,
                        "headers": {
                            "age": "63022",
                            "content-encoding": "br",
                            "content-type": "text/css",
                            "date": "Thu, 25 Mar 2021 10:21:27 GMT",
                            "etag": "W/\\"701a2b685f224b1e25c58a4c10d5a13f\\"",
                            "last-modified": "Thu, 25 Mar 2021 10:19:53 GMT",
                            "server": "AmazonS3",
                            "vary": "Accept-Encoding",
                            "via": "1.1 b7930036df6e15271fcc3588a0cac37e.cloudfront.net (CloudFront)",
                            "x-amz-cf-id": "_Vd_UwSRKrOZNRHBDy5GSvH3mM3ulZw58piXL7VcMtab5qsz60olbQ==",
                            "x-amz-cf-pop": "ICN54-C3",
                            "x-cache": "Hit from cloudfront"
                        },
                        "mimeType": "text/css",
                        "protocol": "h2",
                        "remoteIPAddress": "54.230.62.119",
                        "remotePort": 443,
                        "responseTime": 1.616730707865601e+12,
                        "securityDetails": {
                            "certificateId": 0,
                            "certificateTransparencyCompliance": "compliant",
                            "cipher": "AES_128_GCM",
                            "issuer": "Amazon",
                            "keyExchange": "",
                            "keyExchangeGroup": "X25519",
                            "protocol": "TLS 1.3",
                            "sanList": [
                                "*.tradir.io",
                                "tradir.io"
                            ],
                            "signedCertificateTimestampList": [
                                {
                                    "hashAlgorithm": "SHA-256",
                                    "logDescription":"Google \'Argon2021\' log",
                                    "logId": "F65C942FD1773022145418083094568EE34D131933BFDF0C2F200BCC4EF164E3",
                                    "origin": "Embedded in certificate",
                                    "signatureAlgorithm": "ECDSA",
                                    "signatureData": "30450220625A5D61392E8D6F920ADC2B8C1ACB9D3ED04D5E2F93FD2420F59D042F4BA318022100F49A5CDA9E66E93EDE8DD4DADB6C3389A2C7B73AE9BFED9B00FF88384C3412E4",
                                    "status": "Verified",
                                    "timestamp": 1.605496612931e+12
                                },
                                {
                                    "hashAlgorithm": "SHA-256",
                                    "logDescription": "DigiCert Yeti2021 Log",
                                    "logId": "5CDC4392FEE6AB4544B15E9AD456E61037FBD5FA47DCA17394B25EE6F6C70ECA",
                                    "origin": "Embedded in certificate",
                                    "signatureAlgorithm": "ECDSA",
                                    "signatureData": "304402200C3519BDF015FE6B56EB8031D6F10B8CA1D2D9C5F57C062D38A19D846BC830A0022074CAD3A6EC99578F974A77E30525BE411CEF457CC6BBE5BEBDB5CD29CC6CC0DF",
                                    "status": "Verified",
                                    "timestamp": 1.605496613006e+12
                                }
                            ],
                            "subjectName": "*.tradir.io",
                            "validFrom": 1605484800,
                            "validTo": 1639612799
                        },
                        "securityState": "secure",
                        "status": 200,
                        "statusText": "",
                        "timing": {
                            "connectEnd": -1,
                            "connectStart": -1,
                            "dnsEnd": -1,
                            "dnsStart": -1,
                            "proxyEnd": -1,
                            "proxyStart": -1,
                            "pushEnd": 0,
                            "pushStart": 0,
                            "receiveHeadersEnd": 44.454,
                            "requestTime": 8537.027532,
                            "sendEnd": 0.723,
                            "sendStart": 0.167,
                            "sslEnd": -1,
                            "sslStart": -1,
                            "workerFetchStart": -1,
                            "workerReady": -1,
                            "workerRespondWithSettled": -1,
                            "workerStart": -1
                        },
                        "url": "https://testing.tradir.io/static/css/14.b11059c9.chunk.css"
                    },
                    "timestamp": 8537.072595,
                    "type": "Stylesheet"
                }
            },
            "webview": "A4DE79DC87DD4C983D0328DFB9BACC3C"
        }', 'timestamp': 1616730707866
    },

The problem is that the final result always comes like this:

network_log.txt:

[[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []]

Please provide the expected [MRE - Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example). Show where the intermediate results deviate from the ones you expect. We should be able to paste a single block of your code into file, run it, and reproduce your problem. This also lets us test any suggestions in your context. We expect a minimal working example of the problem, including appropriate code to trace the internal operation. — Prune, Mar 29 '21 at 03:05
Please add your test driver, reduce your data to the minimum needed to exhibit the error, and put in the canonical tracing `print` statements. Note that you have several run-time errors waiting to happen with your misuse of `accumulator`. You seem to assume that it's a global variable, and you aren't following its data flow properly. — Prune, Mar 29 '21 at 03:09

score 1 · Accepted Answer · answered Mar 29 '21 at 03:14

1

You are not dealing with nested dictionaries, or not yet.

Your data structure is:

[
 {'level': string_0 ,'message': string_1 ,'timestamp': int},
 ...
]

Convert string_1 to dict fisrt, and you are good to go. ( using eval() or json.loads())

answered Mar 29 '21 at 03:14

e_yi

66
2

score 1 · Answer 2 · answered Mar 29 '21 at 03:46

If we assume a recursive dictionary structure where the status can be reported at any level, the following code will yield a flat list of all the status codes reported in the dictionary structure. The structure to hold the status codes is constructed and returned up the chain of recursive calls. No need for a global variable, like the accumulator.

If a "deep" list is preferred to mimic the depth of the structure where the status was found, then replace lookups += rec_ret with lookups.append(rec_ret).

l0_f_dict = dict(status=200, nothing=False)
l0_nf_dict = dict(notfound=True, nothing=False)
l1_f_dict = dict(status=301, level=1, node=l0_nf_dict)
l1_nf_dict = dict(notfound=True, level=1, node=l0_nf_dict)
l1_f1_dict = dict(notfound=True, level=1, node=l0_f_dict)
l1_f2_dict = dict(status=302, level=1, node=l0_f_dict)
l2_f3_dict = dict(status=303, level=2, node=l1_f2_dict)

def rec_lookup(key, d):
    if not isinstance(d, dict):
        return []
    lookups = []
    if key in d:
        lookups.append(d[key])
    for v in d.values():
        if isinstance(v, dict):
            rec_ret = rec_lookup(key, v)
            lookups += rec_ret
    return lookups

print(rec_lookup('status', l0_f_dict))
print(rec_lookup('status', l0_nf_dict))
print(rec_lookup('status', l1_f_dict))
print(rec_lookup('status', l1_nf_dict))
print(rec_lookup('status', l1_f1_dict))
print(rec_lookup('status', l1_f2_dict))
print(rec_lookup('status', l2_f3_dict))
# [200]
# []
# [301]
# []
# [200]
# [302, 200]
# [303, 302, 200]

(Python) Recursive Function Not Working on this Dicts of a List

2 Answers2