Search for key, value pairs in complex nested dictionary/list

Question

I have a list made of responses to an API request. The result is the JSON with a lot of nested dictionaries and lists like this:

{'Siri': {'ServiceDelivery': {'ResponseTimestamp': '2020-12-13T08:16:31Z',
   'ProducerRef': 'IVTR_HET',
   'ResponseMessageIdentifier': 'IVTR_HET:ResponseMessage::df80db10-3cd3-4d8d-9cee-27ec6c716630:LOC:',
   'StopMonitoringDelivery': [{'ResponseTimestamp': '2020-12-13T08:16:31Z',
     'Version': '2.0',
     'Status': 'true',
     'MonitoredStopVisit': [{'RecordedAtTime': '2020-12-13T08:16:27.623Z',
       'ItemIdentifier': 'RATP:Item::STIF.StopPoint.Q.40944.A.7:LOC',
       'MonitoringRef': {'value': 'STIF:StopPoint:Q:40944:'},
       'MonitoredVehicleJourney': {'lineRef': {'value': 'STIF:Line::C01742:'},
        'framedVehicleJourneyRef': {'dataFrameRef': {'value': 'any'},
         'datedVehicleJourneyRef': 'RATP:VehicleJourney::RA.A.0916.1:LOC'},
        'journeyPatternName': {'value': 'TEDI'},
        'vehicleMode': [],
        'publishedLineName': [{'value': 'St-Germain-en-Laye. Poissy. Cergy. / Boissy-St-Léger. Marne-la-Vallée.'}],
        'directionName': [{'value': 'St-Germain-en-Laye. Poissy. Cergy.'}],
        'serviceFeatureRef': [],
        'vehicleFeatureRef': [],
        'originName': [],
        'originShortName': [],
        'destinationDisplayAtOrigin': [],
        'via': [],
        'destinationRef': {'value': 'STIF:StopPoint:Q:411359:'},
        'destinationName': [{'value': 'Poissy'}],
        'destinationShortName': [],
        'originDisplayAtDestination': [],
        'vehicleJourneyName': [],
        'journeyNote': [{'value': 'TEDI'}],
        'facilityConditionElement': [],
        'situationRef': [],
        'monitored': True,
        'monitoringError': [],
        'progressStatus': [],
        'trainBlockPart': [],
        'additionalVehicleJourneyRef': [],
        'monitoredCall': {'stopPointRef': {'value': 'STIF:StopPoint:Q:40944:'},
         'stopPointName': [{'value': 'Torcy'}],
         'vehicleAtStop': True,
         'originDisplay': [],
         'destinationDisplay': [],
         'callNote': [],
         'facilityConditionElement': [],
         'situationRef': [],
         'arrivalOperatorRefs': [],
         'expectedDepartureTime': '2020-12-13T08:16:00Z',
         'departurePlatformName': {'value': '2'},
         'departureOperatorRefs': []}}},

I need an efficient way to extract the value pairs corresponding to, for example, the expectedDepartureTime key that's lying somewhere hidden in that maze of nested dictionaries. It has to be efficient because the formatting from the source site isn't very stable so different stations (it's transit data) will have different keys for the same value, e.g. ExpectedDepartureTime or expecteddepartureTime and I might need to try a lot of different things.

P.S.: I tried the solution by @hexerei software here Find all occurrences of a key in nested dictionaries and lists but it just gave me a

<generator object gen_dict_extract at 0x00000200632C8948>

as the output. How do I get the values out of this?

That old answer from 2015 is almost certainly only compatible with Python 2. Did you read the comments underneath? Several of them contain hints for Python 3. — tripleee, Dec 13 '20 at 12:25

score 0 · Accepted Answer · answered Dec 13 '20 at 12:23

0

This function should return a list containing key-value pairs regardless of the depth:

def accumulate_keys(dct): # returns all the key-value pairs
    key_list = []
    def accumulate_keys_recursive(dct): # will accumulate key-value pairs in key_list
        for key in dct.keys():
            if isinstance(dct[key], dict):
                accumulate_keys_recursive(dct[key])
            else:
                key_list.append((key, dct[key]))
    accumulate_keys_recursive(dct)
    return key_list
print(accumulate_keys(dct))

answered Dec 13 '20 at 12:23

abe

957
5
10

Thanks! I tried modifying it to deal with the nested lists by adding elif isinstance(dct[key], list): accumulate_keys_recursive(dct[key][0]) But then I got an 'IndexError: list index out of range'. How can I get an indexerror on dict[key][0]? – AndreMwendwa Dec 13 '20 at 14:37
You should only call the function `accumulate_keys_recursive()` when the instance is a `dict`, because it assumes that there is another nested dictionary. If you clarify what you want to do when a value is a list or nested list, maybe I can be more of help – abe Dec 13 '20 at 15:33
Thanks a lot for all the help so far. I want the function to basically do the same thing, return a series of tuples containing the key-value pairs. Just that it should be able to go inside a list in case there's a list nested inside a dictionary and return the key value pairs e.g. `[{'1': 'dog', '2': 'cow'}]` should return `('1', 'dog'), ('2', 'cow')` – AndreMwendwa Dec 13 '20 at 18:39
Then you should check if a value is a list by `isinstance(dct[key], list)`, then inside that if statement you should also check if `isinstance(dct[key][0], dict)`. If this is also `True`, you should call the recursive function again, i.e. `accumulate_keys_recursive(dct[key][0])` – abe Dec 13 '20 at 20:06
I've added the two checks that the value is a list and that inside it is a dict `elif isinstance(dct[key], list) and isinstance (dct[key][0], dict): accumulate_keys_recursive(dct[key][0])` But I'm still getting my output as `IndexError: list index out of range` – AndreMwendwa Dec 13 '20 at 21:16

Search for key, value pairs in complex nested dictionary/list

1 Answers1