Unable to parse streetAddresses out of a json response

Question

I've written a script to fetch street-addresses out of a json response but I can't reach that portion. The structure seems a bit complicated to me.

Link to the json content

A chunk of the response containing street-addresses:

[[44189579,25735941,-80305513,"$640K",1,0,0,0,["$640K",4,3.0,1963,false,null,"6,000 sqft lot","","ForSale","For Sale by Owner",0,{"zpid": 44189579,"streetAddress": "6811 SW 38th St","zipcode": "33155","city": "Miami","state": "FL","latitude": 25.735941,"longitude": -80.305513,"price": 640000.0,"dateSold": 0,"bathrooms": 3.0,"bedrooms": 4.0,"livingArea": 1963.0,"yearBuilt": -1,"lotSize": 6000.0,"homeType": "SINGLE_FAMILY",

I've tried so far:

import requests

url = "https://www.zillow.com/search/GetResults.htm?spt=homes&status=100000&lt=111101&ht=100000&pr=,&mp=,&bd=0%2C&ba=0%2C&sf=,&lot=0%2C&yr=,&singlestory=0&hoa=0%2C&pho=0&pets=0&parking=0&laundry=0&income-restricted=0&fr-bldg=0&condo-bldg=0&furnished-apartments=0&cheap-apartments=0&studio-apartments=0&pnd=0&red=0&zso=0&days=any&ds=all&pmf=0&pf=0&sch=100111&zoom=11&rect=-80419407,25712692,-80201741,25759392&p=1&sort=days&search=map&rid=72458&rt=7&listright=true&isMapSearch=1&zoom=11"

res = requests.get(url,headers={"User-Agent":"Mozilla/5.0"})
print(res.json()['map']['properties'])

Expected results:

6811 SW 38th St

and so on.....

Possible duplicate of [Parse JSON in Python](https://stackoverflow.com/questions/7771011/parse-json-in-python) — CDspace, Dec 21 '18 at 23:41

score 2 · Answer 1 · answered Dec 21 '18 at 22:46

What you need is a function to recursively traverse the returned object and look for your key in all dictionaries it encounters.

def traverse(source, target_key: str, storage: list):
    if isinstance(source, dict):
        for k, v in source.items():
            if k == target_key:
                storage.append(v)
            elif isinstance(v, (dict, list)):
                traverse(v, target_key, storage)
    elif isinstance(source, list):
        for item in source:
            if isinstance(item, (dict, list)):
                traverse(item, target_key, storage)

key = "streetAddress"
source = [[44189579, 25735941, -80305513, "$640K", 1, 0, 0, 0,
           ["$640K", 4, 3.0, 1963, False, None, "6,000 sqft lot", "", "ForSale", "For Sale by Owner", 0,
            {"zpid": 44189579, "streetAddress": "6811 SW 38th St", "zipcode": "33155", "city": "Miami", "state": "FL",
             "latitude": 25.735941, "longitude": -80.305513, "price": 640000.0, "dateSold": 0, "bathrooms": 3.0,
             "bedrooms": 4.0, "livingArea": 1963.0, "yearBuilt": -1, "lotSize": 6000.0, "homeType": "SINGLE_FAMILY"}]]]    

storage = []
traverse(source, key, storage)
print(storage)

OUTPUT:

['6811 SW 38th St']

score 2 · Accepted Answer · answered Dec 21 '18 at 23:35

@ebro42 is correct, the best way to get the data is to recursively traverse the json data object. I think his suggestion could be improved by not relying on a passed callback container and instead making it a generator which you iterate.

from typing import Iterable

def get_by_key(key: str, collection: Iterable):
    if isinstance(collection, dict):
        for k, v in collection.items():
            if k == key:
                yield v
            elif isinstance(v, Iterable) and not isinstance(v, str):
                yield from get_by_key(key, v)
    elif isinstance(collection, Iterable) and not isinstance(collection, str):
        for i in collection:
            yield from get_by_key(key, i)

for address in get_by_key('streetAddress', res.json()):
    print(address)

score 0 · Answer 3 · answered Jan 14 '19 at 12:28

This does the job.

import requests
from jsonpath_ng.ext import parse
url = "https://www.zillow.com/search/GetResults.htm?spt=homes&status=100000&lt=111101&ht=100000&pr=,&mp=,&bd=0%2C&ba=0%2C&sf=,&lot=0%2C&yr=,&singlestory=0&hoa=0%2C&pho=0&pets=0&parking=0&laundry=0&income-restricted=0&fr-bldg=0&condo-bldg=0&furnished-apartments=0&cheap-apartments=0&studio-apartments=0&pnd=0&red=0&zso=0&days=any&ds=all&pmf=0&pf=0&sch=100111&zoom=11&rect=-80419407,25712692,-80201741,25759392&p=1&sort=days&search=map&rid=72458&rt=7&listright=true&isMapSearch=1&zoom=11"

res = requests.get(url,headers={"User-Agent":"Mozilla/5.0"})
properties = res.json()['map']['properties']

for p in properties:
    found = parse("$..streetAddress").find(p)
    print(found[0].value)

Unable to parse streetAddresses out of a json response

3 Answers3