Getting values from complicated json content

Question

So basically I am web scraping a site and for that I need "id" of all the location from a complicated json content:

https://hilfe.diakonie.de/hilfe-vor-ort/marker-json.php?ersteller=&kategorie=0&text=&n=54.14365551060835&e=19.704533281249986&s=48.00384435890099&w=1.2035567187499874&zoom=20000

I have tried dict.items method but i am getting only 2 values that are in start of the dict and then a list start:

res = requests.get(url).json()

json_obj = res.items()

for key, value in json_obj:
    if key == "id":
        print(value)

json = {
    "count": 17652,
    "items": [
        {
            "lat": 51.17450581504055,
            "lng": 10.007757153036533,
            "count": 17652,
            "north": 54.1425475,
            "east": 15.0019,
            "south": 48.0039543,
            "west": 5.952813,
            "elements": [
                {
                    "id": "5836de61a581c245ae48806b",
                    "o": 'null'
                },
                {
                    "id": "5836de62a581c245ae48814b",
                    "o": 'null'
                },
                {
                    "id": "5836de57a581c245ae487944",
                    "o": 'null'
                },
                {
                    "id": "5836de64a581c245ae4882a8",
                    "o": 'null'
                },
                {
                    "id": "5836de54a581c245ae48772a",
                    "o": 'null'
                },
                {
                    "id": "5836de57a581c245ae487945",
                    "o": 'null'
                }
            ]
        }
    ]
}

Please include a minimal amount of the data itself - formatted as code. Please read [mre]. — wwii, Dec 20 '20 at 17:38

score 1 · Answer 1 · answered Dec 20 '20 at 17:24

The id attribute is nested inside arrays in the elements attribute of objects, which are in turn nested inside an array in the items attribute of the response. Use a list comprehension with 2 loops to extract them:

res = requests.get(url).json()

ids = [ele["id"] for v in res["items"] for ele in v["elements"]]
for id in ids:
    print(id)

score 1 · Answer 2 · answered Dec 20 '20 at 17:25

The JSON consists of a root dictionary with two key-value pairs. One is count, which is an integer, the other is items, which maps to a list of a single item. This item is a dictionary, which has several key-value pairs, one of which is elements, which is a list of dictionaries, each containing an id:

import requests

url = "https://hilfe.diakonie.de/hilfe-vor-ort/marker-json.php?ersteller=&kategorie=0&text=&n=54.14365551060835&e=19.704533281249986&s=48.00384435890099&w=1.2035567187499874&zoom=20000"

response = requests.get(url)
response.raise_for_status()

elements = response.json()["items"][0]["elements"]

# print only the first ten ids
for element in elements[:10]:
    print(element["id"])

Output:

5836de61a581c245ae48806b
5836de62a581c245ae48814b
5836de57a581c245ae487944
5836de64a581c245ae4882a8
5836de54a581c245ae48772a
5836de57a581c245ae487945
5836de61a581c245ae48806c
5836de64a581c245ae4882aa
5836de57a581c245ae487947
5836de62a581c245ae48814d
>>>

yeah i came to the same logic right after asking this question and i worked. — Muhammad Ahmad, Dec 21 '20 at 08:29

score 0 · Answer 3 · answered Dec 20 '20 at 18:00

Same thing but different - using operator.itemgetter.

items = operator.itemgetter('items')
elements = operator.itemgetter('elements')
eyedees = operator.itemgetter('id')

data = elements(items(json)[0])
stuff = map(eyedees,data)
print(list(stuff))

Uses json from the example in the question.

Getting values from complicated json content

3 Answers3