JSON sub for loop produces KeyError, but key exists

Question

I'm trying to add the JSON output below into a dictionary, to be saved into a SQL database.

{'Parkirisca': [
    {    
    'ID_Parkirisca': 2,
    'zasedenost': {
                   'Cas': '2016-10-08 13:17:00',
                   'Cas_timestamp': 1475925420,
                   'ID_ParkiriscaNC': 9,
                   'P_kratkotrajniki': 350
                  }
    }
]}

I am currently using the following code to add the value to a dictionary:

import scraperwiki
import json
import requests
import datetime
import time
from pprint import pprint

html = requests.get("http://opendata.si/promet/parkirisca/lpt/")

data = json.loads(html.text)

for carpark in data['Parkirisca']:

    zas = carpark['zasedenost']
    free_spaces = zas.get('P_kratkotrajniki')
    last_updated = zas.get('Cas_timestamp')

    parking_type = carpark.get('ID_Parkirisca')
    if parking_type == "Avtomatizirano":
        is_automatic = "Yes"
    else:
        is_automatic = "No"

    scraped = datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')

    savetodb = {
    'scraped': scraped,
    'id': carpark.get("ID_Parkirisca"),
    'total_spaces': carpark.get("St_mest"),
    'free_spaces': free_spaces,
    'last_updated': last_updated,
    'is_automatic': is_automatic,
    'lon': carpark.get("KoordinataX_wgs"), 
    'lat': carpark.get("KoordinataY_wgs")
    }

    unique_keys = ['id']

    pprint savetodb

However when I run this, it gets stuck at for zas in carpark["zasedenost"] and outputs the following error:

Traceback (most recent call last):
File "./code/scraper", line 17, in <module>
    for zas in carpark["zasedenost"]:
KeyError: 'zasedenost'

I've been led to believe that zas is in fact now a string, rather than a dictionary, but I'm new to Python and JSON, so don't know what to search for to get a solution. I've also searched here on Stack Overflow for KeyErrror when key exist questions, but they didn't help, and I believe that this might be due to the fact that's a sub for loop.

Update: Now, when I swapped the double quotes for single quotes, I get the following error:

Traceback (most recent call last):
File "./code/scraper", line 17, in <module>
    free_spaces = zas.get('P_kratkotrajniki')
AttributeError: 'unicode' object has no attribute 'get'

Put a `print(carpark)` above the `for` loop so you can see what's in that variable. — user94559, Oct 08 '16 at 11:45
As an aside, the "JSON" you shared is not valid JSON. (It uses single quotes.) — user94559, Oct 08 '16 at 11:46
`print(repr(carpark))` might be even better. That will help you spot if it is indeed a `dict` as expected. — user94559, Oct 08 '16 at 11:47
I also don't think you mean to iterate over `carpark["zasedenost"]` (since it's a dictionary). You probably just want `zas = carpark["zasedenost"]`. — user94559, Oct 08 '16 at 11:50
In general, it would help if you shared actual runnable code that reproduces the issue. If I use `data = {'Parkirisca': ... }` and then run the rest of your code, I do *not* get a `KeyError` on the line you indicated. — user94559, Oct 08 '16 at 11:51
Please provide a Minimal, Complete, and Verifiable example. http://stackoverflow.com/help/mcve — Zixian Cai, Oct 08 '16 at 12:10
I just swapped out all of the double quotes for single quotes, and now I get a completely different error: `AttributeError: 'unicode' object has no attribute 'get'` on this line: `free_spaces = zas.get('P_kratkotrajniki')`. I'll add the full script with the URL in my question now — tristanojbacon, Oct 08 '16 at 12:16
That error is due to the issue I pointed out a couple comments above. You want `zas = carpark["zasedenost"]` rather than `for zas in carpark["zasedenost"]`. — user94559, Oct 08 '16 at 12:18
But that then gives me the same key error as originally reported! — tristanojbacon, Oct 08 '16 at 12:23
Did you add the print statement as I suggested? That would show you that the dictionary with the 'id' 6 doesn't contain 'zasedenost'. So when you get to that element in the array, you see the `KeyError` telling you that key doesn't exist. — user94559, Oct 08 '16 at 12:30
Oh, but you'll need to use `zas = carpark["zasedenost"]` as I said. And you'll have to fix `pprint savetodb` to be something with valid Python syntax. (Maybe `pprint(savetodb)` after an `import pprint from pprint`?) Otherwise the code won't run far enough to get to the array element that's missing that key. — user94559, Oct 08 '16 at 12:33
My mistake, I pasted an older version (pre-your suggestion). Have updated the code. Also included my imports. I'm using Python 2.x, so no parenthesis necessary on the `pprint` as far as I'm aware? So that code above should produce a KeyError if you run it (it's exactly the same as mine now!) — tristanojbacon, Oct 08 '16 at 12:36
See my answer. I still don't know how you're managing to get `pprint savetodb` to work... that certainly doesn't work in Python 2.7.x. Maybe you're using something even earlier? — user94559, Oct 08 '16 at 12:40
For future questions you might ask, please include actual runnable code. E.g., in your question, the very first line uses `requests` which hasn't been defined (or imported). You're forcing people who try to help you to first use trial-and-error to figure out the right imports. — user94559, Oct 08 '16 at 12:42

score 2 · Accepted Answer · answered Oct 08 '16 at 12:39

I fixed up your code:

Added required imports.
Fixed the pprint savetodb line which isn't valid Python.
Didn't try to iterate over carpark['zasedenost'].

I then added another pprint statement in the for loop to see what's in carpark when the KeyError occurs. From there, the error is clear. (Not all the elements in the array in your JSON contain the 'zasedenost' key.)

Here's the code I used:

import datetime
import json
from pprint import pprint
import time

import requests

html = requests.get("http://opendata.si/promet/parkirisca/lpt/")

data = json.loads(html.text)

for carpark in data['Parkirisca']:
    pprint(carpark)
    zas = carpark['zasedenost']
    free_spaces = zas.get('P_kratkotrajniki')
    last_updated = zas.get('Cas_timestamp')

    parking_type = carpark.get('ID_Parkirisca')
    if parking_type == "Avtomatizirano":
        is_automatic = "Yes"
    else:
        is_automatic = "No"

    scraped = datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')

    savetodb = {
        'scraped': scraped,
        'id': carpark.get("ID_Parkirisca"),
        'total_spaces': carpark.get("St_mest"),
        'free_spaces': free_spaces,
        'last_updated': last_updated,
        'is_automatic': is_automatic,
        'lon': carpark.get("KoordinataX_wgs"), 
        'lat': carpark.get("KoordinataY_wgs")
    }

    unique_keys = ['id']

    pprint(savetodb)

And here's the output on the iteration where the KeyError occurs:

{u'A_St_Mest': None,
 u'Cena_dan_Eur': None,
 u'Cena_mesecna_Eur': None,
 u'Cena_splosno': None,
 u'Cena_ura_Eur': None,
 u'ID_Parkirisca': 7,
 u'ID_ParkiriscaNC': 72,
 u'Ime': u'P+R Studenec',
 u'Invalidi_St_mest': 9,
 u'KoordinataX': 466947,
 u'KoordinataX_wgs': 14.567929171694901,
 u'KoordinataY': 101247,
 u'KoordinataY_wgs': 46.05457609543313,
 u'Opis': u'2,40 \u20ac /dan',
 u'St_mest': 187,
 u'Tip_parkirisca': None,
 u'U_delovnik': u'24 ur (ponedeljek - petek)',
 u'U_sobota': None,
 u'U_splosno': None,
 u'Upravljalec': u'JP LPT d.o.o.'}
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    zas = carpark['zasedenost']
KeyError: 'zasedenost'

As you can see, the error is quite accurate. There's no key 'zasedenost' in the dictionary. If you look through your JSON, you'll see that's true for a number of the elements in that array.

I'd suggest a fix, but I don't know what you want to do in the case where this dictionary key is absent. Perhaps you want something like this:

zas = carpark.get('zasedenost')
if zas is not None:
    free_spaces = zas.get('P_kratkotrajniki')
    last_updated = zas.get('Cas_timestamp')
else:
    free_spaces = None
    last_updated = None

JSON sub for loop produces KeyError, but key exists

1 Answers1