skipping a json key if does not exist

Question

I'm running the following:

for server in server_list:
    for item in required_fields:
        print item, eval(item)

There is a possibility that some keys may not exist, but worse it's represented on a parent key not the one I'm scanning for.

So I'm scanning the json for the following key:

server['server_management']['server_total_cost_of_ownership']['description']

Which doesn't exist but it's actually the parent that is null:

server['server_management']['server_total_cost_of_ownership']

How do I write my code to account for this? It's not giving a key error. Right now I get the following traceback:

Traceback (most recent call last):
  File "C:/projects/blah/scripts/test.py", line 29, in <module>
    print item, eval(item)
  File "<string>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'

Full code:

import csv
import json
import os
import requests
import sys

required_fields = ["server['server_name']","server['server_info']['asset_type']['display_name']",
                   "server['asset_status']['display_name']", "server['record_owner']['group_name']",
                   "server['server_management']['server_total_cost_of_ownership']['description']",
                   "server['server_management']['primary_business_owner']['name']",
                   "server['environment']['display_name']", "server['is_virtual']",
                   "server['managed_by']['display_name']", "server['server_info']['billable_ibm']",
                   "server['server_info']['billing_sub_type']['display_name']",
                   "server['server_info']['serial_number']", "server['location']['display_name']",
                   "server['inception_date']", "server['server_info']['decommission_date']" ]

# Query API for all servers
def get_servers_info():
    servers_info = requests.get('url')
    return servers_info.json()

def get_server_info(sid):
    server_info = requests.get('url')
    return server_info.json()

server_list = get_servers_info()
for server in server_list:
    for item in required_fields:
        print item, eval(item)

Not seeing the purpose of `eval` here. Can you explain the code you're using and why you're using it better? Don't need `eval` to access a dict key. — Two-Bit Alchemist, Jul 07 '16 at 14:57
Always post a complete code that we can run and reproduce the error. — chepner, Jul 07 '16 at 15:03
... and you probably *shouldn't* be using `eval` at all. What you simply need to be doing is adding `if` statements to detect that a key is present. — Mike Robinson, Jul 07 '16 at 15:03
That is not complete; we don't know what your call to `requests.get` is returning. However, `required_fields` should *absolutely* be a simple list of strings `required_fields = [ "server_name", "server_info", ...]`. Then you don't need `eval` at all; you just write `for item in required_fields: print server_list[server][required_fields]`. — chepner, Jul 07 '16 at 15:12
`eval` and `exec` should generally be avoided because they can be a security risk. For details, please see [Eval really is dangerous](http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html) by SO veteran Ned Batchelder. — PM 2Ring, Jul 07 '16 at 15:19
the json being returned contains `server['server_management']['server_total_cost_ov_ownership']` but it shows null so the key I'm looking for which is ['description'] isn't in there. — whoisearth, Jul 07 '16 at 15:20
Use the `json` module to parse the JSON; don't assume it is valid Python code that can be evaluated just because the syntax is similar. — chepner, Jul 07 '16 at 15:28
@chepner - so how should I be parsing to only grab the keys in my `required_fields`? — whoisearth, Jul 07 '16 at 15:45
@chepner The content of the list is *not* json, so using `json` to parse it won't work. For example, `"server['server_name']"` is a python *statement*, not json *data*. — MisterMiyagi, Jul 08 '16 at 07:40
It's not the list that should be parsed; it's the response from the server. The list should contain keys only, not snippets of JSON. — chepner, Jul 08 '16 at 11:48

score 2 · Accepted Answer · answered Jul 07 '16 at 15:48

In fact you should avoid eval. After the json load since you know the key name, you can use a list to go deeper in the tree.

server['server_management']['primary_business_owner']['name']" => ["server_management', 'primary_business_owner', 'name']

Here a snippet for a json validation against a list of required fields.

data={
    "d": {
        "p":{
            "r":[
                "test"
            ]
        }
    },
    "a": 3
}


def _get_attr(dict_, attrs):
    try:
        src = attrs[:]
        root = attrs.pop(0)
        node = dict_[root]
        null = object()
        for i, attr in enumerate(attrs[:]):
            try:
                node = node.get(attr, null)
            except AttributeError:
                node = null
            if node is null:
                # i+2 pop and last element
                raise ValueError("%s not present (level %s)" % (attr, '->'.join(src[: i+2])))
        return node
    except KeyError:
        raise ValueError("%s not present" % root)

# assume list of required field
reqs = [
    ["d", "p", "r"],
    ["d"],
    ["k"],
    ["d", "p", "r", "e"],
]

for req in reqs:
    try:
        _get_attr(data, req)
    except ValueError as E:
        print(E)
# prints
# k not present
# e not present (level d->p->r->e)

This seems like serious overkill to just catching a `TypeError` or testing whether an element is `None`. — MisterMiyagi, Jul 08 '16 at 16:31

score 1 · Answer 2 · answered Jul 07 '16 at 15:49

Ignoring the context of the code and not understanding the use of eval here, the way to do this is to use .get() and seed it with reasonable defaults.

For example:

server['server_management']['server_total_cost_of_ownership']['description']

Can be:

server.get('server_management', {}).get('server_total_cost_of_ownership', {}).get('description', '')

Then if any of the keys do not exist you will always get back an empty description ''.

score 0 · Answer 3 · edited May 23 '17 at 11:58

Your problem here is totally unrelated to using eval[1]. The exception you get is the same as if the code would have been there directly. What you are running (via eval) is:

a = server['server_management']
b = a['server_total_cost_of_ownership']
c = b['description']

Yet, b is None, so resolving it to c will fail. Like a KeyError, you can also catch a TypeError:

for server in server_list:
  for item in required_fields:
    try:
      print item, eval(item)
    except TypeError:
      print("Guess you're lucky you didn't include a fork bomb in your own code to eval.")

You may of course alternatively pass, print the offending item, open a browser to some page or do whatever error handling is appropriate given your input data.

[1] While not bickering around, I've made a new answer that works without eval. You can use precisely the same error handling:

for server in server_list:
  for item in required_fields:
  value = server
  for key in parse_fields(field):
    try:
      value = value[key]
    except TypeError:
      print("Remember Kiddo: Eval is Evil!")
      break
  else:  # for: else: triggers only if no break was issued
    print item, value

skipping a json key if does not exist

3 Answers3