Defensive conditions when JSON field is missing in API

Question

I am developing a small Python script in order to get the weather data from forecast.io Once I get the JSON document, I call a Class in order to create a new record to be saved in the database. The problem is that some fields (which are also attributes in my Class) are not always informed in the API so I must include some kind of defensive code or the script will break when field is not found.

I've found this answer of @Alex Martelli which seams pretty good: Reading from Python dict if key might not be present

If you want to do something different than using a default value (say, skip the printing completely when the key is absent), then you need a bit more structure, i.e., either:
for r in results:
    if 'key_name' in r:
        print r['key_name'] 
or
for r in results:
    try: print r['key_name']
    except KeyError: pass

But I am wondering if I must include an "if" or a "try" on every field I want to save or is there a prettier way to do this? (I want to save 27 fields and 27 "if" seems ugly)

This is the code I have so far:

from datetime import datetime

import tornado.web
import tornado.httpclient
from tornado import gen

from src.db.city import list_cities
from src.db.weather import Weather

from motorengine import *


@gen.coroutine
def forecastio_api():
    http_client = tornado.httpclient.AsyncHTTPClient()
    base_url = "https://api.forecast.io/forecast/APIKEY"
    city yield list_cities()
    for city in city:
        url = base_url + "/%s,%s" %(str(city.loc[0]), str(city.loc[1]))
        response = yield http_client.fetch(url)
        json = tornado.escape.json_decode(response.body)
        for day in json['daily']['data']:
            weather = Weather(city=city,
                              time = datetime.fromtimestamp(day['time']),
                              summary = day.get('summary'),
                              icon = day.get('icon'),
                              sunrise_time = datetime.fromtimestamp(day.get('sunriseTime')),
                              sunset_time = datetime.fromtimestamp(day.get('sunsetTime')),
                              moon_phase = day.get('moonPhase'),
                              precip_intensity = day.get('precipIntensity'),
                              precip_intensity_max = day.get('precipIntensityMax'),
                              precip_intensity_max_time = datetime.fromtimestamp(day.get('precipIntensityMaxTime')),
                              precip_probability = day.get('precipProbability'),
                              precip_type = day.get('precipType'),
                              temperature_min = day.get('temperatureMin'),
                              temperature_min_time = datetime.fromtimestamp(day.get('temperatureMinTime')),
                              temperature_max = day.get('temperatureMax'),
                              temperature_max_time = datetime.fromtimestamp(day.get('temperatureMaxTime')),
                              apparent_temperature_min = day.get('apparentTemperatureMin'),
                              apparent_temperature_min_time = datetime.fromtimestamp(day.get('apparentTemperatureMinTime')),
                              apparent_temperature_max = day.get('apparentTemperatureMax'),
                              apparent_temperature_max_time = datetime.fromtimestamp(day.get('apparentTemperatureMaxTime')),
                              dew_point = day.get('dewPoint'),
                              humidity = day.get('humidity'),
                              wind_speed = day.get('windSpeed'),
                              wind_bearing = day.get('windBearing'),
                              visibility = day.get('visibility'),
                              cloud_cover = day.get('cloudCover'),
                              pressure = day.get('pressure'),
                              ozone = day.get('ozone')
            )
            weather.create()


if __name__ == '__main__':
    io_loop = tornado.ioloop.IOLoop.instance()
    connect("DATABASE", host="localhost", port=27017, io_loop=io_loop)
    forecastio_api()
    io_loop.start()

and this is the Weather Class using Motornegine:

from tornado import gen
from motorengine import Document
from motorengine.fields import DateTimeField, DecimalField, ReferenceField, StringField

from src.db.city import City


class Weather(Document):
    __collection__ = 'weather'
    __lazy__ = False
    city = ReferenceField(reference_document_type=City)
    time = DateTimeField(required=True)
    summary = StringField()
    icon = StringField()
    sunrise_time = DateTimeField()
    sunset_time = DateTimeField()
    moon_phase = DecimalField(precision=2)
    precip_intensity = DecimalField(precision=4)
    precip_intensity_max = DecimalField(precision=4)
    precip_intensity_max_time = DateTimeField()
    precip_probability = DecimalField(precision=2)
    precip_type = StringField()
    temperature_min = DecimalField(precision=2)
    temperature_min_time = DateTimeField()
    temperature_max = DecimalField(precision=2)
    temperature_max_time = DateTimeField()
    apparent_temperature_min = DecimalField(precision=2)
    apparent_temperature_min_time = DateTimeField()
    apparent_temperature_max = DecimalField(precision=2)
    apparent_temperature_max_time = DateTimeField()
    dew_point = DecimalField(precision=2)
    humidity = DecimalField(precision=2)
    wind_speed = DecimalField(precision=2)
    wind_bearing = DecimalField(precision=2)
    visibility = DecimalField(precision=2)
    cloud_cover = DecimalField(precision=2)
    pressure = DecimalField(precision=2)
    ozone = DecimalField(precision=2)
    create_time = DateTimeField(auto_now_on_insert=True)

    @gen.coroutine
    def create(self):
        yield self.save()

What exactly do you want your code to do for each of the missing fields? — jwodder, Jan 07 '16 at 17:47
As none are mandatory I'll not include them when creating the Class record. The Class is used by Mongoengine, a Mongodb ODM — user3159821, Jan 07 '16 at 18:03
It might be helpful to provide a little more of your code. Are you going to be doing something different for each field if it doesn't exist? Or will it just be NULL or None? It will be helpful to see how your converting the json to a Class. — Brendan Abel, Jan 07 '16 at 18:07
isn't it json you're working on the, fields (results you want to save) are all grouped together like a python dictionary or do you have 27 json files? — danidee, Jan 07 '16 at 18:12

score 1 · Answer 1 · answered Jan 07 '16 at 17:50

You can check Schematics. This library helps you define objects that can be easily populated from dicts(you can easily turn json to python dict). It allows you to define validation rules on each property. The object will throw ModelValidationError error when some properties are missing or in the wrong format. Schematics allows you to add default values and a lot more nice stuff when you define your models.

Defensive conditions when JSON field is missing in API

1 Answers1