How to fetch JSON data from API, format / encode / write to a file?

Question

I need to fetch some data from a weather API, extract certain info and send it to std. output (in my case this is the console/terminal; I am playing around with python API scripting and do not yet have a web site/app do show fetched data).

Example Python code from the API provider (simple to understand and works):

import urllib2
import json

API_KEY='mysuperawesomekey'


f = urllib2.urlopen('http://api.wunderground.com/api/' + API_KEY + '/geolookup/conditions/q/IA/Cedar_Rapids.json')

json_string = f.read()
parsed_json = json.loads(json_string)
location = parsed_json['location']['city']
temp_f = parsed_json['current_observation']['temp_f']

print "Current temperature in %s is: %s" % (location, temp_f)

f.close()

Since I am on a free plan, I don't want to use up my API requests but rather fetch the JSON data, save it to a file and use in another script to practice. I found a couple of solutions here on StackOverflow but neither seem to work completely (nice formatting of the file):

Attempt 1. to save the fetched data, add. to the orig. code above:

import io
import codecs

with open('data.json', 'w') as outfile:
json.dump(json_string, outfile, indent=4, sort_keys=True, separators=(",", ':'))

Attempt 2:

with codecs.open('data.json', 'w', 'utf8') as jasonfile:
  jasonfile.write(json.dumps(parsed_json, sort_keys = True, ensure_ascii=False))

Both of my attempts work ("kind of") as I do get a .json file. But, upon inspecting it in my editor (Atom) I am seeing this (first few soft-break lines):

Output:

"\n{\n  \"response\": {\n  \"version\":\"0.1\",\n\"termsofService\":\"http://www.wunderground.com/weather/api/d/terms.html\",\n  \"features\": {\n  \"geolookup\": 1\n  ,\n  \"conditions\": 1\n  }\n\t}\n\t\t,\t\"location\": {\n\t\t\"type\":\"CITY\",\n\t\t\"country\":\"US\",\n\t\t\"country_iso3166\":\"US\",\n\t\t\"country_name\":\"USA\",\n\t\t\"state\":\"IA\",\n\t\t\"city\":\"Cedar Rapids\",\n\t\t\"tz_short\":\"CDT\",\n\t\t\"tz_long\":\"America/Chicago\",\n\t\t\"lat\":\"41.97171021\",\n\t\t\"lon\":\"-91.65871429\",\n\t\t\"zip\":\...

It is all on one line, with newlines and tabs showing. I have a couple of questions:

How can I check what format / kind of data-type API is returning? (FOr example is it just a raw string which needs to be JSON-iffied?)
Is my JSON not written in a human readable format because of encoding, delimiters fault?
Line json_string = f.read() from example code seems to "mimic" working with a "real" internal file object, is this correct?

Many thanks!

Your first question - check HTTP Headers, if it has application/json, it's trying to serve json :) — iScrE4m, Sep 08 '16 at 14:11
Bingo! It's serving a json file! :) SO, they fetch the file with `urlopen` into `f`, then `read()` this file, here it gets muddy... Should I save `f` to my filesystem od this `json_string` which is the result of reading the fetched `f`? Or even further down the line, maybe after `parsed_json = json.loads()`? — Alexander Starbuck, Sep 08 '16 at 14:17
seems like you need to decode your json,http://stackoverflow.com/questions/12965203/how-to-get-json-from-webpage-into-python-script — Pavneet_Singh, Sep 08 '16 at 14:18
If you are storing the json just to mock a server and practice json on it, don't worry about how it looks, only worry if you can load it. Json as a format escapes almost everything iirc — iScrE4m, Sep 08 '16 at 14:20
Cool. I CAN load it and all, but I'd like to re-format it nicely so I can use it as a human too :), inspect bits etc. Or, I need to be able to print it "nicely" to the console... — Alexander Starbuck, Sep 08 '16 at 14:22

score 1 · Answer 1 · answered Sep 08 '16 at 14:33

Seems that it was quite a simple solution. In my original code, I was saving a "non-parsed" variable to a file:

import urllib2
import json

API_KEY='key'

f = urllib2.urlopen('http://api.wunderground.com/api/' 
+ API_KEY + '/geolookup/conditions/q/IA/Cedar_Rapids.json')

# Saving the below variable into a .json file gives all the JSON data
# on a single line
json_string = f.read()

with open('data.json', 'w') as outfile:
    json.dump(json_string, outfile, indent=4, sort_keys=True, separators=(",", ':'))

which, of course, resulted in a file with JSON data on a single line. What I should have been doing is saving the parsed variable:

import urllib2
import json

API_KEY='key'

f = urllib2.urlopen('http://api.wunderground.com/api/' + API_KEY + '/geolookup/conditions/q/IA/Cedar_Rapids.json')

json_string = f.read()

# THIS one is correct!
parsed_json = json.loads(json_string)

with open('data.json', 'w') as outfile:
  json.dump(parsed_json, outfile, indent=4, sort_keys=True, separators=(",", ':'))

How to fetch JSON data from API, format / encode / write to a file?

1 Answers1