what is wrong with json response

Question

json response which is working correctly :

obj = urllib.urlopen("http://www.omdbapi.com/?t=Fight Club")
response_str = obj.read()
response_json = simplejson.loads(response_str)

above code making the json request which looks like :

{
    "Title":"Fight Club",
    "Year":"1999",
    "Rated":"R",
    "Released":"15 Oct 1999",
     ......
    "Response":"True"
}

so i can sleep now... but

json response which is not working correctly :

obj = urllib.urlopen("https://api.stackexchange.com/2.1/answers?order=desc&sort=activity&site=stackoverflow")
response_str = obj.read()
response_json = simplejson.loads(response_str)

above code making the json request which looks like :

{

    "items": [
        {
            "question_id": 18384375,
            "answer_id": 18388044,
            "creation_date": 1377195687,
            "last_activity_date": 1377195687,
            "score": 0,
            "is_accepted": false,
            "owner": {
                "user_id": 1745001,
                "display_name": "Ed Morton",
                "reputation": 10453,
                "user_type": "registered",
                "profile_image": "https://www.gravatar.com/avatar/99a3ebae89496eb16afe453aae97f5be?s=128&d=identicon&r=PG",
                "link": "https://stackoverflow.com/users/1745001/ed-morton"
            }
        },
        {
            "question_id": 18387447,
            "answer_id": 18388040,
            "creation_date": 1377195667,
            "last_activity_date": 1377195667,
            "score": 0,
            "is_accepted": false,
            "owner": {
                "user_id": 2494429,
                "display_name": "kpark91",
                "reputation": 140,
                "user_type": "registered",
                "profile_image": "https://www.gravatar.com/avatar/d903a03e7c5b6d9b21ff598c632de575?s=128&d=identicon&r=PG",
                "link": "https://stackoverflow.com/users/2494429/kpark91"
            }
        }

       ]
}

returns

JSONDecodeError at /
No JSON object could be decoded: line 1 column 0 (char 0)

instead of using simplejson i tried json which gave following error :

ValueError at /
No JSON object could be decoded

what i tried and failed:

I tried my questions's answers on stackoverflow having same problem but none of them gave clear cut solution although every one helped me in some way

1) i checked whether json content encoding is correct or not

>>obj.info()

Content-Type: application/json; 
charset=utf-8
Content-Length: 2398

2) i decoded the response_str in utf-8

json.loads(response_str).decode("utf-8")

3) i used jsonlint to check format of json response

Parse error on line 1:

^
Expecting '{', '['

surprisingly following link of rescued my sleep. JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Basically i was using exact code to get json response in both cases , but whats wrong with the second json response, only difference i noticed that the structure of second json response was different from first one.

please provide explanation to understand the issue.

The server is returning the response in gzipped encoding. E.g., on the command line `curl 'https://api.stackexchange.com/2.1/answers?order=desc&sort=activity&site=stackoverflow' | gunzip` will show you the JSON that you're expecting. I'm not a Python user, but apparently the `requests` library that some of the answers are mentioning handle the gzipped content correctly. — Joshua Taylor, Aug 22 '13 at 18:59
There are some relevant questions about urllib[2] and gzipped content on SO: [Does python urllib2 will automaticly uncompress gzip data from fetch webpage](http://stackoverflow.com/q/3947120/1281433); [urlopen trouble while trying to download a gzip file](http://stackoverflow.com/q/18146389/1281433); [Why I got messy characters while opening url using urllib2?](http://stackoverflow.com/q/7231280/1281433); [Convert gzipped data fetched by urllib2 to HTML](http://stackoverflow.com/q/1704754/1281433); [Python urllib2 parse html problem](http://stackoverflow.com/q/6899273/1281433). — Joshua Taylor, Aug 22 '13 at 19:06
Hey, added a picture-video gif to help you out. If you choose to work with requests, you will get better sleep, as alecxe has promised! :P — Games Brainiac, Aug 22 '13 at 19:29

Games Brainiac · Accepted Answer · 2013-08-22T20:02:56.353

Let me give you some advice. Using urllib is great and all, but you will run into a lot of trouble. So, let me save you the pain, and just lead you to requests. Its the best way to deal with url requests.

This short IDLE session will demonstrate:

>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}

To install it, just go to your command prompt and pip install requets. requets is free and open source, and its written in pure python so the installation is simple.

Here is a video to demonstrate: enter image description here

Your particular problem: enter image description here

score 4 · Answer 2 · answered Aug 22 '13 at 18:56

One way to have a good sleep is to use requests:

import requests
import simplejson

response = requests.get("https://api.stackexchange.com/2.1/answers?order=desc&sort=activity&site=stackoverflow")
response_json = simplejson.loads(response.text)
print response_json

No errors, just works.

score 0 · Answer 3 · edited May 23 '17 at 12:05

Other answers are mentioning the requests library, but based on some other answers on StackOverflow, it seems like you could also just modify your code to include the GZip decompression. Based on this answer, I think this would be:

import StringIO, gzip 

obj = urllib.urlopen("https://api.stackexchange.com/2.1/answersorder=desc&sort=activity&site=stackoverflow")
data = StringIO.StringIO(obj.read())
gzipper = gzip.GzipFile(fileobj=data)
response_json = simplejson.loads(gzipper.read())

what is wrong with json response

3 Answers3