1

I have been working with a couple of web APIs but this one has me perplexed. I can't work out what im going wrong.

this code works on one api, but not this one.

response = urllib.request.urlopen(self.query_base)
reader = codecs.getreader("utf-8")
obj = json.load(reader(response))
return obj

this gives me the following errror

UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f602' in position 4096: character maps to <undefined>

I have tried:

response = urllib.request.urlopen(self.query_base)
obj = json.load(response.decode("utf-8"))
return obj

which gives:

AttributeError: 'HTTPResponse' object has no attribute 'decode'

and,

response = urllib.request.urlopen(self.query_base).read()
obj = json.load(response)
return obj

which gives

AttributeError: 'bytes' object has no attribute 'read'

and,

response = urllib.request.urlopen(self.query_base)
obj = json.load(response)

which gives

TypeError: the JSON object must be str, not 'bytes'

along with maany other combinations of things I have found in other similar threads on here

I don't remember ever having this problem before, im sure ive missed something but I can't see what.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
inadequateMonkey
  • 115
  • 1
  • 1
  • 9
  • 1
    one thing to look out for, urllib tends to run into difficulty when it encounters 'invalid' ssl certs on https URIs. without the proper flags it will fail somewhat silently. – Matt Joyce Sep 18 '15 at 18:52
  • 1
    Are you **certain** you got that `UnicodeEncodeError` from the code you posted? I'd expect `Unicode` **`Decode`** `Error` exceptions. Please post a **full** traceback for your error, but I am pretty sure your code works but you tried to *print* the results. – Martijn Pieters Sep 18 '15 at 18:56
  • ^ good point, i will look into that. although if i open the url with the browser, it downloads the json file and opens it with no problems. – inadequateMonkey Sep 18 '15 at 18:56
  • @MartijnPieters ahh nice catch, that error actually came from a print statement in the next function (there only for testing as im writing) i guess i have to do something to it before being able to display the data? – inadequateMonkey Sep 18 '15 at 18:58

3 Answers3

0

Your initial attempt is already correct. If it wasn't, you'd get decoding errors.

You have, instead, got encoding errors, moving from Unicode back to bytes. This is invariably caused by you using print() or trying to write the data to a file object. When printing, this is usually caused by your console not being able to handle the output. See python3 print unicode to windows xp console encode cp437 for example.

Your second attempt failed because you didn't read the response, then use json.loads() (since you are passing in a string now):

response = urllib.request.urlopen(self.query_base)
obj = json.loads(response.read().decode("utf-8"))
#              ^         ^^^^^^^
return obj

Your 3rd attempt did use .read() but you forgot to decode that time, and again did not use json.loads():

response = urllib.request.urlopen(self.query_base).read()
#                          you didn't forget this ^^^^^^^
obj = json.loads(response.decode('utf-8'))
#              ^         ^^^^^^^^^^^^^^^^
return obj

The last attempt passed in the raw response, without decoding the data.

Community
  • 1
  • 1
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • when using "obj = json.load(response.read().decode("utf-8"))" i get an error AttributeError: 'str' object has no attribute 'read', i guess, as my first attempt has worrked with other APIs i guess i will find a way to fix that. thanks for the info – inadequateMonkey Sep 18 '15 at 19:04
0
response = urllib.request.urlopen(self.query_base).read()
obj = json.loads(response)
return obj

This code should work. json.load is looking to read from a file stream. json.loads is what you want when reading JSON from a string.

Samantha Quiñones
  • 11
  • 1
0

ok, just incase anyone has this issue and reads this thread later

response = urllib.request.urlopen(self.query_base)
reader = codecs.getreader("utf-8")
obj = json.load(reader(response))
return ascii(obj)

worked fine.

thanks to Martijn Pieters

inadequateMonkey
  • 115
  • 1
  • 1
  • 9