2

I thought json.load() should be able to read objects exactly like http.client.HTTPResponse, but it seems to be tripping up on its read() being a bytes-like object. (I am using Python 3.3.) To my surprise, I found no resource directly addressing this use though I thought this was a primary use case.

import urllib.request, json

# Y!F url
yf = 'http://d.yimg.com/autoc.finance.yahoo.com/autoc'

# Mock lookup
data = urllib.parse.urlencode({'query': 'Ford', 'callback': 'YAHOO.Finance.SymbolSuggest.ssCallback'})
data = data.encode('utf-8')
request = urllib.request.Request(yf)
request.add_header('User-Agent','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14')
request.add_header('Content-type','text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8;charset=utf-8')
request.add_header('Accept','text/plain')

mock = urllib.request.urlopen(request, data)
json.load(mock)

This results in the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/__init__.py", line 264, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/__init__.py", line 309, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/decoder.py", line 352, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: can't use a string pattern on a bytes-like object
Matthew Trevor
  • 14,354
  • 6
  • 37
  • 50
László
  • 3,914
  • 8
  • 34
  • 49
  • 1
    It seems you may have hit this issue: http://stackoverflow.com/questions/6862770/python-3-let-json-object-accept-bytes-or-let-urlopen-output-strings – Matthew Trevor Oct 02 '12 at 05:27
  • @MatthewTrevor, indeed, thanks. The readall-encode-loads workaround got me further, but only this far: `ValueError: No JSON object could be decoded` I am pretty sure Yahoo! returns a JSON object from the URL in my question. What am I doing wrong now? (Maybe unrelated to the original question, I know.) – László Oct 02 '12 at 13:12
  • I see. Because it had the function wrapper around the JSON object. I thought this is how it is supposed to work. Fixed now. – László Oct 02 '12 at 14:56

1 Answers1

0

This has been solved in a previous thread: Python 3, let json object accept bytes or let urlopen output strings (HT: Matthew Trevor)

Also, what Yahoo! returns here is not just to the JSON object but also a YAHOO.Finance.SymbolSuggest.ssCallback() wrapper. Stripping that fixes things. (Though still sad it's needed.)

This works:

import urllib.request, json, re

# Y!F url
yf = 'http://d.yimg.com/autoc.finance.yahoo.com/autoc'

# Mock lookup
data = urllib.parse.urlencode({'query': 'Ford', 'callback': 'YAHOO.Finance.SymbolSuggest.ssCallback'})
data = data.encode('utf-8')
request = urllib.request.Request(yf)
response = urllib.request.urlopen(request, data)
j = json.loads(re.search(r'{.*}',response.readall().decode('utf-8')).group())
Community
  • 1
  • 1
László
  • 3,914
  • 8
  • 34
  • 49