Python 3 Urlopen problems

Question

I know it's been heavily documented and I've tried to place the .read().decode('utf-8') everywhere I think it should go but I'm still missing something and I'm not sure where else to turn. I've got a script that works fine in Python 2.7 to scrape ESRI REST URL's for data by fetching the object ID's. However, it does not seem to be cooperating in Python 3. It continuously fails on line 29 (see below). Any help would be GREATLY appreciated.

        version = json.loads(webresp.decode('utf-8'))['currentVersion']

I'm always getting this error consistently:

Traceback (most recent call last):
  File "<pyshell#20>", line 1, in <module>
    main(url, jsonfilelocation, jsonfilename)
  File "\\SERVER\Projects\Python\TOUCH\esrirestjson.py", line 73, in main
    oids = ESRIJSON().getobjrange(webconn, path)
  File "\\SERVER\Projects\Python\TOUCH\esrirestjson.py", line 29, in getobjrange
    version = json.loads(webresp.decode('utf-8'))['currentVersion']
  File "C:\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

Can you add code that is the example of you actually instantiating the class `ESRIJSON` ? I can definitely provide the answer and test your code myself quickly with that. Thanks — james-see, Oct 12 '17 at 16:54
Your error is here: **version = json.loads(webresp.decode('utf-8'))['currentVersion']** What did you try? - Why would you need to post the whole code for this? — Anton vBR, Oct 12 '17 at 16:54
https://stackoverflow.com/questions/31708519/request-returns-bytes-and-im-failing-to-decode-them Maybe try to add .content? webconn.getresponse().content — Anton vBR, Oct 12 '17 at 16:55
Added the extra code as requested. As far as what I've tried, I've tried adding `.read().decode('utf-8')` at each `json.loads()` instance as is usually seen. — AGH_TORN, Oct 12 '17 at 16:58
@ATCH_torn python3 doesnt need the decode stuff, leave decode encode in python 27x crap ;) also you added the main() function but how about the `main(url, jsonfilelocation, jsonfilename)` I need the actual url you are trying, assuming something with arcgis? also you should pass defaults like this: `main(url="defaulurl.com", jsonfilelocation="test/", jsonfilename="output.json") — james-see, Oct 12 '17 at 17:00
@ATCH_torn not sure why you are getting down-voted, this is a legit question to do with python 2 to python 3 issues that are well-known. — james-see, Oct 12 '17 at 17:02
Yeah I'm not quite sure either... Anyway, here's the URL I'm trying on: `http://arcgis4.roktech.net/arcgis/rest/services/FranklinCountyAL/Web/MapServer/43/query` No matter what site I try on I keep getting the same error, I'm not sure what I'm doing wrong. Forgive me I'm a Python 3 extreme novice. — AGH_TORN, Oct 12 '17 at 17:05

score 1 · Accepted Answer · answered Oct 12 '17 at 17:25

1

I got your error fixed by changing json.loads() to json.load and removed all the decode('utf-8'). Now I encounter another bug due to you not handling string formatting properly in Python 3. The qs variables need to be in the format "?where=&outFields=*&returnGeometry={}.format(variable to add to string) which saves tons of worrying about encoding issues. I am still getting ascii errors with that qs string but hopefully this points you in the right direction. Python 3 is 10x better than 2.7 and well worth the effort.

answered Oct 12 '17 at 17:25

james-see

12,210
6
40
47

Interesting, when I remove the `json.loads()` and switch to `json.load()` I get an `AttributeError: 'bytes' object has no attribute 'read'`. – AGH_TORN Oct 12 '17 at 18:22
Sorry. Also remove the .read(). Not necessary either. I removed as well and forgot to put that in answer. You can test by print(version) and it works great or even print(webresp) after you json.load() it. Always print your data as you go to see what you are dealing with. – james-see Oct 12 '17 at 18:32
Well it's an `AttributeError` from the actual json module. `File "C:\ArcGISPro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py", line 265, in load return loads(fp.read(),` – AGH_TORN Oct 12 '17 at 19:02
remember I said remove .loads() and make it .load()... right? Too much to fix in your code honestly. I removed every .read .decode and replaced .loads with .load and printed out the response object after loaded into json and printed fine with data in there from the endpoint. I have no more to add. – james-see Oct 12 '17 at 19:04

Python 3 Urlopen problems

1 Answers1