UnicodeEncodeError Converting Requests Text to JSON in Python

Question

I am trying to scrape website but with certain searches it gives me error:

File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 5399-5400: character maps to <undefined>

My Python script is:

import requests
import json

url = 'https://www.protocols.io/api/v3/protocols?filter=%20public%20&order_field=relevance&key=%20gel%22electrophoresis%20'

r = requests.get(url)
text = r.text 
jason = json.loads(text)
print (jason)

I have tried to change the encoding as suggested here by

text = text.encode('utf-8')

But still recieve the same error. It seems to be using the cp1252 encoding not utf-8?

Any suggestions will be greatly appreciated!

Update: This seems to be a issue in git bash or atom as works without error in Spyder or windows cmd

score 1 · Answer 1 · answered May 20 '20 at 10:29

1

Maybe is because you haven't imported sys,

like this:

import sys
import codecs

import requests
import json

url = 'https://www.protocols.io/api/v3/protocols?filter=%20public%20&order_field=relevance&key=%20gel%22electrophoresis%20'

r = requests.get(url)
text = r.text 
text = text.encode('utf-8') 
jason = json.loads(text)
print (jason)

answered May 20 '20 at 10:29

Edouard Yonga

477
4
12

thanks but this doesn't seem to work for me. I still get the same error. What version of python are you using (I am on 3.7.4) – JEJS May 20 '20 at 10:34
python 3.6.4 i don't think is the version that disturbs How are you running the codes? – Edouard Yonga May 20 '20 at 10:40
i run mine on the terminal using this: **python3 pytest.py** – Edouard Yonga May 20 '20 at 10:43
I just updated to 3.8.3. I am running in atom text editor – JEJS May 20 '20 at 10:44
I ran on terminal but still have same error! Would you mind sharing your output? – JEJS May 20 '20 at 10:47

score 0 · Answer 2 · answered May 20 '20 at 10:57

0

My OUTPUT: as you can see is a json of items

answered May 20 '20 at 10:57

Edouard Yonga

477
4
12

Thank you. This is very wierd. I run in atom and terminal i get this error. I ran in Spyder (python 3.7) and it works, any ideas? – JEJS May 20 '20 at 11:08
I am going to leave it open because I think the original code works without importing sys or codecs, and i need a solution that works on terminal or atom. Thanks for the help though – JEJS May 20 '20 at 11:41

UnicodeEncodeError Converting Requests Text to JSON in Python

2 Answers2