16

I'm getting UnicodeEncodeError: 'charmap' codec can't encode character '\u200b' in position 756: character maps to error while running this code::

from bs4 import BeautifulSoup
import requests
r = requests.get('https://stackoverflow.com').text
soup = BeautifulSoup(r, 'lxml')
print(soup.prettify())

and the output is:

Traceback (most recent call last):
  File "c:\Users\Asus\Documents\Hello World\Web Scraping\st.py", line 5, in <module>
    print(soup.prettify())
  File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u200b' in position 756: character maps to <undefined>

I'm using python 3.8.1 and UTF-8 in vs code. How to solve this?

Trần Hữu Hiền
  • 872
  • 1
  • 9
  • 22
Asir Shahriar Roudra
  • 338
  • 1
  • 3
  • 15
  • Does this answer your question? [UnicodeEncodeError: 'charmap' codec can't encode characters](https://stackoverflow.com/questions/27092833/unicodeencodeerror-charmap-codec-cant-encode-characters) – Ulrich Eckhardt Dec 09 '21 at 17:30

3 Answers3

17

There are hints in the full error message... I will keep here what seems most important:

Traceback ...
  File "...\cp1252.py", ...
UnicodeEncodeError: 'charmap' codec can't encode character '\u200b' ...

The error is caused by the print call. Somewhere in you text, you have a ZERO WIDTH SPACE character (Unicode U+200B), and if you print to a Windows console, the string is internally encoded into the Windows console code page (cp1252 here). And the ZERO WIDTH SPACE is not represented in that code page. BTW the default console is not really unicode friendly in Windows.

There is little to do in a Windows console. I would advise you to try one of these workarounds:

  • do not print to the console but write to a (utf8) file. You will then be able to read it with a utf8 enabled text editor like notepad++

  • manually encode anything before printing it, with errors='ignore' or errors='replace'. That way, possibly offending characters will be ignored and no error will arise

      print(soup.prettify().encode('cp1252', errors='ignore'))
Serge Ballesta
  • 143,923
  • 11
  • 122
  • 252
16

You can explore little bit on your own... but for python 2.7 what i usually do is use this to clean my text:

text = text.encode('utf-8').decode('ascii', 'ignore')

python 3 equivalent for this is simply:

text = str(text)

For your case, try this:

r = requests.get('https://stackoverflow.com').text.encode('utf8').decode('ascii', 'ignore')

otherwise normally:

r = requests.get('https://stackoverflow.com')
soup = BeautifulSoup(r.content, 'lxml')
print soup

(I don't think this should give any error.)

vaibhav s
  • 185
  • 8
  • are you saying to convert soup to string and then use prettify? – Asir Shahriar Roudra Jun 30 '20 at 14:05
  • bro getting a new error by running ur code ,, I am getting the following output: ``` Traceback (most recent call last): File "c:\Users\Asus\Documents\Hello World\Web Scraping\reddit.py", line 6, in print(soup) File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u200b' in position 734: character maps to ``` – Asir Shahriar Roudra Jun 30 '20 at 14:31
9

For python3, you can use:

text = str(text.encode('utf-8'))
Suraj Rao
  • 29,388
  • 11
  • 94
  • 103
Faizan Alam
  • 91
  • 1
  • 2
  • Thanks! This solutions worked for me. Firstly, I tried the solution by vaibhab. However, it didnt work – rafaoc Jul 05 '22 at 15:40
  • Worked for me for error `UnicodeEncodeError: 'charmap' codec can't encode character '\u021d'` with `df[file] = pd.read_csv(data_path+file, encoding="ISO-8859-1") #if utf-8 encoding error print(str(file.encode('utf-8')))` `print('{0} was created successfully'.format(str(tbl_name.encode('utf-8'))))` – Lod Nov 15 '22 at 17:25