Why am I getting UnicodeEncode error?

Question

I'm trying to make a small parsing script and testing out waters. I am not sure why am I getting this error

my code is

from bs4 import BeautifulSoup as bs
import requests
r = requests.get('http://www.marca.com/en/football/real-madrid.html?intcmp=MENUPROD&s_kw=english-real-madrid')

data = r.text.encode() 

soup = bs(data,'html.parser')

print (soup.prettify())

and the error

print (soup.prettify())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 2153-2154: ordinal not in range(128)

however if I use .encode() in my print line, it works fine. I just want to be 100% sure I am accurate with this. Have 0 experience parsing HTML/XML

It should be caused by the retrieved html content has non unicode characters. — Sphinx, Jan 22 '18 at 18:32
Normally one html page must declare what kind of charset it uses in its head. Give a try to get that charset name then apply appropriate decoder. — Sphinx, Jan 22 '18 at 18:36

score 1 · Accepted Answer · answered Jan 22 '18 at 18:55

The solution is this

from bs4 import BeautifulSoup as bs
import requests

req = requests.get('http://www.marca.com/en/football/real-madrid.html?intcmp=MENUPROD&s_kw=english-real-madrid')
data = req.text

soup = bs(data,'html.parser')

print (soup.prettify('latin-1'))

with the help of this question

Why am I getting UnicodeEncode error?

1 Answers1