in python, UnicodeEncodeError: 'ascii' codec can't encode characters in position 15-18: inal not in range(128)

Question

in this part of code , I fetch alot of URLs from stored URLs in (url.txt) file but when the code face non-English URL, it broke and give error,

any simple code to fix this problem ? Thanks

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq

page_url = "url.txt"

with open("url.txt", "r") as fr:
    for url in map(lambda x: x.strip(), fr.readlines()): 
      print(url)
      uClient = uReq(url)
      page_soup = soup(uClient.read(), "html.parser")


      # the rest logic

Full error message

E:\Desktop>question.py
Traceback (most recent call last):
  File "E:\Desktop\question.py", line 12, in <module>
    uClient = uReq(url)
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\urllib\req
uest.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\urllib\req
uest.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\urllib\req
uest.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\urllib\req
uest.py", line 502, in _call_chain
    result = func(*args)
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\urllib\req
uest.py", line 1362, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\urllib\req
uest.py", line 1319, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\http\clien
t.py", line 1230, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\http\clien
t.py", line 1241, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\http\clien
t.py", line 1096, in putrequest
    self._output(self._encode_request(request))
  File "C:\Users\A-Data\AppData\Local\Programs\Python\Python38-32\lib\http\clien
t.py", line 1176, in _encode_request
    return request.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 15-18: ord
inal not in range(128)

Does this answer your question? [How to fetch a non-ascii url with Python urlopen?](https://stackoverflow.com/questions/4389572/how-to-fetch-a-non-ascii-url-with-python-urlopen) — Iain Shelvington, Apr 25 '20 at 05:12
I did read the other article but get confused about what should I add exactly to my code — user2653711, Apr 25 '20 at 05:14
Use `urllib.parse.quote` https://stackoverflow.com/a/29231552/548562 — Iain Shelvington, Apr 25 '20 at 05:15
Use urllib.parse.quote is good when have 1 url , but I have alot — user2653711, Apr 25 '20 at 05:54

score 1 · Accepted Answer · answered Apr 28 '20 at 08:51

1

If you want a simple fix, use the requests module instead of urllib. It implicitly transforms Unicode urls, so you don't have to.

from bs4 import BeautifulSoup as soup
import requests

page_url = "url.txt"

with open("url.txt", "r") as fr:
    for url in map(lambda x: x.strip(), fr.readlines()): 
      print(url)
      response = requests.get(url)
      page_soup = soup(response.text, "html.parser")

answered Apr 28 '20 at 08:51

GordonAitchJay

4,640
1
14
16

you are GENIUS, problem solved , thank you very much – user2653711 Apr 29 '20 at 09:59

in python, UnicodeEncodeError: 'ascii' codec can't encode characters in position 15-18: inal not in range(128)

1 Answers1