1

Hi I am using python requests library to read contents of url. On local jupyter notebook following is python code:

import json
import requests
response = (requests.get("https://ws.bbc34.cs.streamuk.com/services/channelFeed.php?channel=zee5-punjabi&group=punjabi&output=rss2.0").text)
print((response))

and the response is something like:

xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/"> Zee5 Punjabi Zee5 Punjabi video rss feed https://ws.bbc34.cs.streamuk.com/ <atom:link href="https://ws.bbc34.cs.streamuk.com/services/channelFeed.php" rel="self" type="application/rss+xml"/> Fri, 25 Sep 2020 08:01:54 GMT StreamUK Crowdsourcer (http://www.streamuk.com) https://ws.bbc34.cs.streamuk.com/images/feed-logo.png Zee5 Punjabi https://ws.bbc34.cs.streamuk.com/ Fri, 25 Sep 2020 03:23:34 GMT openSearch:startIndex1</openSearch:startIndex> openSearch:itemsPerPage167</openSearch:itemsPerPage> openSearch:totalResults167</openSearch:totalResults> ਖੇਤੀ ਬਿਲਾਂ ’ਤੇ ਕਿਸਾਨ ਮੋਰਚਿਆਂ ’ਚੋਂ ਪੰਜਾਬ ਦੀਆਂ ਸਿਆਸੀ ਧਿਰਾਂ ਕੀ ਲੱਭ ਰਹੀਆਂ ਹਨ https://ws.bbc34.cs.streamuk.com/ ਪੰਜਾਬ ਦੀਆਂ ਸਿਆਸੀ ਧਿਰਾਂ ਕਿਸਾਨਾਂ ਦੇ ਸਕੇ ਹੋਣ ਦੇ ਦਾਅਵੇ ਤਾਂ ਕਰ ਰਹੀਆਂ ਹਨ ਪਰ ਇਸ ਸੰਘਰਸ਼ ਵਿੱਚ ਉਨ੍ਹਾਂ ਦੇ ਨਿੱਜੀ ਮੁਫ਼ਾਦ ਕੀ ਹਨ ਨਿੱਜੀ,ਸਿਆਸੀ,ਸਿਆਸੀ ਧਿਰਾਂ,ਸੰਘਰਸ਼,ਧਿਰਾਂ,ਮੋਰਚਿਆਂ,ਮੁਫ਼ਾਦ,ਉਨ੍ਹਾਂ,ਕਿਸਾਨ,ਕਿਸਾਨਾਂ,ਰਹੀਆਂ,ਬਿਲਾਂ,ਦਾਅਵੇ,ਦੀਆਂ ਸਿਆਸੀ,ਦੀਆਂ ਸਿਆਸੀ ਧਿਰਾਂ,ਪੰਜਾਬ,ਪੰਜਾਬ ਦੀਆਂ,ਪੰਜਾਬ ਦੀਆਂ ਸਿਆਸੀ Fri, 25 Sep 2020 01:00:03 GMT https://ws.bbc34.cs.streamuk.com/:7e3ac643-fede-11ea-9778-000c29b9294a <media:content url="https://ws.bbc34.cs.streamuk.com/content/channels/zee5-punjabi/streams/p08sgysb294a7e3ac64387.mp4" fileSize="204902218" bitrate="4690" type="video/mp4" medium="video" duration="362" width="1920" height="1080"/> <media:thumbnail url="https://ws.bbc34.cs.streamuk.com/content/channels/zee5-punjabi/thumbnails/p08sgysb294a7e3ac643.jpg" width="170" height="128"/> media:keywordsਨਿੱਜੀ,ਸਿਆਸੀ,ਸਿਆਸੀ ਧਿਰਾਂ,ਸੰਘਰਸ਼,ਧਿਰਾਂ,ਮੋਰਚਿਆਂ,ਮੁਫ਼ਾਦ,ਉਨ੍ਹਾਂ,ਕਿਸਾਨ,ਕਿਸਾਨਾਂ,ਰਹੀਆਂ,ਬਿਲਾਂ,ਦਾਅਵੇ,ਦੀਆਂ ਸਿਆਸੀ,ਦੀਆਂ ਸਿਆਸੀ ਧਿਰਾਂ,ਪੰਜਾਬ,ਪੰਜਾਬ ਦੀਆਂ,ਪੰਜਾਬ ਦੀਆਂ ਸਿਆਸੀ</media:keywords> ਖੇਤੀ ਬਿੱਲ: ‘ਆਹ ਸਾਡੀ ਹੋਂਦ ਨੂੰ ਖ਼ਤਮ ਕਰਨ ਦਾ ਇੱਕ ਜ਼ਰੀਆ ਹੈ’ https://ws.bbc34.cs.streamuk.com/ ਕਿਸਾਨ ਧਰਨਿਆਂ ਵਿਚ ਪੁਰਸ਼ਾਂ ਦੇ ਨਾਲ ਇਸ ਵਾਰ ਮਹਿਲਾਵਾਂ ਵੀ ਵੱਡੀ ਗਿਣਤੀ ਪਹੁੰਚ ਕੇ ਆਪਣੀ ਹਾਜ਼ਰੀ ਭਰ ਰਹੀਆਂ ਹਨ ਗਿਣਤੀ,ਧਰਨਿਆਂ,ਮਹਿਲਾਵਾਂ,ਕਿਸਾਨ,ਰਹੀਆਂ,ਹਾਜ਼ਰੀ,ਪਹੁੰਚ,ਪੁਰਸ਼ਾਂ,ਜ਼ਰੀਆ Thu, 24 Sep 2020 15:47:35 GMT https://ws.bbc34.cs.streamuk.com/:b4417aea-fecb-11ea-9778-000c29b9294a <media:content url="https://ws.bbc34.cs.streamuk.com/content/channels/zee5-punjabi/streams/p08sfy25294ab4417aea87.mp4" fileSize="177778979" bitrate="4690" type="video/mp4" medium="video" duration="306" width="1920" height="1080"/> <media:thumbnail url="https://ws.bbc34.cs.streamuk.com/content/channels/zee5-punjabi/thumbnails/p08sfy25294ab4417aea.jpg" width="170" height="128"/> media:keywordsਗਿਣਤੀ,ਧਰਨਿਆਂ,ਮਹਿਲਾਵਾਂ,ਕਿਸਾਨ,ਰਹੀਆਂ,ਹਾਜ਼ਰੀ,ਪਹੁੰਚ,ਪੁਰਸ਼ਾਂ,ਜ਼ਰੀਆ</media:keywords> ਪੀਐੱਮ ਮੋਦੀ ਦੀ ਕਿਹੜੀ ਗੱਲ 'ਤੇ ਵਿਰਾਟ ਕੋਹਲੀ ਆਪਣਾ ਹਾਸਾ ਨਹੀਂ ਰੋਕ ਸਕੇ

This is as expected:

But the same piece of code I am trying to run on EC2 server with same requests library installed and same python version,But i am not able to print response.I am getting the following error.

print((response)) UnicodeEncodeError: 'ascii' codec can't encode characters in position 1168-1171: ordinal not in range(128)

I did check both the versions of python and requests library and were same. python version:3.6 requests version:2.23

I am not able to find why this error. Is it beacause of "ਗਿਣਤੀ,ਧਰਨਿਆਂ,ਮਹਿਲਾਵਾਂ,ਕਿਸਾਨ,ਰਹੀਆਂ,ਹਾਜ਼ਰੀ,ਪਹੁੰਚ,ਪੁਰਸ਼ਾਂ,ਜ਼ਰੀਆ" characters,but it worked fine in local jupyter.I am not getting it .Please help.

sumesh shetty
  • 251
  • 2
  • 9
  • i ran locale on both my local and ec2 server,the results are: – sumesh shetty Sep 25 '20 at 08:32
  • [ec2-user@ip-172-31-37-106 news_xml_code]$ locale locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory LANG=en_US.UTF-8 LC_CTYPE=UTF-8 LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= [ec2-user@ip-172-31-37-106 news_xml_code]$ result of ec2 server – sumesh shetty Sep 25 '20 at 08:36
  • 124s-MacBook-Air:~ a124$ locale LANG="" LC_COLLATE="C" LC_CTYPE="UTF-8" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL= 124s-MacBook-Air:~ a124$ this is local machine – sumesh shetty Sep 25 '20 at 08:36
  • What is the output of `locale charmap` on the server? – snakecharmerb Sep 25 '20 at 08:37
  • [ec2-user@ip-172-31-37-106 news_xml_code]$ locale charmap locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory ANSI_X3.4-1968 – sumesh shetty Sep 25 '20 at 08:38
  • 1
    The problem is that charmap value, which is basically ASCII. You could try running your script with the PYTHONIOENCODING environment variable set, as described in this answer https://stackoverflow.com/a/57224678/5320906 – snakecharmerb Sep 25 '20 at 08:54
  • Thanks a lot that worked but i cannot set the environment while running as it will be autmated in future ,can this be done via adding something in python code or can i set the server environment permanently. – sumesh shetty Sep 25 '20 at 08:58
  • You could perhaps set it in the server user's `.bashrc` or `.profile` file. – snakecharmerb Sep 25 '20 at 10:07

0 Answers0