(Python) Beautifull soup and encoding (utf-8, cp1252,ascii...)

Question

Please help, I am so loosing nerves now.I am having this problems since I started learning Python. Always come to a same issue and no one online can give any valid answer

My code:

from bs4 import BeautifulSoup
import requests

page = requests.get(
    'https://forecast.weather.gov/MapClick.php?lat=34.05349000000007&lon=-118.24531999999999#.XswiwMCxWUk')
soup = BeautifulSoup(page.content, 'html.parser')
week = soup.find(id='seven-day-forecast-body')
items = week.find_all(class_='forecast-tombstone')

print(items[0].find(class_='period-name').get_text())
print(items[0].find(class_='short-desc').get_text())
print(items[0].find(class_='temp temp-high').get_text())

period_names = [item.find(class_='period-name').get_text() for item in items]
short_descp = [item.find(class_='short-desc').get_text() for item in items]
temp = [item.find(class_='temp temp-high').get_text() for item in items]
print(period_names)
print(short_descp)
print(temp)

output :

[Running] python -u "c:\Users\dukasu\Documents\Python\test.py"
ThisAfternoon
Partly Sunny
High: 76 �F
Traceback (most recent call last):
  File "c:\Users\dukasu\Documents\Python\test.py", line 20, in <module>
    temp = [item.find(class_='temp temp-high').get_text() for item in items]
  File "c:\Users\dukasu\Documents\Python\test.py", line 20, in <listcomp>
    temp = [item.find(class_='temp temp-high').get_text() for item in items]
AttributeError: 'NoneType' object has no attribute 'get_text'

[Done] exited with code=1 in 0.69 seconds

Issue is due to a utf-8 encoding (my PC is at cp1252), but how to solve it terminally (I think problem is cos it cant operate with degree symbol). There is a simple code in Python 2, but how to solve it in Python 3.xx. How to set encoding at start of the code and forget about this issue. anp please pardon my english, it is not my native language.

try this [post](https://stackoverflow.com/questions/11741574/how-to-print-utf-8-encoded-text-to-the-console-in-python-3) — dboy, May 29 '20 at 20:50
Why do you assume the error has something to do with utf-8? I can't see that anywhere in the error message you posted. — Mark Ransom, May 29 '20 at 20:51
error is coming from class name `class_='temp temp-high'` @MarkRansom — 0m3r, May 29 '20 at 20:58
@0m3r the problem is that `item.find` is returning `None`, which again has nothing to do with utf-8. — Mark Ransom, May 29 '20 at 21:06

0m3r · Answer 1 · 2020-05-29T21:09:23.973

The error is coming from class name which is returning None, use only class_='temp Not class_='temp temp-high

Example

temp = [item.find(class_='temp').get_text() for item in items]

Full code

from bs4 import BeautifulSoup
import requests

page = requests.get(
    'https://forecast.weather.gov/MapClick.php?lat=34.05349000000007&lon=-118.24531999999999#.XswiwMCxWUk')
soup = BeautifulSoup(page.content, 'html.parser')
week = soup.find(id='seven-day-forecast-body')
items = week.find_all(class_='forecast-tombstone')

print(items[0].find(class_='period-name').get_text())
print(items[0].find(class_='short-desc').get_text())
print(items[0].find(class_='temp temp-high').get_text())

period_names = [item.find(class_='period-name').get_text() for item in items]
short_descp = [item.find(class_='short-desc').get_text() for item in items]
temp = [item.find(class_='temp').get_text() for item in items]
print(period_names)
print(short_descp)
print(temp)

Prints Out

ThisAfternoon
Partly Sunny
High: 76 °F
['ThisAfternoon', 'Tonight', 'Saturday', 'SaturdayNight', 'Sunday', 'SundayNight', 'Monday', 'MondayNight', 'Tuesday']
['Partly Sunny', 'Patchy Fog', 'Patchy Fogthen MostlySunny', 'Patchy Fog', 'Patchy Fogthen PartlySunny', 'Patchy Fog', 'Patchy Fogthen MostlyCloudy', 'Mostly Cloudy', 'Partly Sunny']
['High: 76 °F', 'Low: 58 °F', 'High: 75 °F', 'Low: 59 °F', 'High: 80 °F', 'Low: 61 °F', 'High: 78 °F', 'Low: 61 °F', 'High: 77 °F']

score 0 · Answer 2 · edited Aug 29 '20 at 09:43

This turned out to be a simple issue.

OK changed but here is a printout:

[Running] python -u "c:\Users\dukasu\Documents\Python\test.py"
ThisAfternoon
Partly Sunny
High: 76 �F
['ThisAfternoon', 'Tonight', 'Saturday', 'SaturdayNight', 'Sunday', 'SundayNight', 'Monday', 'MondayNight', 'Tuesday']
['Partly Sunny', 'Patchy Fog', 'Patchy Fogthen MostlySunny', 'Patchy Fog', 'Patchy Fogthen PartlySunny', 'Patchy Fog', 'Patchy Fogthen MostlyCloudy', 'Mostly Cloudy', 'Partly Sunny']
['High: 76 �F', 'Low: 58 �F', 'High: 75 �F', 'Low: 59 �F', 'High: 80 �F', 'Low: 61 �F', 'High: 78 �F', 'Low: 61 �F', 'High: 77 �F']

[Done] exited with code=0 in 0.619 seconds

How to print out degree symbol ° ?

later I added

import sys
sys.stdout.reconfigure(encoding='utf-8')

and print out :

High: 76 °F

(Python) Beautifull soup and encoding (utf-8, cp1252,ascii...)

2 Answers2