Dealing with hexadecimal values in BeautifulSoup response?

Question

I am using beautiful soup to scrape some data with:

url = "https://www.transfermarkt.co.uk/jorge-molina/profil/spieler/94447"
heads = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}

response = requests.get(url,headers=heads)
soup = BeautifulSoup(response.text, "lxml")

Then, I extract I particular piece of information with:

height = soup.find_all("th", string=re.compile("Height:"))[0].findNext("td").text
print(height)

which works as intended, printing

1,74 m

but when I try to evaluate that string with this function:

def format_height(height_string):
    return int(height_string.split(" ")[0].replace(',',''))

I get the following error:

format_height(height)
Traceback (most recent call last):
  File "get_player_info.py", line 73, in <module>
    player_info = get_player_info(url)
  File "get_player_info.py", line 39, in get_player_info
    format_height(height)
  File "/Users/kompella/Documents/la-segunda/util.py", line 49, in format_height
    return int(height_string.split(" ")[0].replace(',',''))
ValueError: invalid literal for int() with base 10: '174\xa0m'

I am wondering how I should evaluate the hexadecimal values I am getting?

score 1 · Accepted Answer · answered Oct 07 '19 at 06:30

Use an attribute=value selector to target height instead then use function as is

import requests
from bs4 import BeautifulSoup as bs

def format_height(height_string):
    return int(height_string.split(" ")[0].replace(',',''))

r = requests.get('https://www.transfermarkt.co.uk/jorge-molina/profil/spieler/94447', headers = {'User-Agent':'Mozilla\5.0'})
soup = bs(r.content,'lxml')
height_string = soup.select_one('[itemprop=height]').text

print(format_height(height_string))

score 0 · Answer 2 · answered Oct 07 '19 at 02:06

Everything is perfectly fine, just deconstruct them & you can do whatever you want after.

import requests
import re
from bs4 import BeautifulSoup

url = "https://www.transfermarkt.co.uk/jorge-molina/profil/spieler/94447"
heads = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}

response = requests.get(url,headers=heads)
soup = BeautifulSoup(response.text, "lxml")

height = soup.find_all("th", string=re.compile("Height:"))[0].findNext("td").text
numerals = [int(s) for s in re.findall(r'\b\d+\b', height)]
print (numerals)
#output: [1, 74]
print ("Height is: " + str(numerals[0]) +"."+ str(numerals[1]) +"m")
#output: Height is: 1.75m
print ("Height is: " + str(numerals[0]) + str(numerals[1]) +"cm")
#output: Height is: 175cm

Anyways, the same question was discuss in this thread. You may take a look: ValueError: invalid literal for int() with base 10: ''

Dealing with hexadecimal values in BeautifulSoup response?

2 Answers2