Why does Beautifulsoup.find() not give the specific result?

Question

I have this code below, and I am trying to get 'Oswestry, England' as the result.

label = soup.findall('span',{'class':"ProfileHeaderCard-locationText"})
print(label)

But, it doesn't give me a value.

Here is what the HMTL code looks like

<span class="ProfileHeaderCard-locationText u-dir" dir="ltr">
     <a data-place-id="5b756a1991aa8648" href="/search?q=place%3A5b756a1991aa8648">Oswestry, England</a>
     </span>

When I print label the result is the HTML code I posted above. Here is my full code:

import requests as req
from bs4 import BeautifulSoup

usernames = #list of username

location_list = []

for x in usernames:
    url= "https://twitter.com/" + x
    try:
        html = req.get(url)
    except Exception as e:
        print("Failed to")
        continue
    soup = BeautifulSoup(html.text,'html.parser')
    try:
        label = soup.find('span',{'class':"ProfileHeaderCard-locationText"})
        label_formatted = label.string.lstrip()
        label_formatted = label_formatted.rstrip()
        if label_formatted != "":
            location_list.append(label_formatted)
            print(x + ' : ' + label_formatted) 
        else:
            print('Not found')
    except:
        print('Not found')

The code works for most page with HTML that looks like this: Oswestry, England But, if the HTML code looks like this Oswestry, England I can no longer get Oswestry England. — Mtrinidad, Apr 10 '20 at 02:10
I think you should target the 'a' tag via 'id' attribute instead. — arantebw, Apr 10 '20 at 02:12
I looked at the HTML of several twitter accounts, didn't see the class `ProfileHeaderCard-locationText` in any of them. — Barmar, Apr 10 '20 at 02:19
Does this answer your question? [beautiful soup .find can't find anything](https://stackoverflow.com/questions/59587283/beautiful-soup-find-cant-find-anything) — αԋɱҽԃ αмєяιcαη, Apr 10 '20 at 08:10

score 1 · Answer 1 · answered Apr 10 '20 at 02:10

1

You should call find, not find_all to get a single element. Then use the .text attribute to get the text content.

label = soup.find('span',{'class':"ProfileHeaderCard-locationText"})
print(label.text)

answered Apr 10 '20 at 02:10

Barmar

741,623
53
500
612

This also does not work. Because the HTML of the website looks like this [ Oswestry, England ] – Mtrinidad Apr 10 '20 at 02:12
The HTML isn't inside `` and ``? – Barmar Apr 10 '20 at 02:12

score 0 · Answer 2 · edited Apr 10 '20 at 06:31

It seems that you were searching for a span tag with the class attribute exactly matching your query class. As the span has two classes, your test failed and no results returned.

Using css selectors, you could try your solution as:

from bs4 import BeautifulSoup as BS
soup = BS('''<span class="ProfileHeaderCard-locationText u-dir">.....</span>''', 'html.parser')
soup.select('span.ProfileHeaderCard-locationText')

returns span tags that contain your prescribed class.

Why does Beautifulsoup.find() not give the specific result?

3 Answers3