How to scrape text based on a specific link with BeautifulSoup?

Question

I'm trying to scrape text from a website, but specifically only the text that's linked to with one of two specific links, and then additionally scrape another text string that follows shortly after it.

The second text string is easy to scrape because it includes a unique class I can target, so I've already gotten that working, but I haven't been able to successfully scrape the first text (with the one of two specific links).

I found this SO question ( Find specific link w/ beautifulsoup ) and tried to implement variations of that, but wasn't able to get it to work.

Here's a snippet of the HTML code I'm trying to scrape. This patter recurs repeatedly over the course of each page I'm scraping:

<em>[<a href="forum.php?mod=forumdisplay&fid=191&amp;filter=typeid&amp;typeid=19">女孩</a>]</em> <a href="thread-443414-1-1.html" onclick="atarget(this)" class="s xst">寻找2003年出生2004年失踪贵州省黔西南布依族苗族自治州贞丰县珉谷镇锅底冲  黄冬冬289179</a>

The two parts I'm trying to scrape and then store together in a list are the two Chinese-language text strings.

The first of these, 女孩, which means female, is the one I haven't been able to scrape successfully.

This is always preceded by one of these two links: forum.php?mod=forumdisplay&fid=191&filter=typeid&typeid=19 (Female) forum.php?mod=forumdisplay&fid=191&filter=typeid&typeid=15 (Male)

I've tested a whole bunch of different things, including things like:

gender_containers = soup.find_all('a', href = 'forum.php?mod=forumdisplay&fid=191&amp;filter=typeid&amp;typeid=19')

print(gender_containers.get_text())

But for everything I've tried, I keep getting errors like:

ResultSet object has no attribute 'get_text'. You're probably treating a list of items like a single item. Did you call find_all() when you meant to call find()?

I think that I'm not successfully finding those links to grab the text, but my rudimentary Python skills thus far have failed me in figuring out how to make it happen.

What I want to have happen ultimately is to scrape each page such that the two strings in this code (女孩 and 寻找2003年出生2004年失踪贵州省...)

<em>[<a href="forum.php?mod=forumdisplay&fid=191&amp;filter=typeid&amp;typeid=19">女孩</a>]</em> <a href="thread-443414-1-1.html" onclick="atarget(this)" class="s xst">寻找2003年出生2004年失踪贵州省黔西南布依族苗族自治州贞丰县珉谷镇锅底冲  黄冬冬289179</a>

...are scraped as two separate variables so that I can store them as two items in a list and then iterate down to the next instance of this code, scrape those two text snippets and store them as another list, etc. I'm building a list of list in which I want each row/nested list to contain two strings: the gender (女孩 or 男孩）and then the longer string, which has a lot more variation.

(But currently I have working code that scrapes and stores that, I just haven't been able to get the gender part to work.)

Maybe I'm missing something, but if `` is always female and `` is always male, why bother with the text? — Jack Fleeting, May 05 '19 at 11:58
I don't need to bother with the text necessarily, but I don't know how to scrape/store that data in _any_ format currently. (I realize this is probably a very noob problem, but I am very noob...) — custerc, May 06 '19 at 02:35
Marking the answer below as accepted as I have resolved this, but it was really @JackFleeting 's comment that helped me figure it out. I realized the challenge was trying to grab these two separate items together, so instead I scraped them separately into two lists and then used `zip()` to combine them into a final list. Thanks! — custerc, May 06 '19 at 03:31

score 0 · Answer 1 · answered May 05 '19 at 00:39

Try the following code.

from bs4 import BeautifulSoup
data='''<em>[<a href="forum.php?mod=forumdisplay&fid=191&amp;filter=typeid&amp;typeid=19">女孩</a>]</em> <a href="thread-443414-1-1.html" onclick="atarget(this)" class="s xst">寻找2003年出生2004年失踪贵州省黔西南布依族苗族自治州贞丰县珉谷镇锅底冲  黄冬冬289179</a>'''
soup=BeautifulSoup(data,'html.parser')
print(soup.select_one('em').text)

OutPut:

[女孩]

QHarr · Accepted Answer · 2019-05-06T04:31:44.953

Sounds like you could use attribute = value css selector with $ ends with operator

If there can only be one occurrence per page

soup.select_one("[href$='typeid=19'], [href$='typeid=15']").text

This is assuming those typeid=19 or typeid=15 only occur at the end of the strings of interest. The "," between the two in the selector is to allow for matching on either.

You could additionally handle possibility of not being present as follows:

from bs4 import BeautifulSoup
html ='''<em>[<a href="forum.php?mod=forumdisplay&fid=191&amp;filter=typeid&amp;typeid=19">女孩</a>]</em> <a href="thread-443414-1-1.html" onclick="atarget(this)" class="s xst">寻找2003年出生2004年失踪贵州省黔西南布依族苗族自治州贞丰县珉谷镇锅底冲  黄冬冬289179</a>'''
soup=BeautifulSoup(html,'html.parser')
gender = soup.select_one("[href$='typeid=19'], [href$='typeid=15']").text if soup.select_one("[href$='typeid=19'], [href$='typeid=15']") is not None else 'Not found'
print(gender)

Multiple values:

genders = [item.text for item in soup.select_one("[href$='typeid=19'], [href$='typeid=15']")]

This works, but how could I get it to find that more than once? There are multiple instances of that code on the page. Here's what I have now, but it's just pulling the first example it finds rather than all ```python link_containers = soup.find_all('a', class_ = 's xst') # isolate the forum title links gender_containers = soup.find_all('em') for link in gender_containers: gender = soup.find_all("[href$='typeid=19'], [href$='typeid=15']").text for link in link_containers: link_text = link.get_text() testlist1.append([link_text, gender]) ``` — custerc, May 06 '19 at 02:28
do you need to group them with other info or just return a list of them? I've shown returning a list above. Can update further of course. — QHarr, May 06 '19 at 04:30

How to scrape text based on a specific link with BeautifulSoup?

2 Answers2