How do I get the text element of a hyperlink from a web page using Python?

Question

I am scraping web data and need to return just the text element associated with a hyperlink. The hyperlink and text are unknown. The class is known. Here is example HTML:

<div class="a-column SsCol" role = "gridcell">
    <h3 class="a-spacing-none SsName">
        <span class="a-size-medium a-text-bold">
            <a href="/gp/aag/main/ref=sm_name_2?ie=UTF8&ids=15112acd">Direct Name</a>
        </span>
    </h3>
</div>

Alternatively, the desired text may be associated with an image instead of a hyperlink:

<div class="a-column SsCol" role = "gridcell">
    <h3 class="a-spacing-none SsName">
            <img alt="Direct Name" src="https://images-hosted.com//01x-j.gi">
    </h3>
</div>

I have tried the method below:

from lxml import html
import requests
response = requests.get('https://www.exampleurl.com/')
doc = html.fromstring(response.content)
text1 = doc.xpath("//*[contains(@class, 'SsName')]/text()")

I am using lxml instead of BeautifulSoup, but am willing to switch if it is recommended. The desired result is:

print(text1)
['Direct Name']

When associated with img will it always be the value of the alt attribute for the text you want? — QHarr, Dec 07 '18 at 07:32

KC. · Accepted Answer · 2018-12-07T07:11:24.533

//*[contains(@alt, '')]/@alt find all tags which have alt element. In reality, this xpath is extended from XPath Query: get attribute href from a tag. And you can select specific tag, as my text2 showed

from lxml import html

text = """
<div class="a-column SsCol" role = "gridcell">
    <h3 class="a-spacing-none SsName">
        <span class="a-size-medium a-text-bold">
            <a href="/gp/aag/main/ref=sm_name_2?ie=UTF8&ids=15112acd">Direct Name</a>
        </span>
    </h3>
</div>
<div class="a-column SsCol2" role = "gridcell">
    <h3 class="a-spacing-none SsName">
            <img alt="Direct Name" src="https://images-hosted.com//01x-j.gi">
    </h3>
</div>

"""

doc = html.fromstring(text)
text1 = doc.xpath("//*[contains(@alt, '')]/@alt")
print(text1)
text2 = doc.xpath("//div[contains(@class, 'a-column SsCol2')]//*[contains(@alt, '')]/@alt")
print(text2)

score 0 · Answer 2 · answered Dec 07 '18 at 01:52

I would definitely give Beautiful Soup a try:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')

Some common ways to navigate over the structure

soup.title
# <title>The Dormouse's story</title>

soup.title.name
# u'title'

soup.title.string
# u'The Dormouse's story'

soup.title.parent.name
# u'head'

soup.p
# <p class="title"><b>The Dormouse's story</b></p>

soup.p['class']
# u'title'

soup.a
# <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

soup.find_all('a')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.find(id="link3")
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

One common task is extracting all the URLs found within a page’s tags:

for link in soup.find_all('a'):
    print(link.get('href'))
# http://example.com/elsie
# http://example.com/lacie

Another common task is extracting all the text from a page:

print(soup.get_text())
# The Dormouse's story
#
# The Dormouse's story
#
# Once upon a time there were three little sisters; and their names were
# Elsie,
# Lacie and...

If you need anything else you might wanna check their documentation: Beautiful Soup

How do I get the text element of a hyperlink from a web page using Python?

2 Answers2