BeautifulSoup - modifying all links in a piece of HTML?

Question

I need to be able to modify every single link in an HTML document. I know that I need to use the SoupStrainer but I'm not 100% positive on how to implement it. If someone could direct me to a good resource or provide a code example, it'd be very much appreciated.

Thanks.

score 48 · Accepted Answer · edited Feb 09 '22 at 05:53

48

Maybe something like this would work? (I don't have a Python interpreter in front of me, unfortunately)

from bs4 import BeautifulSoup
soup = BeautifulSoup('<p>Blah blah blah <a href="http://google.com">Google</a></p>')
for a in soup.findAll('a'):
  a['href'] = a['href'].replace("google", "mysite")

result = str(soup)

edited Feb 09 '22 at 05:53

nude

35
6

answered Jan 20 '09 at 03:02

Lusid

4,518
1
24
24

score 32 · Answer 2 · answered Jan 20 '09 at 03:09

32

from BeautifulSoup import BeautifulSoup
soup = BeautifulSoup('<p>Blah blah blah <a href="http://google.com">Google</a></p>')
for a in soup.findAll('a'):
    a['href'] = a['href'].replace("google", "mysite")
print str(soup)

This is Lusid's solution, but since he didn't have a Python interpreter in front of him, he wasn't able to test it and it had a few errors. I just wanted to post the working condition. Thank's Lusid!

answered Jan 20 '09 at 03:09

Evan Fosmark

98,895
36
105
117

4

You probably want to check the crappy-HTML edge case where the a element you're testing doesn't have an href. – Robert Rossney Jan 20 '09 at 03:16
@Robert, yes you're right. I'll be sure to do so. Thanks for the heads up. – Evan Fosmark Jan 20 '09 at 03:34
@Evan, glad I was able to at least help you get there. My Python is a tad bit on the rusty side. :) – Lusid Jan 20 '09 at 03:38
2

`soup.prettify()` is easier for human eyes than `str(soup)`. – jfs Jan 20 '09 at 04:47
Actually, that's a pretty common situation for named anchor tags: – Peter Rowell Jan 20 '09 at 06:51
5

@PeterRowell: `soup.findAll('a', href=True)` ensures that `a['href']` works – jfs Mar 22 '14 at 04:03

score 8 · Answer 3 · answered Mar 22 '14 at 00:03

8

I tried this and it worked, it's easier to avoid using regexp for matching each 'href':

from bs4 import BeautifulSoup as bs
soup = bs(htmltext)
for a in soup.findAll('a'):
    a['href'] = "mysite"

Check it out, on bs4 docs.

answered Mar 22 '14 at 00:03

Aziz Alto

19,057
5
77
60

BeautifulSoup - modifying all links in a piece of HTML?

3 Answers3

Linked

Related