In mechanize we click links either by using follow_link or click_link. Is there a similar kind of thing in beautiful soup to click a link on a web page?
Asked
Active
Viewed 6.1k times
2 Answers
21
BeautifulSoup is an HTML parser.
Further discussion really depends on the concrete situation you are in and the complexity of the particular web page.
If you need to interact with a web-page: submit forms, click buttons, scroll etc - you need to use a tool that utilizes a real browser, like selenium.
In certain situations, for example, if there is no javascript involved in submitting a form, mechanize would also work for you.
And, sometimes you can handle it by simply following the link with urllib2 or requests.
alecxe
- 462,703
- 120
- 1,088
- 1,195
-
Thanks. I was wondering if it is possible to modify the link using python as we do using inspect element in browsers? I tried using mechanize. The link got modified as it appeared after printing it but when clicked it still opened the same page. – nishantsingh May 15 '14 at 13:25
-
@user3286661 ok, could you show the code you are using? What is the purpose of it? Thanks. – alecxe May 15 '14 at 13:28
-
Oh yes! I have posted a question already. Here it is: http://stackoverflow.com/questions/23677442/changing-the-link-in-python-mechanize – nishantsingh May 15 '14 at 13:30
-
@user3286661 gotcha, will take a look at your particular mechanize issue. I also don't think there is smth to add for the current question. – alecxe May 15 '14 at 13:34
-
@alecxe can you please help out with a problem I have with beautiful soup? – Nobi Feb 16 '17 at 22:17
0
print(soup.find('h1',class_='pdp_product_title'))
does not give any result
<div class="pr2-sm css-1ou6bb2"><h2 class="headline-5-small pb1-sm d-sm-ib css-1ppcdci" data-test="product-sub-title">Women's Shoe</h2><h1 id="pdp_product_title" class="headline-2 css-zis9ta" data-test="product-title">Nike Air Force 1 Shadow</h1></div>
Muhammad Umer Lari
- 62
- 7