1

I want to get the source code only of a section from website instead of whole page and then parsing out the section, as it will be faster than loading whole page and then parsing. I tried passing the section link as url parameter but still getting whole page.

 url = 'https://stackoverflow.com/questions/19012495/smooth-scroll-to-div-id-jquery/#answer-19013712'
 response = requests.get(url)
 print(response.text)
om_prakash
  • 70
  • 9
  • 1
    Not possible, HTTP(S) protocol does not allow that. – Błotosmętek May 23 '20 at 07:53
  • 3
    The server sends you the whole html file anyway, if you have control over the server, then you can use query params to pre-filter the html file before it's sent at the server side – kareem_emad May 23 '20 at 07:54

2 Answers2

0

You cannot get specific section directly with requests api, but you can use beautifulsoup for that purpose. A small sample is given by dataquest website:

import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(page.content, 'html.parser')
page = requests.get("http://dataquestio.github.io/web-scraping-pages/ids_and_classes.html")

page.content

Running the above script will output this html String.

<html>
<head>
<title>A simple example page
</title>
</head>
<body>
<div>
<p class="inner-text first-item" id="first">
First paragraph.
</p><p class="inner-text">
Second paragraph.
</p></div>
<p class="outer-text first-item" id="second"><b>
First outer paragraph.
</b></p><p class="outer-text"><b>
Second outer paragraph.
</b>
</p>
</body>
</html>

You can get specific section by finding it through tag type, class or id.

By tag-type:

soup.find_all('p')

By class:

soup.find_all('p', class_='outer-text')

By Id:

soup.find_all(id="first")
Shubham Agrawal
  • 104
  • 10
0

HTTPS will not allow you to do that.

You can use the Stackoverflow API instead. You can pass the answer id 19013712. And thus only get that specific answer via the API.

Note, you may still have to register for an APP key

Abhishek J
  • 2,386
  • 2
  • 21
  • 22
  • I want to use the concept at some other place, which don't have any API. In the ques, I stated stackoverflow for example case only. – om_prakash Jul 17 '20 at 14:08