Getting all HTML from requests.get()

Question

I just started with web scraping with Python and hit the wall. I am using the requests library to get the HTML code from a website. For example, the Google search result website: "https://www.google.com/?gws_rd=ssl#q=ball"

When I hit F12 and check the HTML, it looks different than with:

site = requests.get("https://www.google.com/?gws_rd=ssl#q=ball")
print(site.text)

with requests.get, text is much shorter and not all information is visible (it starts with !doctype, however). Because of that I am unable to work with this HTML.

Can you tell me where the mistake is?

This is actually an exercise from the book "Automate the boring stuff with Python". The task is to search for some item Google and then find few first results with HTML locators. I cannot do it because when I use requests.get() I cannot see any objects for links in the HTML code.

Apart from the fact certain factors influence what google returns, there is also content dynamically created with js — Padraic Cunningham, Sep 07 '16 at 22:04
Looks like this might also be helpful - http://stackoverflow.com/questions/22623798/google-search-with-python-requests-library — Sandwich Heat, Sep 07 '16 at 22:07

score 2 · Accepted Answer · answered Sep 07 '16 at 22:18

The HTML you see using the browser's development tools is what the browser is currently working with. This includes any changes performed via Javascript. The data you are getting when using Requests is before any Javascript has operated on the page. (Note that Requests doesn't process Javascript so you will be unable to acquire a javascript processed page using just Requests.)

If you're specifically looking to scrape Google Search, use a url like https://www.google.com/search?q=test. This particular url is for Google's non-javascript site. Keep in mind that Google (and most other sites) doesn't appreciate scraping so you may run into other issues when doing so.

And that works! Thank you very much. The problem was that I have used: "https://www.google.com/?gws_rd=ssl#q=" as a base (I have copied it directly from my browser). Instead I should use "https://www.google.com/search?q=" like you said. Thank you again. — Maciejjy, Sep 07 '16 at 22:28

score 0 · Answer 2 · answered Sep 07 '16 at 22:04

0

Some HTML elements are generated by JavaScript.

Use "show source code" from your browser to see the original code. It must be similar to the Request response text.

answered Sep 07 '16 at 22:04

Laurent LAPORTE

21,958
6
58
103

Is there a simpler way to do this in the Chrome developer tools itself? – Raunak Thomas Jun 10 '18 at 17:32

score 0 · Answer 3 · answered Oct 27 '21 at 08:15

It's probably because there's no user-agent being passed into requests headers thus when no user-agent is specified while using requests library, it defaults to python-requests and Google understands that it's a bot/script, then it blocks a request (or whatever it does) and you receive a different HTML (with some sort of an error) with different CSS selectors. Check what's your user-agent.

Pass user-agent:

headers = {
    'User-agent':
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582'
}

requests.get('URL', headers=headers)

Alternatively, you can achieve the same thing by using Google Organic Results API from SerpApi. It's a paid API with a free plan.

The difference is that you don't have to spend time trying to bypass blocks from Google and figuring out why certain things don't work as they should. Instead, you need to focus on the data you want to extract from the structured JSON. Check out the playground.

Disclaimer, I work for SerpApi.

Getting all HTML from requests.get()

3 Answers3