0

I want to get the INSPECT ELEMENT data of a website. Let's say Truecaller. So that i can get the Name of the person who's mobile number I searched. But whenever i make a python script it gives me the PAGE SOURCE that does not contain the required information.

Kindly help me. I am a beginner so kindly excuse me of any mistake in the question.

Manish Devgan
  • 3
  • 1
  • 2
  • Try this: `print requests.get('http://stackoverflow.com/questions/42757866/how-do-i-get-the-data-of-a-website-as-shown-in-inspect-element-and-not-in-view-p').text()` – TheChetan Mar 13 '17 at 07:05
  • Try this post: [How to get data from inspect element of a webpage using Python] (http://stackoverflow.com/questions/25027339/how-to-get-data-from-inspect-element-of-a-webpage-using-python?rq=1) – raviriley Mar 13 '17 at 07:40

3 Answers3

3

TL;DR: Use Selenium (and PhantomJS)

The view page source will give you the html that was loaded when you made a request for the page (which is most likely what you are getting when you make a request from python.

Since nowadays a lot of pages load things and modify the DOM after the initial html was loaded, you will not get most of the information you want just by looking into that initial response. To get the inspect element information you will need some sort of web browser to actually go to the page, wait for the information you want to load, and then use it. However you still want to do this in your python script.

Enter selenium, which is a tool for browser automation (mostly used for testing webpages). You can create a python script that opens a browser page and executes whatever code you write for it to do (even wait for a while and search for an after load DOM element!). Your script will still open a browser (which is kind of weird I would guess).

Enter PhantomJS, another library that you can use to have a headless browser to do all your web testing without having to rely on the actual browser UI.

Using selenium only you might achieve your goals, but with phantomjs you can do that in an even cleaner way! Good Luck.

Gustavo Guimaraes
  • 72
  • 7
0

INSPECT ELEMENT and VIEW PAGE SOURCE are not the same.

View source shows you the original HTML source of the page. When you view source from the browser, you get the HTML as it was delivered by the server, not after javascript does its thing.

The inspector shows you the DOM as it was interpreted by the browser. This includes for example changes made by javascript which cannot be seen in the HTML source.

shubham agrawal
  • 3,435
  • 6
  • 20
  • 31
0

what you see in the element inspector is not the source-code anymore. You see a javascript manipulated version.

Instead of trying to execute all the scripts on your own which may lead into multiple problems like cross origin security and so on,

search the network tab for the actual search request and its parameters. Then request the data from there, that is the trick.

Also it seems like you need to be logged in to search on the url you provided so you need to eventually adapt cookie/session/header and stuff, just like a request from your browser would.

So what i want to say is, better analyse where the data you look for is coming from if it is not in the source

john Smith
  • 17,409
  • 11
  • 76
  • 117