2

I am new to ajax and javascript.
I am crawling a website in which I am able to fetch relevant piece of details with the help of XPath after downloading webpage using Python(urllib2/request/mechanize).

In the webpage these are some information that is visible only after clicking a link. And that link calls XHR to fetch details that I found out using firefox's web developer tools. (Ctrl+Shift+Q or Tools >> Web Developer >> Network) I am showing that link and its javascript attributes that I can see using Firefox web developer tool (Ctrl+Shift+C or Tools >> Web Developer >> Inspector) in attached image under thick black rectanguls.

Link and its attributes

I am also able to see ajax request url, headers, response and parameters though same firefox web developer tools. same image is visible at https://i.stack.imgur.com/9jhfr.png

headers,response and parameters

I am thinking that I have all the payload for POST request. How can I make http post call this using Python with the help of request/urllib2 etc? so in response I can see details which i show in webpage after clicking that link. Like
requests.get(url, data=<paramter_to_post which i can see in firefox>, headers=<request headers that I can see in firefox>)

I short
How to simulate ajax call using python? or how to fetch infromation which I see after clicking that link?

I can automate this task using Selenium/PhantomJS or other headless browser. but I want to solve this using HTTP Post and Get which is exactly happening in Firefox when I click the link.

Alok
  • 7,734
  • 8
  • 55
  • 100

1 Answers1

1

Well first of all install firebug (https://getfirebug.com/)

Then go to your page, launch the firebug and go to the Net tab in firebug panel. Now in this tab you can see all of the get/posts calls your firefox sends to the website.

enter image description here

Now you can click around, refresh the page and see what calls are being made. In your case click the button and you'll see new calls are being made, you will probably find it at html tab.

There you can find a call once you click on it you'll see the request and other details.

make a dictionary of the parameters and attach it to "data=" in your post. you can also make headers by making a dictionary of it and attaching it to the "headers=" in your post.

Take note: a lot of websites use cookies to identify if the calls are made by a legit browser so it might require quite a bit of fiddling with cookies and urls. !

it's hard to give examples if you don't give us the website.

Granitas
  • 302
  • 1
  • 4
  • 11
  • no need to install firebug. I am able to see everything with firefox's inbuilt web developer tools. I can see all the details of XHR call with `Ctrl+Shift+Q`. then clicking XHR tab. there I can see all the request headers, response headers, cookies, parameters etc. so its not the problem at all. my problem is how to make HTTP get request with all the information? because in parameters's value there are man made functions. you can see the values at http://i.stack.imgur.com/9jhfr.png – Alok May 08 '14 at 10:18
  • @AlokSinghMahor have you tried just copying all of it and using it as "data=" in your requests.post call? also turning it into dict and then using it as "data=" value? "callcount": "1" instead of callcount=1 in your case – Granitas May 08 '14 at 12:36
  • no did not try. because I guess I will not success because values have mad made functions. like in 6th line of parameters in http://i.stack.imgur.com/9jhfr.png – Alok May 08 '14 at 12:44
  • well I don't really have any experience with AJAX but if it says it's sending this particular data it's not sending anything else. It seems like it isn't sending the whole function just the name of it. I really have no clue, but trying it shouldn't take you more than few minutes, so why not give it a go? – Granitas May 08 '14 at 14:11