How do I send a POST request to the page after it has been opened using Python?

Question

I want to send a POST request to the page after opening it using Python (using urllib2.urlopen). Webpage is http://wireless.walmart.com/content/shop-plans/?r=wm

Code which I am using right now is:

url = 'http://wireless.walmart.com/content/shop-plans/?r=wm'
user_agent = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1;Trident/5.0)'   
values = {'carrierID':'68',
'conditionToType':'1',
'cssPrepend':'wm20',
'partnerID':'36575'}
headers = { 'User-Agent' : user_agent }
data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
page = response.read()
walmart = open('Walmart_ContractPlans_ATT.html','wb')
walmart.write(page)

This is giving me page which opens by default, after inspecting the page using Firebug I came to know that carrierID:68 is sent when I click on the button which sends this POST request.

I want to simulate this browser behaviour.

Please help me in resolving this.

You might check http://stackoverflow.com/questions/6348499/making-a-post-call-instead-of-get-using-urllib2?lq=1 — Adam Morris, May 17 '13 at 13:05
"when I click on the button which sends this POST request", wchih button are you referring to? — elyase, May 17 '13 at 13:13
So do you want to extract Plans for a specific carrier or do you want to extract the carrier's IDs from the main page? — elyase, May 17 '13 at 13:30

score 1 · Accepted Answer · edited Feb 03 '21 at 20:21

1

For webscraping I prefer to use requests and pyquery. First you download the data:

import requests
from pyquery import PyQuery as pq

url = 'http://wireless.walmart.com/content/getRatePlanInfo'
payload = {'carrierID':68, 'conditionToType':1, 'cssPrepend':'wm20'}
r = requests.post(url, data=payload)
d = pq(r.text)

After this you proceed to parse the elements, for example to extract all plans:

plans = []
plans_selector = '.wm20_planspage_planDetails_sub_detailsDiv_ul_li'
plans = d(plans_selector).each(lambda i, n: plans.append(pq(n).text()))

Result:

 ['Basic 200',
 'Simply Everything',
 'Everything Data 900',
 'Everything Data 450',
 'Talk 450',
 ...

edited Feb 03 '21 at 20:21

DisappointedByUnaccountableMod

6,656
4
18
22

answered May 17 '13 at 21:44

elyase

39,479
12
112
119

thanks for the answer, couldn't accept earlier because of the weekend. – atams May 20 '13 at 06:17
How did you get the result? When I am printing `plans` I am getting this sort of o/p `[, , ` – atams May 20 '13 at 13:22
add this `from __future__ import print_function`, but honestly I am not sure why that is needed. I will look into it when I have some time. – elyase May 20 '13 at 13:43
Ok I updated my answer, I changed `n.text` to `pq(n).text()`, it should work now without the `print_function` thing. – elyase May 20 '13 at 13:51
How did you find this link `http://wireless.walmart.com/content/getRatePlanInfo` as the link I gave in the question was different. – atams May 30 '13 at 05:28

score 0 · Answer 2 · answered May 17 '13 at 12:58

0

I recommend looking at a browser emulator like mechanize, rather than trying to do this with raw HTTP requests.

answered May 17 '13 at 12:58

Ben Butler-Cole

2,011
1
17
23

I only know basic form filling using Mechanize, when I opened this page using it and checked for br.forms() I couldn't find any which can be filled and submitted. Can we send POST request also using Mechanize? If yes, then how? – atams May 17 '13 at 13:21

How do I send a POST request to the page after it has been opened using Python?

2 Answers2