1
import requests

headers ={
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding":"gzip, deflate",
"Accept-Language":"en-US,en;q=0.5",
"Connection":"keep-alive",
"Host":"mcfbd.com",
"Referer":"https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx",
"User-Agent":"Mozilla/5.0(Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"}

a = requests.session()
soup = BeautifulSoup(a.get("https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx").content)

payload = {"ctl00$ContentPlaceHolder1$txtSearchHouse":"",
"ctl00$ContentPlaceHolder1$txtSearchSector":"",
"ctl00$ContentPlaceHolder1$txtPropertyID":"",
"ctl00$ContentPlaceHolder1$txtownername":"",
"ctl00$ContentPlaceHolder1$ddlZone":"1",
"ctl00$ContentPlaceHolder1$ddlSector":"2",
"ctl00$ContentPlaceHolder1$ddlBlock":"2",
"ctl00$ContentPlaceHolder1$btnFind":"Search",
"__VIEWSTATE":soup.find('input',{'id':'__VIEWSTATE'})["value"],
"__VIEWSTATEGENERATOR":"14039419",
"__EVENTVALIDATION":soup.find("input",{"name":"__EVENTVALIDATION"})["value"],
"__SCROLLPOSITIONX":"0",
"__SCROLLPOSITIONY":"0"}

b = a.post("https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx",headers = headers,data = payload).text
print(b)

above is my code for this website.

https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx

I checked firebug out and these are the values of the form data. however doing this:

b = requests.post("https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx",headers = headers,data = payload).text
print(b)

throws this error:

[ArgumentException]: Invalid postback or callback argument

is my understanding of submitting forms via request correct?

1.open firebug

2.submit form

3.go to the NET tab

4.on the NET tab choose the post tab

5.copy form data like the code above

I've always wanted to know how to do this. I could use selenium but I thought I'd try something new and use requests

Zion
  • 1,570
  • 6
  • 24
  • 36

1 Answers1

2

The error you are receiving is correct because the fields like _VIEWSTATE (and others as well) are not static or hardcoded. The proper way to do this is as follows:

Create a Requests Session object. Also, it is advisable to update it with headers containing USER-AGENT string - 

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36",}`
s = requests.session()

Navigate to the specified url - 

r = s.get(url)

Use BeautifulSoup4 to parse the html returned - 

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.content, 'html5lib')

Populate formdata with the hardcoded values and dynamic values - 

formdata = {
   '__VIEWSTATE': soup.find('input', attrs={'name': '__VIEWSTATE'})['value'],
   'field1': 'value1'
}

Then send the POST request using the session object itself - 

s.post(url, data=formdata)
Vikas Ojha
  • 6,742
  • 6
  • 22
  • 35