0

I am trying to POST data to ASPX form using the Python requests library. But I'm not able to get it right. My plan is to scrape user data, e.g. name, address, etc.

Here is the web form: https://www.icsi.in/student/Members/MemberSearch.aspx - I am searching only with one input: "CP Number" = 16803.

Here is my code:

import requests
from bs4 import BeautifulSoup

def make_request_to_url():
    url = "https://www.icsi.in/student/Members/MemberSearch.aspx"
    post_data = {
        'dnn$ctr410$MemberSearch$txtFirstName': None,
        'dnn$ctr410$MemberSearch$txtLastName': None,
        'dnn$ctr410$MemberSearch$ddlMemberType': None,
        'dnn$ctr410$MemberSearch$txtMembershipNumber': None,
        'dnn$ctr410$MemberSearch$txtCpNumber': 16803,
        'dnn$ctr410$MemberSearch$txtCity': None,
        'dnn$ctr410$MemberSearch$txtOrganisation': None,
        'dnn$ctr410$MemberSearch$txtAddress2': None,            
        'dnn$ctr410$MemberSearch$txtAddress3': None,
        'dnn$ctr410$MemberSearch$txtEmail': None
    }
    r = requests.post(url, data = post_data)
    soup = BeautifulSoup(r.content,'lxml')
    head_name =  soup.find_all("div",{"class":"name_head"})
    print "head_name",head_name

I get content but not the required data line with name, address, etc. to scrape. What to do?

Tomalak
  • 332,285
  • 67
  • 532
  • 628
ashutosh
  • 1
  • 4
  • i found this useful link: https://stackoverflow.com/questions/43149898/login-using-python-request-module-on-a-aspx-webpage/43188546 – ashutosh Aug 14 '18 at 03:06
  • instead of data, use json. like r = requests.post(url, json= post_data). This might help. – SanthoshSolomon Aug 14 '18 at 05:01
  • not working with r = requests.post(url, json= post_data) – ashutosh Aug 14 '18 at 13:41
  • Possible duplicate of [How to send a "multipart/form-data" with requests in python?](https://stackoverflow.com/questions/12385179/how-to-send-a-multipart-form-data-with-requests-in-python) – Tomalak Aug 18 '18 at 07:12
  • Your web form requires the the `multipart/form-data` content type, but the `requests` module sends HTTP POST requests as `application/x-www-form-urlencoded` by default. See the [duplicate question](https://stackoverflow.com/q/12385179/18771) for the way to send the form in the server expects. – Tomalak Aug 18 '18 at 07:15

0 Answers0