1

i have this small problem. i have written a curl script to first click a link and then fill up the form that the link provides.

the link is "parcel enquiry" on http://www.fikeandfike.com/propertytax/Grundy/MainMenu.aspx?c=32

and the form is on http://www.fikeandfike.com/propertytax/Grundy/Inquiry.aspx

but my problem is the script is only filling up the form and not doing anything with it

when i echo the result of curl it shows the form filled up but i want to see the result of form

can you please tell me why is this happening? can you suggest some trick to get past this problem?

user1511443
  • 175
  • 1
  • 11
  • Two thoughts that don't quite qualify as answers: 1) have you checked exactly what gets POSTed by a browser filling in that form, including e.g. the value of the Submit button itself, hidden fields, etc? 2) It looks like the site is using ASP.net, which is pretty resistant to this kind of "web scraping"/"spoofing", because it doesn't work like normal HTML forms. – IMSoP Aug 19 '12 at 20:04
  • @IMSoP yes all hidden fields are mentioned in the script.please tell how to find what all gets posted.this is unknown to me. i am wondering what happens when submit button is clicked. please visit the site and help me i will be very greatful – user1511443 Aug 19 '12 at 21:44
  • Browser extensions such as Firebug and URLParams can show you the exact data posted, and some let you edit them to see the effect of different values. Remember that some hidden fields may be populated with a different value on every page load. – IMSoP Aug 19 '12 at 22:39
  • @IMSoP thank you for the information. you are right this is ASP.net website and it is really resistant to scraping. i have done some findings and found that the value of event validation is required. do you think this can help me? – user1511443 Aug 19 '12 at 22:48

1 Answers1

0

As mentioned in the comments, the problem is that the site you are trying to "scrape" is built using ASP.net, which doesn't use form data in the conventional way.

I do know someone who successfully get this working, but it is fiddly. The key hidden fields to watch out for are:

  • __EVENTTARGET and __EVENTARGUMENT: These are set by Javascript when you submit the form, but always to the same value. If you use a browser debug tool (Firebug, URLParams, etc) to inspect the result of a real POST, you will be able to fill these in no problem.
  • __VIEWSTATE: This is generated afresh every time the page loads, so you'll need to scrape it out of the HTML of the loaded form each time you request it.
  • __EVENTVALIDATION: This field exists specifically to prevent the kind of spoofing you are attempting. Like __VIEWSTATE, it is regenerated on each page load, but its presence may mean you can only change one field at a time, since it will validate what you submit against the previous state.

I may be wrong on some of the specifics here, but that should give you somewhere to start...

IMSoP
  • 89,526
  • 13
  • 117
  • 169