1

I have an .aspx page hosted locally that contains one text field where a user can enter his/her ID and get some information.

I understand how HttpWebRequest works and have used it many times to POST to many PHP and JSP pages, but now the problem is that the ASP .NET page that I am trying to POST to has a lot of hidden fields like __VIEWSTATE that are dynamic and sent to the server each time the user enters his/her ID and presses the submit button.

How can I POST the values of those dynamic fields along with the ID?

The header looks like this (grabbed from the developer tools in Firefox):

__EVENTTARGET=&__EVENTARGUMENT=&__LASTFOCUS=&__VIEWSTATE=%2FwEPDwUJMjk5MjYyMjY5D2QWAgIDD2QWBAIHDxcAZAIJDxcGBQZIZWlnaHQHAAAAAAA8kEAFBHJzSUQFFENyeXN0YWxSZXBvcnRTb3VyY2UxBQ9SZXBvcnRWaWV3U3RhdGUXBgUKRGVzaWduTW9kZWgFElBhZ2VSZXF1ZXN0Q29udGV4dBcEBQpQYWdlTnVtYmVyAgEFDVByb21wdGluZ0luZm8UKVhTeXN0ZW0uQnl0ZSwgbXNjb3JsaWIsIFZlcnNpb249Mi4wLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj1iNzdhNWM1NjE5MzRlMDg5ygUDPAM%2FA3gDbQNsAyADdgNlA3IDcwNpA28DbgM9AyIDMQMuAzADIgM%2FAz4DDQMKAzwDQQNyA3IDYQN5A08DZgNQA2EDcgNhA20DZQN0A2UDcgNGA2kDZQNsA2QDIAN4A20DbANuA3MDOgN4A3MDaQM9AyIDaAN0A3QDcAM6Ay8DLwN3A3cDdwMuA3cDMwMuA28DcgNnAy8DMgMwAzADMQMvA1gDTQNMA1MDYwNoA2UDbQNhAy0DaQNuA3MDdANhA24DYwNlAyIDIAN4A20DbANuA3MDOgN4A3MDZAM9AyIDaAN0A3QDcAM6Ay8DLwN3A3cDdwMuA3cDMwMuA28DcgNnAy8DMgMwAzADMQMvA1gDTQNMA1MDYwNoA2UDbQNhAyIDPgMNAwoDIAMgAzwDUANhA3IDYQNtA2UDdANlA3IDRgNpA2UDbANkAyADXwNEA2UDZgNhA3UDbAN0A1YDYQNsA3UDZQNEA2kDcwNwA2wDYQN5A1QDeQNwA2UDPQMiAzADIgMgA1ADYQNyA2EDbQNlA3QDZQNyA0YDaQNlA2wDZANOA2EDbQNlAz0DIgNAA1IDZQNnA2kDcwN0A2UDcgNhA3QDaQNvA24DbgN1A20DYgNlA3IDIgMgA1ADcgNvA20DcAN0A1QDZQN4A3QDPQMiAyIDIANSA2UDcANvA3IDdANOA2EDbQNlAz0DIgMiAyADRQNkA2kDdANNA2EDcwNrAz0DIgMiAyADSANhA3MDQwN1A3IDcgNlA24DdANWA2EDbAN1A2UDPQMiA3QDcgN1A2UDIgMgA0UDbgNhA2IDbANlA0EDbANsA28DdwNFA2QDaQN0A2kDbgNnA0QDZQNmA2EDdQNsA3QDVgNhA2wDdQNlAz0DIgN0A3IDdQNlAyIDIANFA24DYQNiA2wDZQNBA2wDbANvA3cDTQN1A2wDdANpA3ADbANlA1YDYQNsA3UDZQM9AyIDZgNhA2wDcwNlAyIDIANFA24DYQNiA2wDZQNOA3UDbANsA1YDYQNsA3UDZQM9AyIDdANyA3UDZQMiAyADVQNQA2EDcgNhA20DZQN0A2UDcgNWA2EDbAN1A2UDSwNpA24DZAM9AyIDMAMiAyADVQNQA2EDcgNhA20DZQN0A2UDcgNUA3kDcANlAz0DIgMwAyIDIANVA0QDaQNzA2MDcgNlA3QDZQNPA3IDUgNhA24DZwNlA0sDaQNuA2QDPQMiAzADIgMgA1UDRANlA2YDYQN1A2wDdANWA2EDbAN1A2UDUwNvA3IDdANPA3IDZANlA3IDPQMiAzADIgMgA1UDRANlA2YDYQN1A2wDdANWA2EDbAN1A2UDUwNvA3IDdANNA2UDdANoA28DZAM9AyIDMAMiAz4DDQMKAyADIAMgAyADPANQA2EDcgNhA20DZQN0A2UDcgNDA3UDcgNyA2UDbgN0A0QDaQNzA2MDcgNlA3QDZQNWA2EDbAN1A2UDIANVA0sDaQNuA2QDPQMiAzADIgM%2BAw0DCgMgAyADIAMgAyADIAM8A0QDZQNzA2MDcgNpA3ADdANpA28DbgMgAy8DPgMNAwoDIAMgAyADIAMgAyADPANWA2EDbAN1A2UDUwN0A3IDaQNuA2cDPgMxAzEDMwMxAzADNgM5AzQDPAMvA1YDYQNsA3UDZQNTA3QDcgNpA24DZwM%2BAw0DCgMgAyADIAMgAzwDLwNQA2EDcgNhA20DZQN0A2UDcgNDA3UDcgNyA2UDbgN0A0QDaQNzA2MDcgNlA3QDZQNWA2EDbAN1A2UDPgMNAwoDIAMgAzwDLwNQA2EDcgNhA20DZQN0A2UDcgNGA2kDZQNsA2QDPgMNAwoDPAMvA0EDcgNyA2EDeQNPA2YDUANhA3IDYQNtA2UDdANlA3IDRgNpA2UDbANkAz4FFUlzTGFzdFBhZ2VOdW1iZXJLbm93bmcFDkxhc3RQYWdlTnVtYmVyAgIFB0ZhY3RvcnkFlgFDcnlzdGFsRGVjaXNpb25zLlJlcG9ydFNvdXJjZS5SZXBvcnRTb3VyY2VGYWN0b3J5LENyeXN0YWxEZWNpc2lvbnMuUmVwb3J0U291cmNlLCBWZXJzaW9uPTEwLjIuMzYwMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPTY5MmZiZWE1NTIxZTEzMDQFB1JlZnJlc2hoBQlSZXBvcnRVUkllBQlScHRTb3VyY2UFN0NyeXN0YWxEZWNpc2lvbnMuUmVwb3J0U291cmNlLk5vbkhUVFBDYWNoZWRSZXBvcnRTb3VyY2UFA2Nzc2UFBEJBU0UPFgYeBkhlaWdodBsAAAAAADyQQAEAAAAeBVdpZHRoGwAAAAAAKIhAAQAAAB4EXyFTQgKAA2QFBVdpZHRoBwAAAAAAKIhAFgICAg8XAQUCYnMC3%2F7%2F%2Fw8WAgILDxAPFgIeC18hRGF0YUJvdW5kZ2QPFgFmFgEQBQtNYWluIFJlcG9ydAWoA0FBRUFBQUQvLy8vL0FRQUFBQUFBQUFBRUFRQUFBQnhUZVhOMFpXMHVRMjlzYkdWamRHbHZibk11U0dGemFIUmhZbXhsQndBQUFBcE1iMkZrUm1GamRHOXlCMVpsY25OcGIyNElRMjl0Y0dGeVpYSVFTR0Z6YUVOdlpHVlFjbTkyYVdSbGNnaElZWE5vVTJsNlpRUkxaWGx6QmxaaGJIVmxjd0FBQXdNQUJRVUxDQnhUZVhOMFpXMHVRMjlzYkdWamRHbHZibk11U1VOdmJYQmhjbVZ5SkZONWMzUmxiUzVEYjJ4c1pXTjBhVzl1Y3k1SlNHRnphRU52WkdWUWNtOTJhV1JsY2dqc1VUZy9Bd0FBQUFvS0N3QUFBQWtDQUFBQUNRTUFBQUFRQWdBQUFBTUFBQUFHQkFBQUFBNU1ZWE4wVUdGblpVNTFiV0psY2dZRkFBQUFGVWx6VEdGemRGQmhaMlZPZFcxaVpYSkxibTkzYmdZR0FBQUFDbEJoWjJWT2RXMWlaWElRQXdBQUFBTUFBQUFJQ0FJQUFBQUlBUUVJQ0FFQUFBQUxnFgFmZBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WCwUgQ3J5c3RhbFJlcG9ydFZpZXdlcjEkY3RsMDIkY3RsMDAFIENyeXN0YWxSZXBvcnRWaWV3ZXIxJGN0bDAyJGN0bDAxBSBDcnlzdGFsUmVwb3J0Vmlld2VyMSRjdGwwMiRjdGwwMgUgQ3J5c3RhbFJlcG9ydFZpZXdlcjEkY3RsMDIkY3RsMDMFIENyeXN0YWxSZXBvcnRWaWV3ZXIxJGN0bDAyJGN0bDA0BSBDcnlzdGFsUmVwb3J0Vmlld2VyMSRjdGwwMiRjdGwwNQUgQ3J5c3RhbFJlcG9ydFZpZXdlcjEkY3RsMDIkY3RsMDYFIENyeXN0YWxSZXBvcnRWaWV3ZXIxJGN0bDAyJGN0bDA3BSBDcnlzdGFsUmVwb3J0Vmlld2VyMSRjdGwwMiRjdGwxMAUgQ3J5c3RhbFJlcG9ydFZpZXdlcjEkY3RsMDIkY3RsMTIFIENyeXN0YWxSZXBvcnRWaWV3ZXIxJGN0bDAyJGN0bDE0jI2pRkImY%2FKY6U%2Fd3%2FdGqZgVpS8%3D&__PREVIOUSPAGE=JxnGN13k9OAmlpqF3Z8SW9w-B2b4vBfG7WIz-XQ5-GZ_4P1GIClbgn0lEpxdarTxDVAekfH3QhOqCwpdtv2ml9I79qLZb5EFP7Nh-zun2qPkprmn0&TextBox1=11310694&Button1=Show&CrystalReportViewer1%24ctl02%24ctl09=&CrystalReportViewer1%24ctl02%24ctl11=AAEAAAD%2F%2F%2F%2F%2FAQAAAAAAAAAEAQAAABxTeXN0ZW0uQ29sbGVjdGlvbnMuSGFzaHRhYmxlBwAAAApMb2FkRmFjdG9yB1ZlcnNpb24IQ29tcGFyZXIQSGFzaENvZGVQcm92aWRlcghIYXNoU2l6ZQRLZXlzBlZhbHVlcwAAAwMABQULCBxTeXN0ZW0uQ29sbGVjdGlvbnMuSUNvbXBhcmVyJFN5c3RlbS5Db2xsZWN0aW9ucy5JSGFzaENvZGVQcm92aWRlcgjsUTg%2FAwAAAAoKCwAAAAkCAAAACQMAAAAQAgAAAAMAAAAGBAAAAA5MYXN0UGFnZU51bWJlcgYFAAAAFUlzTGFzdFBhZ2VOdW1iZXJLbm93bgYGAAAAClBhZ2VOdW1iZXIQAwAAAAMAAAAICAIAAAAIAQEICAEAAAAL&CrystalReportViewer1%24ctl02%24ctl13=&CrystalReportViewer1%24ctl02%24ctl15=100

On the contrary, all the information I have available to POST is TextBox1=11310694

If I use the same header as above in my code, I get a 500 INTERNAL SERVER ERROR.

The C# code that I am using to fetch the page:

try{
ASCIIEncoding encoding = new ASCIIEncoding();
string post_data = "TextBox1=123456";
byte[] data = encoding.GetBytes(post_data);
WebRequest share_url = WebRequest.Create("http://172.19.2.6:8080/webopac/html/memberlogin");
share_url.Method = "POST";
share_url.ContentLength = data.Length;
share_url.ContentType = "application/x-www-form-urlencoded";
Stream stream = share_url.GetRequestStream();
stream.Write(data, 0, data.Length);
stream.Close();
WebResponse resp = share_url.GetResponse();
stream = resp.GetResponseStream();
StreamReader sr = new StreamReader(stream);
var oup = sr.ReadToEnd();
Console.WriteLine(oup);
Console.ReadLine();
}   
catch (Exception err)
{
Console.WriteLine("ERROR: " + err.Message);
}

How can I POST those VIEWSTATE fields?

Adaline Simonian
  • 4,596
  • 2
  • 24
  • 35
sudo
  • 906
  • 2
  • 14
  • 32

1 Answers1

1

There are essentially two ways of going about what you want to accomplish.

The best way: APIs and Web Services

If you are trying to retrieve information from an ASP .NET web application that is under your control, it is best to use some form of API rather than communicating directly with pages meant to be viewed in browser.

This does several things (among others):

  1. You save yourself a lot of code.

    Instead of needing to parse/generate/submit form data which is really only meant to assist in giving the user an interactive webpage, the code you write only has to handle sending and receiving the information you want.

  2. You save yourself bandwidth.

    This also saves you bandwidth, as the extra data (that your client application does not need) does not have to be transmitted back and forth. This in turn makes your client application faster, and also saves the web server the extra processing time it would need to render an ASP .NET WebForm.

  3. You avoid extra, unnecessary error checks and instability.

    When you communicate with a web page meant for the end-user, you may often need to implement many error checks in the event that the web page doesn't return what you want. In the case, for example, of an error on the page that still results in a 200 OK HTTP status code, you may need to parse the page for error text. This takes up processing time, code, and is prone to breaking when changes are made to the page.

To avoid all these problems, you can set up an API or web service that will act as a special point of communication which your web application uses to talk to your client application. You send only the data you need, simplify your code, and make your application more reliable.

What it looks like

If you just query a WebForm or other web page for the form values you need, your program will have to sort through all this:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" >
  <head><script type="text/javascript" charset="UTF-8">
(function (global) {
    global.outsystems = global.outsystems || {};
    global.outsystems.internal = global.outsystems.internal || {};
})(this);

outsystems.internal.timerspolyfillResponseStart = new Date().getTime();
outsystems.internal.beaconURL = '/PerformanceProbe/Beacon.aspx';
</script><title>
    Login
</title><link rel="apple-touch-icon" href="apple-touch-icon.png" /><link type="image/x-icon" rel="shortcut icon" href="/WodifyUI/img/favicon.png" /><script type="text/javascript" src="/wodify/js/libs/modernizr-2.5.3.min.js"></script><script type="text/javascript" src="/WodifyUI/js/pusher.min.js"></script>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="Content-Script-Type" content="text/javascript" /><meta http-equiv="Content-Style-Type" content="text/css" />
</head>
  <body>
    <form name="WebForm1" method="post" action="Login.aspx" onsubmit="javascript:return WebForm_OnSubmit();" id="WebForm1">
<div>
<input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value="" />
<input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value="" />
<input type="hidden" name="__OSVSTATE" id="__OSVSTATE" value="eNp1jcEKgkAURXOwFi1yE1MRzB+0SGrhUloJJSK0lodvNEFnZGbM+vpSQaigzYN3D/eel+WgM53vXPfged7ePTp0xtaFiEBB5ZelbP3GyJhjoXhqcrZpzVVzFULFA1E3JrmAQDBSPQcWgdatVPjLKCGdiJJsPPb4977l6KvrskjBFFIESCy2uoM6px9pzNNuXNMJ2/5BCZdZjkjsfv3L6VDCFkPrVHIQIX+YG+Ib9DFO1Q==" />
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="" />
</div>

<script type="text/javascript">
//<![CDATA[
var theForm = document.forms['WebForm1'];
if (!theForm) {
    theForm = document.WebForm1;
}
function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
...

If you use an API or a web service, you end up something like the code below, which is much easier to work with:

<?xml version="1.0" encoding="UTF-8"?>
<member>
    <id>12345</id>
    <name>Jane Doe</name>
    <dob>01/01/1980</dob>
</member>

Option 1: Small-scale

Personally, I feel that this is the best option for you, as it seems you only intend to transmit a small bit of data.

If you only have a couple pieces of data that you need to communicate, and you generally don't rely on communicating with your web application from a client application, you may consider using a Generic Handler (.ashx). This is a quick and easy way of transferring data from your web application to your client application.

Pros:

  1. Very easy to implement, especially for developers not as familiar with MVC and RESTful services.
  2. Requires very little time to complete.

Cons:

  1. Not easily scalable. If you plan on increasing the amount of data you need to communicate between server and client, consider a full-fledged web service or API.
  2. Can lead to unstandardized code without care. In other words, because there is no set standard way of communicating with Generic Handlers, if you don't keep to a certain set of rules (such as, every time I need to access a member's name, I'll use a Name field, and for their date of birth, DOB), you may end up with a lot of code that is very different from one another and hard to maintain when changes are needed.

Here's some pseudocode that shows you just how easy it is to create a Generic Handler:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web;
using System.Xml;
using MyDatabase;

namespace MyWebApplication
{
    public class MemberHandler : IHttpHandler
    {

        public void ProcessRequest(HttpContext context)
        {
            Member requestedMember = null;
            Exception error = null;

            try
            {
                requestedMember = MyDatabaseClient.GetMemberByID(int.Parse(context.Request.QueryString["id"]));
            }
            catch (Exception ex)
            {
                error = ex;
            }

            context.Response.ContentType = "text/xml";
            context.Response.ContentEncoding = Encoding.UTF8;

            using (XmlTextWriter xmlTextWriter = new XmlTextWriter(context.Response.OutputStream, Encoding.UTF8) { Formatting = Formatting.Indented })
            {
                xmlTextWriter.WriteStartDocument();
                if (requestedMember != null)
                {
                    xmlTextWriter.WriteStartElement("member");

                    xmlTextWriter.WriteStartElement("id");
                    xmlTextWriter.WriteValue(requestedMember.ID);
                    xmlTextWriter.WriteEndElement();

                    xmlTextWriter.WriteStartElement("name");
                    xmlTextWriter.WriteValue(requestedMember.Name);
                    xmlTextWriter.WriteEndElement();

                    xmlTextWriter.WriteStartElement("dob");
                    xmlTextWriter.WriteValue(requestedMember.DOB);
                    xmlTextWriter.WriteEndElement();

                    xmlTextWriter.WriteEndElement();
                }
                else
                {
                    xmlTextWriter.WriteStartElement("error");
                    xmlTextWriter.WriteValue(error.Message);
                    xmlTextWriter.WriteEndElement();
                }
            }
        }

        public bool IsReusable
        {
            get
            {
                return false;
            }
        }
    }
}

Now, with a simple web request to /MemberHandler.ashx?id=12345, you can retrieve just the right information you need from your console application. Now, this is just an example, so, for example, it doesn't send a 404 if the member could not be found. But, these are really easy features to implement!

Then, from your client application, you would simply retrieve the XML and parse it using the classes in System.Xml. This makes the work you have to do close to nothing, and is a easy, quick and fast implementation of communicating between server and client.

There are a number of good resources online that will help you making handlers if you do a good search, one such is .NET: A simple AJAX service using Plain Old XML (POX)

Option 2: Large-scale

If you plan to have a lot of data that needs to be communicated between a client application and your web application, it's best to use a technology such as the ASP .NET Web API to facilitate communication between the two, as it gives you an easy way to create a web service that has a programmable API.

Pros:

  1. Very easy to scale and make changes to. You can easily add and remove different pieces of information as you wish with minimal effort and time.
  2. Less work and code for you to write in the long run. Many times code will be partially generated for you based off of your database or other data source, and templates are readily available to get you started. You also don't have to reinvent the wheel, as many classes and utility methods are given to you, so you can focus on what's important: your data.

Cons:

  1. Takes a while to get set up initially and adapt to. As you get used to using something like the Web API, it becomes much more familiar, but in the beginning it might feel like entering a whole new world.
  2. Can be cumbersome and bulky for small projects. If all you need is to have a single page or two that sends/receives a limited amount of information, it might be better to make a simple handler instead.

Going into depth with frameworks such as the Web API is a very long ordeal, and not best suited for being dealt with right here. However, there are extensive amounts of resources ready to help you out, such as the official Getting Started with ASP.NET Web API 2 tutorial.

The not-so-great way: Scraping information

I highly recommend, that, whenever possible, you try to avoid this method. If you're not in control of the web application you need to talk to, first check to see if the developer/company that supplied the web application provides a documented API. It will be much easier to develop that way instead.

However, sometimes there is little of a choice, and usually this is the case when

  1. You are not in control of the web application's code, and
  2. There is no documented API available.

In this case, you must rely on "scraping" the information you need off of the web page. The theory is simple, but the execution gets rather nasty.

In your example, if you had to rely on scraping, you would first dispatch a request to the web page in question to retrieve its VIEWSTATE and any other form values you may need. In addition, if the web application relies on cookies, you may need to store them and reuse them on the next request, when you try to retrieve the data you need.

This requires trial and error and careful analysis of the web page and the requests/responses that are sent/received when you browse the site normally (which is what you've done in finding the form data that gets sent).

In addition, you have to run two requests to the server, which introduces more points of failure and also makes the client run more slowly.

There are many resources available which detail how to scrape information off of a web page. Many of them that deal with .NET will mention the HtmlAgilityPack library (available on NuGet), which is a wonderful tool that lets you get pieces and bits of information off of an HTML page without having to run inaccurate and expensive searches through the raw text.

Imagine it as though you have a version of the Javascript DOM available on your server, so instead of searching through the HTML as a string, you would use something like:

HtmlAgilityPack.HtmlNode FormElement = htmlResponse.DocumentNode.SelectSingleNode("//form");

If you need help getting started with scraping, Scraping HTML DOM elements using HtmlAgilityPack (HAP) in ASP.NET and How to use HTML Agility pack are good starting points.

Choosing which way to go

As I mentioned before, the method you choose to use really depends on the situation you're in.

  • If this application is under your control, I strongly recommend using some form of API:
    • If you really only need to transmit a couple pieces of information, use something like a Generic Handler
    • If you plan to make communication between client and server an important piece of your solution, use something like the ASP .NET Web API.
  • If you're not in charge of the web application:
    • If there is any API available, look up the documentation and use it!
    • If there is no API available, and no other way, as a last resort, use the scraping technique.

Best of luck!

Community
  • 1
  • 1
Adaline Simonian
  • 4,596
  • 2
  • 24
  • 35