Advanced HTTP POST Protection?

Question

I've been stuck here for about 24 hours on a problem I can not get my head around.

The insurance company I work for rely on requesting quote data from a number of websites, some for analysis, some for quotations to customers. I'm creating a class for the software I developed to add a new insurance provider to our current providers.

I basically send a POST request with the customers information, and our referral. But for the life of me I can not get this to work. I've done this hundreds of times with no problems.

I've monitored the headers in Fiddler, and copied them completely. The only thing the site seems to be setting is 4 cookie values. One is xsrf (which is automatically set when you visit the submission page, I am able to retrieve this from the source code, or by accessing the CookieContainer), the other 2 seem to be session related but are encrypted. So what I do is get my software to visit the page, the cookies are stored, then submit the post request.

I've tried submitting the form manually with JavaScript disabled. And it works. So I can assume there are no variables or cookies being set with JavaScript.

What I can't understand is why the form isn't being submitted.

The only thing I can think of is the session data in the cookies is encrypted, and is storing some values provided by the browser. But without JavaScript what sort of values can the browser be providing that my software isn't?

I've set all the usual User-Agent etc. As I said I've done this hundreds of times and never faced an issue like this.

I've also used Fiddler to get the cookie information, and put that directly into the software (using the same browser as the User Agent is set to on the software), which theoretically should work, but it's not.

I've compared my POST request with the POST request from a browser, side by side, and they are both identical. The only thing that differs is the session cookie values, which are encrypted.

There is no error being returned from the web server. The response code is 200. The only difference is when the quote is successfully submitted the page will include the text "Quotation Successful". Which I'm unable to achieve with the software.

I've called the insurance provider who are unable to help as they don't manage their website. They don't have an API but have allowed our company to make requests via the software so long as we provide the referral ID.

Any ideas on what could be going on here?

For the record I am using C# and HTTPClient. I'm not sure if that's relevant.

Edit:

One thing I have noticed is that upon the GET request to the page that contains the quotation form - using the browser - I notice the following header being returned from the server:

 P3P: CP="CAO PSA OUR"

Also, when the POST request is successfully sent in the browser it also returns this header.

But, when I issue the GET request with the software, I notice the server responds with the P3P header, but on the POST request is doesn't. Could this be relevant/significant?

Answer 1

You may be way ahead of me, and it seems rather outré, but is it possible they're using some form of temporal or request-conditional protection? For example:

• You must request X page and Y page before POSTing the form (the encrypted cookies might include prior requested URIs, or resulting session state from the server)

• You must request X page Y n seconds before POSTing the form (the encrypted cookies might include that date/time)

• You must NOT have POSTed this form previously / within a certain timeframe, with/out cookies being adjusted accordingly

Perhaps some programmer was attempting to foil automated submission or close a hypthetical attack vector.

I'm not certain whether you've already done this, but it might be worth trying a clean site visit from its front page (or as close as you have to get for form submission to work by hand) with clear cookies and cache and watching the HTTP request/response traffic from the start, to see:

• Exactly what headers a browser is sending with each request
• Which response contained the cookies in question (and what that request contained)

To do this, I'm probably preaching to the choir, but with the Chrome browser you can clear cookies, open a blank tab, hit F12 for dev tools, type a URL and then via the F12'd window, select Network and you'll see a list of all request/response pairs. Click any one and look at the request and response source text, and look for the Cookies tab which lets you see cookies both sent and received - so you can see which request yielded the cookies. Perhaps a visit to that page is mandatory/tracked.

(Googling suggests that P3P header is an electronic privacy statement and so unlikely to be related.)

Answer 2

If I were in your situation, I would try doing things to get more information to trouble shoot with. Here are some off the top of my head.

1. Open a new windows form and put a webbrowser control on it.
2. Navigate to the form page you are having problems with.

3. Use the Document property to fill out the form. Something like this should do.

   webBrowser1.Document.GetElementById("QuoteID").SetAttribute(
       "value", 
       "100"
   );    
   
   etc...
   

4. Submit the form and see what happens.

       foreach (HtmlElement f in webBrowser1.Document.Forms)
           f.InvokeMember("submit");
   

Or, If you know the ID of the form button:

        HtmlElement form = webBrowser1.Document.GetElementById("FormID");

        if (form != null)
            form.InvokeMember("submit");

If it works, then the worst case scenario is that you end up having to use a heavy-weight control just for one freakin' web site in your app. But, on the upside, at least you'll have one potential solution to the problem. BTW... If it ends up being that way, I would abstract it away in its own assembly and try to make the calling code similar to the existing code base. That way, you'll just be enhancing your app by adding a new web connectivity option:).

I would also try using the WebClient class instead of the HTTPClient and see if they behave differently..

I might try setting the user agent to each of the major browsers, just to confirm the result.

I would try deleting the cookie cache right before using the HTTPClient and verify that it is getting it's own session id.

Let's see.. this is obscure.. is there any possibility that one of the fields you are sending programmatically has an apostrophe in a field, or any characters that might cause a problem for the server app?

I would try at least to get help from the Insurance Company. They may not control the web app, but they might be able to confirm whether they have captured your tests. Maybe the form is being submitted and the error happens afterwards. Is it possible that the problem is related to a redirection that the HTTPClient doesn't like. For instance, could the control handle it if after the form is submitted it gets redirected to a secure site before being redirecting to the un-secure reply page?

If possible, could you setup a dummy page somewhere that accepts your form data and prints it on the screen so you can see exactly what is hitting the server?

As for the encryption, cookies can be either encrypted or, more often than not, they are just base64 encoded. Either way, the client usually only looks at fields that it created or requested. Otherwise, it just passes along information to the server. So, it doesn't seem a likely culprit to me. I'm not saying it's definitely not. But, I would keep looking elsewhere also.

I know some of this might be silly but when you're stuck, you have to rule everything out. The redirection theory seems interesting because it could explain why this is the only time you've encountered this problem. Maybe it's on their end. They're worried about security and sales, not what you're doing. So, you just never know.

Anyhow, that's all I got for now. I really hope you find the problem. Let me know how things turn out. Take care.

Update

I glanced over your question again to see if I could think of anything else and I had a thought. Is the confirmation page a frames page or is the result you're looking for inside an iframe?

Just food for thought.

Answer 3

Since you have a cookie named xsrf I would guess this is a cookie for CSRF protection.

If you look at the response from the GET request your software makes is there a hidden field containing a token?

e.g. <input type="hidden" name="xsrfToken" value="123456" />

When your software makes the POST, it could be that value of this hidden field is omitted or set to the incorrect value. It may need to be the same as the dynamic value retrieved in the GET request in order to validate the CSRF protection. Usually this value will be the same as the cookie (xsrf cookie in this case), or it will contain an encoded or encrypted version of it (or possibly the cookie will be an encoded encrypted version of the hidden field).

Answer 4

There is no error being returned from the web server. The response code is 200. The only difference is when the quote is successfully submitted the page will include the text "Quotation Successful". Which I'm unable to achieve with the software.

Since, you are looking for the text: "Quotation Successful" in the webpage, then make sure that this text is not dynamically loaded in the page. It is a corner case (just like your problem :)) but it is possible that the page sends some AJAX requests to server after DOM load or some other event and is getting this piece of text. What might be the metric to determine if your software is working fine or not is to compare the source-code (HTML via view-source/Ctrl+U) that the normal form-submission is returning and what your software is returning.

Answer 5

If your generated request is exactly the same as the browser request but still fails, I would investigate the resources loaded by the submission page, it may contain an pixel tag that is used as a server-side session validation marker.

The expected submission process may be something like this:

• GET the submission page and store the cookies set by the server
• GET the security pixel tag from the server, with correct headers, to validate the session server-side
• POST the form values with the correct headers

Using pixel tags like this is a pretty simple and effective anti-spam technique since few bots bother to download page resources.

Answer 6

This is an off the cuff response based on a very similar experience I had recently. In the end, after days of trying to get consistency across requests that I was sending to various installations of the same endpoint, it worked out to be a missing or incorrectly set Content-Length header that the server was expecting. The different servers I was sending to were configured slightly differently.

Basically, I had manually changed the content of one or two of the requests I was testing with, and either failed to update the Content-Length property, or missed it out of the request entirely.

I say check and double check that your HTTP headers are correct.