How to programatically download csv data via ashx on a web page?

Question

I've looked at the links suggested by SO but none seem relevant so here goes..

(In the following the actual url and header data has been obfuscated)

I would really appreciate some help to automate the download of data from a https web page, using Delphi and (preferably) Twebbrowser as I'm not having a lot of success with Indy.

In Firefox I navigate to the url: https://www.thedomain.com/folder1/Showdata.aspx?code=111 that shows some data, there I can click an image that will download the data as csv.

I am trying to automate that process using Twebbrowser in Delphi by navigating to the url and then clicking that image, or by programatically calling the same function that the image does..

The web page shows the image with the anchor text...

<a class="button dlCSV" href="javascript:void(0);" onclick="MyLib.DownloadCsv();return false;">Download</a>

It looks like clicking this causes a POST to

https://www.thedomain.com/folder1/AnotherFolder/Sendcsv.ashx

(with a load of headers) that actually does the download.

I first tried using TWebbrowser to navigate to

https://www.thedomain.com/folder1/Showdata.aspx?code=111

and then clicked the button manually in the Twebbrowser window. That downloaded an empty file called Sendcsv.ashx.

Then I tried programatically clicking the image using code found here: http://www.experts-exchange.com/Programming/Languages/Pascal/Delphi/Q_27493399.html. That didn't seem to do anything. It found the image and called its click method but nothing was downloaded and no error message was shown.

Can anyone help me with the code needed to download this data using Delphi?

---

Additional info that might be useful.

Using the network tab in the debugger in Firefox I managed to find out that clicking the image caused a POST to https://www.thedomain.com/folder1/AnotherFolder/Sendcsv.ashx

and there were loads of headers shown as follows. However I don't know how I write the code that will send these headers to Sendcsv.ashx (if that's what I need to do!) .

Response headers

• Cache-Control:"max-age=0"
• Content-Disposition:"attachment; filename=thedata.csv"
• Content-Encoding:"gzip"
• Content-Length:"6050"
• Content-Type:"text/html; charset=utf-8"
• Date:"Tue, 03 Nov 2014 10:05:43 GMT"
• Pragma:"public"
• Server:"Microsoft-IIS/8.0"
• X-AspNet-Version:"4.0.30319"
• X-Powered-By:"ASP.NET"

Request headers

• Host:"www.thedomain.com" User-Agent:"Mozilla/5.0 (Windows NT 6.1;
• WOW64; rv:33.0) Gecko/20100101 Firefox/33.0"
• Accept:"text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8"
• Accept-Language:"en-GB,en;q=0.5" Accept-Encoding:"gzip, deflate"
• Referer:"https://www.thedomain.com/folder1/Showdata.aspx?code=111"
• Cookie:"rememberme=True" Connection:"keep-alive"
• Content-Type:"application/x-www-form-urlencoded"
• Content-Length:"15972"

Request headers from upload stream

• Content-Type:"application/x-www-form-urlencoded"
• Content-Length:"15972"

Answer 1

Lately, for situations like this, I've been using the MSXML2_TLB.pas unit, obtained by importing the Microsoft XML type library.

It has a class XMLHTTP, which does exactly what a XmlHttpRequest does, like you might recognize from web-development.

uses SysUtils, ActiveX, AxCtrls, MSXML2_TLB;

//be sure to call CoInitialize(nil); on application or thread startup

//procedure ...
var
  reqData:string;
  req:XMLHTTP;
  str:TOleStream;
  f:TFileStream;
begin
  //Did you catch the request data? load/set/build it here:
  //reqData:=

  req:=CoXMLHTTP.Create;
  req.open('POST','https://www.thedomain.com/folder1/AnotherFolder/Sendcsv.ashx',false,'','');
  req.setRequestHeader('Host','www.thedomain.com');
  // try first without this one: it may not be required:
  //req.setRequestHeader('Referer','https://www.thedomain.com/folder1/Showdata.aspx?code=111');
  // same here, may not affect response
  //req.setRequestHeader('Cookie','rememberme=True');
  req.setRequestHeader('Content-Type','application/x-www-form-urlencoded');
  req.setRequestHeader('Content-Length',IntToStr(Length(reqData)));
  req.send(reqData);

  if req.status=200 then
   begin
    str:=TOleStream.Create(IUnknown(req.responseStream) as IStream);
    try
      f:=TFileStream.Create('thedata.csv',fmCreate);//or TMemoryStream?
      try
        f.CopyFrom(str,str.Size);
      finally
        f.Free;
      end;
    finally
      str.Free;
    end;
   end
  else
    raise Exception.Create('CSV request failed: '+req.statusText);//parse req.responseText?
end;