Delphi + Synapse: How to check the size of the response with TTCPBlockSocket

Question

I have finally got my Delphi application to send data using direct sockets with the Synapse library through HTTPS.

However, I am having trouble determining the size of the data being returned to me.

Currently, I have the following code:

Socket     := TTCPBlockSocket.Create;
status     := TStringList.Create;  

status.Append('LineBuffer length: ' + IntToStr(Length(Socket.LineBuffer)));
status.Append('Waiting bytes: ' + IntToStr(Socket.WaitingDataEx));
status.Append('recvBufferStr: ' + CRLF + Socket.RecvBufferStr(240, 25000) );
status.Append('LastError = ' + Socket.LastErrorDesc);
status.Append('Error code: ' + IntToStr(Socket.LastError));
Memo1.Lines.AddStrings(status); 

and I get the following in Memo1:

socket lastError = 0
LineBuffer length: 0
Waiting bytes: 0
recvBufferStr: 
HTTP/1.1 200 OK
Date: Thu, 22 Dec 2011 01:06:07 GMT
Server: Apache
Content-Length: 237
Connection: close
Content-Type: text/plain; charset=utf-8

[***The returned Data***]

---

If Socket.RecvBufferStr's first parameter (length to receive) is too large, I get winsock error 10054 because i'm still waiting for data even though the server is done sending it.

If it is too short, which it almost always is, then I only get the specified amount of data.

Waiting bytes and linebuffer length show 0 (not sure if that's because they're longInt's and I'm doing IntToStr) so I don't think that's how I check the amount of data. And I have also tried using CanRead and CanReadEx to no avail.

I would like to do something like the following: Socket.RecvBufferStr([length to receive], [until all data is received] or 25000).

If there is another function, that is fine, but I would like to stick with TTCPBlockSocket as HTTPSend and others that I have tried from synapse do not work for my purposes.

How do I check - using the TTCPBlockSocket socket from the Synapse library with Delphi/Pascal - how much data the server is returning to me?

Answer 1

Try the following code. It shows how to get the Content-Length from the response header and read the content itself.

Note, that the HTTP protocol is far not that easy, but if you are going to communicate with your own server or with the server where you know how it works, you might take this as an inspiration. And don't forget to include the OpenSSL binaries (version 0.9.8 is surely compatible with Synapse) when using this code.

uses
  blcksock, ssl_openssl;

function HTTPGetText(const Host: string): AnsiString;
var
  ContentLength: Integer;
  HTTPHeader: AnsiString;      
  TCPSocket: TTCPBlockSocket;
begin
  Result := '';

  TCPSocket := TTCPBlockSocket.Create;
  try
    // initialize TTCPBlockSocket
    TCPSocket.ConvertLineEnd := True;
    TCPSocket.SizeRecvBuffer := c64k;
    TCPSocket.SizeSendBuffer := c64k;

    // connect to the host, port 443 is default for HTTP over SSL
    TCPSocket.Connect(Host, '443');
    // check for socket errors
    if TCPSocket.LastError <> 0 then
      Exit;

    // server name identification
    TCPSocket.SSL.SNIHost := Host;
    // initialize and connect over OpenSSL
    TCPSocket.SSLDoConnect;
    // server name identification
    TCPSocket.SSL.SNIHost := '';
    // check for socket errors
    if TCPSocket.LastError <> 0 then
      Exit;

    // build the HTTP request header
    HTTPHeader :=
      'GET / HTTP/1.0' + CRLF +
      'Host: ' + Host + ':443' + CRLF +
      'Keep-Alive: 300' + CRLF +
      'Connection: keep-alive' + CRLF +
      'User-Agent: Mozilla/4.0' + CRLF + CRLF;

    // send the HTTP request
    TCPSocket.SendString(HTTPHeader);
    // check for socket errors
    if TCPSocket.LastError <> 0 then
      Exit;

    // read the response status, here some servers might give you the content
    // note, that we are waiting in a loop until the timeout or another error
    // occurs; if we get some terminated line, we break the loop and continue
    // to check if that line is the HTTP status code
    repeat
      HTTPHeader := TCPSocket.RecvString(5000);
      if HTTPHeader <> '' then
        Break;
    until
      TCPSocket.LastError <> 0;

    // if the line we've received is the status code, like 'HTTP/1.1 200 OK'
    // we will set the default value for content length to be read
    if Pos('HTTP/', HTTPHeader) = 1 then
    begin
      ContentLength := -1;

      // read the response header lines and if we find the 'Content-Length' we
      // will store it for further content reading; note, that again, we are
      // waiting in a loop until the timeout or another error occurs; if the
      // empty string is received, it means we've read the whole response header
      repeat
        HTTPHeader := TCPSocket.RecvString(5000);
        if Pos('Content-Length:', HTTPHeader) = 1 then
          ContentLength := StrToIntDef(Trim(Copy(HTTPHeader, 16, MaxInt)), -1);
        if HTTPHeader = '' then
          Break;
      until
        TCPSocket.LastError <> 0;

      // and now let's get the content, we know it's size, so we just call the
      // mentioned RecvBufferStr function
      if ContentLength <> -1 then
        Result := TCPSocket.RecvBufferStr(ContentLength, 5000);
    end;

    TCPSocket.CloseSocket;

  finally
    TCPSocket.Free;
  end;
end;

procedure TForm1.Button3Click(Sender: TObject);
begin
  Memo1.Text := HTTPGetText('www.meebo.com');
end;

Answer 2

While testing WaitingDataEx again, as TLama mentioned, I decided to put a line that output WaitingDataEx after RecvBufferStr.

IntToStr(WaitingDataEx) worked and displayed the left over data.

WaitingDataEx checks for waiting data at that particular instant, and because there was no real pause from sending the data, to looking for it, it saw that there was no data to receive yet.

I solved the issue with the following code:

httpRsp := httpRsp + Socket.RecvBufferStr(1, 25000);
httpRsp := httpRsp + Socket.RecvBufferStr(Socket.WaitingDataEx, 100); 

1. This waits for 1 byte of data to be received, and then adds that data to string variable httpRsp.
2. It then uses Socket.WaitingDataEx to read the rest of the bytes, as the first RecvBufferStr command has already done the waiting to ensure there are bytes to receive.

EDIT to above

The above code broke from a specific server I am communicating with, though I am not sure why. It would only receive the headers, not the content from one server I am communicating with. I resolved the issue with the below code:

httpRsp := httpRsp + Socket.RecvPacket(50000);
while Socket.WaitingDataEx > 0 do
  httpRsp := httpRsp + Socket.RecvPacket(50000);