TIdTCPServer/TIdTCPClient: the best way to transfer a RawByteString?

Question

I'm looking for a most fastest (stock market application, extremely time critical) solution for transfer a lot of encrypted RawByteString-data (up to 500 records/s, ) from TIdTCPServer to TIdTCPClient.

What is the best way to do that?

Thanks in advance!

Edit:

In my project I need to use a n'Software library for the encryption of strings. The fastest encryption method returns a RawByteString. Theoretically, I should only set the CodePage for encrypted string and transfer it UTF8 encoded, but all my attempts failed.

So seemed to me the most logical:

Server:

var
  rbs: RawByteString;
begin
  rbs := EncryptString(AInput, DEFAULT_ENCRYPTION_KEY);
  SetCodePage(rbs, 65001, true);
  AContext.Connection.IOHandler.WriteLn(rbs, IndyTextEncoding_UTF8);
  ...
end;

Client:

var
  rbs: string;
  Output: string;
begin
  rbs := IdTCPClient1.IOHandler.ReadLn(IndyTextEncoding_UTF8);
  Output := DecryptString(rbs, DEFAULT_ENCRYPTION_KEY);
  ...
end;

Well, it seems if you have already chosen the components you wish to use, there's not too many other options, now are there? Did you want us to read the documentation for you, or do Google searches for you? — Jerry Dodge, Apr 08 '17 at 15:23
Transferring a binary encrypted `RawByteString` as a UTF-8 string is just plain wrong, no matter how fast it may be. The correct way to transfer binary data with Indy is to use a `TIdBytes` or `TStream`. The `IOHandler` has reading/writing methods for both. Look at Indy's `RawToBytes()` and `BytesToRaw()` functions, or the `TIdMemoryBufferStream` class. — Remy Lebeau, Apr 10 '17 at 00:09
@Remy Lebeau Thank You Sir, this is this exactly what i'm looking for! — vitaliy-zh, Apr 11 '17 at 07:54

score 0 · Accepted Answer · answered Apr 11 '17 at 15:53

0

Transferring a binary encrypted RawByteString as a UTF-8 string is just plain wrong, no matter how fast it may be. The best way to transfer binary data with Indy 10 is to use a TIdBytes or TStream instead. The IOHandler has reading/writing methods for both, for example:

var
  rbs: RawByteString;
begin
  rbs := EncryptString(AInput, DEFAULT_ENCRYPTION_KEY);
  AContext.Connection.IOHandler.Write(Int32(Length(rbs)));
  AContext.Connection.IOHandler.Write(RawToBytes(PAnsiChar(rbs)^, Length(rbs)));
  ...
end;

var
  buf: TIdBytes;
  rbs: RawByteString;
  Output: string;
begin
  IdTCPClient1.IOHandler.ReadBytes(buf, IdTCPClient1.IOHandler.ReadInt32);
  SetLength(rbs, Length(buf));
  BytesToRaw(buf, PAnsiChar(rbs)^, Length(rbs));
  Output := DecryptString(rbs, DEFAULT_ENCRYPTION_KEY);
  ...
end;

Alternatively:

var
  rbs: RawByteString;
  strm: TIdMemoryBufferStream;
begin
  rbs := EncryptString(AInput, DEFAULT_ENCRYPTION_KEY);
  strm := TIdMemoryBufferStream.Create(PAnsiChar(rbs), Length(rbs));
  try
    AContext.Connection.IOHandler.LargeStream := False;
    AContext.Connection.IOHandler.Write(strm, 0, True);
  finally
    strm.Free;
  end;
  ...
end;

var
  strm: TMemoryStream;
  rbs: RawByteString;
  Output: string;
begin
  strm := TMemoryStream.Create;
  try
    IdTCPClient1.IOHandler.LargeStream := False;
    IdTCPClient1.IOHandler.ReadStream(strm, -1, False);
    SetLength(rbs, strm.Size);
    Move(strm.Memory^, PAnsiChar(rbs)^, Length(rbs));
  finally
    strm.Free;
  end;

  { alternatively:

  SetLength(rbs, IdTCPClient1.IOHandler.ReadInt32);
  strm := TIdMemoryBufferStream.Create(PAnsiChar(rbs), Length(rbs));
  try
    IdTCPClient1.IOHandler.ReadStream(strm, Length(rbs), False);
  finally
    strm.Free;
  end;
  }

  Output := DecryptString(rbs, DEFAULT_ENCRYPTION_KEY);
  ...
end;

answered Apr 11 '17 at 15:53

Remy Lebeau

555,201
31
458
770

Great! Thank You! – vitaliy-zh Apr 14 '17 at 15:25
Is this correct that Write() doesn't causing an exception after default timeout (unlike WriteLn), if the client disconnects abnormally? How can the server detect abnormally disconnected clients in this case? – vitaliy-zh Apr 18 '17 at 23:16
The `Write...()` methods funnel down to `Write(TIdBytes)`, which eventually calls `WriteDataToTarget()` for the actual send. So any timeouts and error handling apply to all write operations equally. But remember, a send is not an immediate transmit, it just buffers data inside the socket kernel for eventual transmit. So an abnormally disconnected socket could still "send" data until the OS invalidates the socket after a (lengthy) timeout, or until the buffer fills up and a write timeout elapses... – Remy Lebeau Apr 18 '17 at 23:32
... Indy does not use write timeouts by default, but you can enable a write timeout manually on the underlying socket directly (via `TIdSocketHandle.SetSockOpt()`) or enable TCP keep-alives (via `TIdSocketHandle.SetKeepAliveValues()`). Generally, if you send something to the other party and are expecting a reply back, you would use a read timeout (or TCP keepalives) to expire and close the connection when the reply does not appear in a timely manner. But, if you are just sending data, and not expecting a reply back, then enabling write/keep-alive timeouts manually is about all you can do. – Remy Lebeau Apr 18 '17 at 23:34
By WriteLn() the server causes an exception after approx. 30 seconds if the client is no longer connected. How is this possible if there is no write-timeout? – vitaliy-zh Apr 18 '17 at 23:48
Like I said, **all** of the write methods go through a single point of entry (`Write(TIdBytes)`). `WriteLn()` simply converts a `string` (with `EOL` appended to it) to a `TIdBytes` and then calls `Write(TIdBytes)`. So there is no possible way that `WriteLn()` can behave differently than any other `Write...()` method in terms of timeout/error handling. – Remy Lebeau Apr 19 '17 at 00:15
But in any case, just because You have not enabled a timeout doesn't mean there is no timeout at all. The OS has a built-in timeout (it just happens to be fairly lengthy by default), since it does have to invalidate a dead connection eventually. Also, if you happen to be using `TIdSSLIOHandlerSocketOpenSSL` on Windows Vista+ with no custom `ReadTimeOut` assigned to it, it enables a 30-second read and write timeout at the socket layer. – Remy Lebeau Apr 19 '17 at 00:18

TIdTCPServer/TIdTCPClient: the best way to transfer a RawByteString?

1 Answers1