How to determine response data encoding in TIdHTTP?

Question

I am new to HTTP. I have several HTTP GETs which return text encoded in either ANSI or UTF-8. Is there a way to determine which is returned so that I can handle the conversion in the method rather than doing trials and hard coding the necessary conversions for each? or if it can be handled by Indy itself?

Here's what I use to GET:

Main Method

procedure TIdHTTPHelper.SendGet
          ( const AURL     : String;
            var   AResponse: TMemoryStream );
begin

  Request.ContentType   := '';
  Request.ContentLength := -1;
  Get ( AURL, AResponse );
  AResponse.Position := 0;

end;

Overloaded Method For Strings

procedure TIdHTTPHelper.SendGet
          ( const AURL     : String;
            var   AResponse: String);
var
  Response : TMemoryStream;
begin

  Response      := TMemoryStream.Create;
  SendGet       ( AURL,          Response );
  SetLength     ( AResponse,     Response.Size div 2 );
  Response.Read ( AResponse [1], Response.Size );
  Response.Free;

end;

You should examine a `Charset` parameter for `Content-Type` header. — Free Consulting, Feb 14 '14 at 12:52
I had a similar question some time ago, I think that [Remy Lebeau's answer](http://stackoverflow.com/a/18832982/800214) can help. Anyway, don't use TMemoryStream for Strings, use TStringStream — whosrdaddy, Feb 14 '14 at 14:46
DON'T use `TStringStream`! In D2009+, it requires you to specify a `TEncoding`in the constructor, and you cannot change it afterwards. Unless you know the exact charset ahead of time, you can easily mismatch the encoding and cause a conversion failure. — Remy Lebeau, Feb 15 '14 at 16:42

score 1 · Accepted Answer · edited May 23 '17 at 10:31

TIdHTTP has an overloaded version of Get() that returns a String (which is UnicodeString in D2009+ and AnsiString in earlier versions). It detects the response charset, even to go as far as parsing it from HTML and XML data when it is not present in the HTTP headers, and uses it to decode the response data to UTF-16 in D2009+ and Ansi in earlier versions for you:

procedure TIdHTTPHelper.SendGet
          ( const AURL     : String;
            var   AResponse: String);
begin
  AResponse :=  Get ( AURL );
end;

Have a look at this answer where I outlined the exact charset detection logic that TIdHTTP uses:

TIdHTTP character encoding of POST response

On a separate note, your other SendGet() method should not be taking a TMemoryStream by reference, since you are not returning a new TMemoryStream object to the caller. It should take a general TStream by value instead. That would make your code much more flexible:

procedure TIdHTTPHelper.SendGet
          ( const AURL     : String;
            AResponse: TStream );
begin
  Get ( AURL, AResponse );
  AResponse.Position := 0;
end;

How to determine response data encoding in TIdHTTP?

1 Answers1