Getting XML string from response stream using Indy's IDTCPClient

Question

I need help with a component I have taken over as a new employee with the task of adding new methods to the component. The component uses idTCPClient and NOT idHTTP. See the responses in the two questions below for the reasons why.

Getting XML from response stream using Indy's IDTCPClient

Getting HTML from response stream using Indy's IDTCPClient

Yes Remy & Jerry, I have begun the rewrite of the component to utilize tidHTTP instead of idTCPClient, but I am doing it on my own free time, not the companies time. SO, I will need to finish it tonight. Ok, back to this post.

Calling the get function below, I get the following response back

HTTP/1.0 200 OK 
Content-Length: 2069
Date: Wed, 20 Nov 2013 15:00:11 GMT
Content-Type: text/xml; charset=UTF-8


function TMyConnector.GET(aRawHeader: String): String;
begin

 if Not Connected then Connected := True;
if Connected then
  begin
  FRawRequest :=  'GET /'+ EncodeUrl(aRawHeader) + ' HTTP/'+HTTPVerText+#13#10+
                  'Host: '+FHost+#13#10+

                  'Cookie: UserHPos=IOGLO00003000090000C000BS; '+
                  'LOSID=qsBiy/wEDCq6tOXFzGbOlTD1lmo5AXdFnCkbzzPn6+qCeheYVyTcumRrjsqh+Hds4Fr2gZDazfDzGN1RA+nnHuQQeBy78ZUgctrZyyy9MnGl2qI/ulkV6EPxAfmmLg/lopRq99f5gAcG/dgtytAJjS+aD5DqtHGrAqjiqgtkwuA=; '+
                  'LoginHPos=IOGLO00003000090000C000BS; '+
                  'UIHPos=IOGLO00003000020000500003; '+
                  'LOSG=61939308-7C83-47ED-B909-2D2D10AD7026; '+
                  'fControllingBusiness=IOGLO000030000900001000050000200001'+#13#10+

                  'Connection: Close'+#13#10+
                  #13#10;

   FSock.Socket.Write(FRawRequest);
   FRawResponse := FSock.Socket.ReadLn(#13#10#13#10,nil);

  Result := FRawResponse;

  if ResponseStream = nil then ResponseStream := TMemoryStream.Create
   else ResponseStream.SetSize(0);

   FSock.Socket.ReadStream(ResponseStream,-1,True);

  if Connected and (Not KeepAlive) then Connected := False;
  end;
end;

How do I now get the XML string from responseStream?

There is a current method that already exist for JSON

Procedure TMyConnector.GenerateJSON;
begin
if ResponseStream <> nil then
  Begin
   ResponseJSON_V := TJSONObject.ParseJSONValue(StreamToArray(ResponseStream),0) as TJSONValue;
  End;
end;

I would like to create a method for XML, and I tried the following:

Procedure TMyConnector.GenerateXML;
var
 S: String;
begin
if ResponseStream <> nil then
   // code here to convert ResponseStream to XML
    ResponseXML_v:= S;
end;

Save all the received data into TXT file. Then open it in some hex+text editor and investigate. Also read HTTP protocol description on Wikipedia and HTTP RFC. Returned data would hopefully consists of lines, separated by value==10 bytes (or 13, 10 sequences). Initially there would be "header" - responce description in 7-bit (0..127) AnsiChar. There would be headers like length of data, charset/codepage of data and the conversion method of that data into 7-bit bytes (like ZIP+base64 or any other combinations of transformations). Then there would be an empty line and the data, whole or chunk of. — Arioch 'The, Nov 20 '13 at 15:41
Is this a webservice you are communicating with? Then this is not the way to go... — whosrdaddy, Nov 20 '13 at 15:41
So you would have to parse the HTTP headers, read the data to the end, if it is chunked to multiple packets, then build the graph of transformations and de-transform the data into the native Delphi string. Or you can just take any of the ready-made HTTP components: http://stackoverflow.com/questions/19653886 — Arioch 'The, Nov 20 '13 at 15:44
@Marcus - sorry, no, it is not working, I get a blank string. I will redit post — LIVESTUFF, Nov 20 '13 at 15:44
@whosrdaddy - No, what makes you think that? Something in my post? — LIVESTUFF, Nov 20 '13 at 15:45
Regarding InputBufferAsString - http://stackoverflow.com/questions/8761490 // in general youwould have to read HTTP specifications and observe data interchanges with server between yor program and some etalon working one (like www browser or some toher). You would have to gradually eliminate all viaolations of HTTP by your program then to make it behave liek the working program do. http://stackoverflow.com/questions/17546558 — Arioch 'The, Nov 20 '13 at 15:53
@Arioch, surely there is a simpler solution? If one can get JSON data from a stream to a string, it should be able to get the XML from the stream into a string? Yes? — LIVESTUFF, Nov 20 '13 at 16:06
if ResponseStream <> nil then Begin ResponseJSON_V := TJSONObject.ParseJSONValue(StreamToArray(ResponseStream),0) as TJSONValue; End; - surely there is a similiar approach using XML? I just need help determing the calls or objects to use — LIVESTUFF, Nov 20 '13 at 16:08
HTTP is not a "stream" - it is a very complex beast, developed for about 20 years and extended time and again. So either you use HTTP or a plain non-HTTP binary stream. In the latter case you would have to solve one way or another few questions: which charset is used for string ? how long is string ? when the stream had ended or not yet ? So, if you would not use HTTP server - then you would have to devise and implement some other protocol, fixing or negotiating those questions. — Arioch 'The, Nov 20 '13 at 16:15
I also assume you understand that the same string in different charsets is different binary data. And TCP transfers bytes, not letters. And so you have to fix or negotiate the transformation rules "from bytes to letters" — Arioch 'The, Nov 20 '13 at 16:17
@Arioch - Your not getting my point. I f I have a stream that clearly works in my method GetJSON method, why can't i so the same way in my GetXML ( clearly speaking, I would need a different set of objects of routines to convert the stream back to XML). If yiou have a solution, I ask you to post it — LIVESTUFF, Nov 20 '13 at 16:20
There is no GetXML/GetJSON procedures above. Also, debugging the code that works by sheer luck and would break as soon as moon phase changes is not a proper process — Arioch 'The, Nov 20 '13 at 16:25
GetJSON works upon ResponseStream and GenerateXML works against some other data unrelated to ResponseStream - so why should they work similarly ? — Arioch 'The, Nov 20 '13 at 16:26

score 1 · Answer 1 · edited May 23 '17 at 12:19

How do I now get the XML string from responseStream?

Well, at very least you have to read some data from responseStream. But you do not read it in GenerateXML code.

.

Procedure TMyConnector.GenerateXML;
var
 S: String; SS: TStringStream;
begin
 if ResponseStream <> nil then begin
   S := '';
   SS := TStringStream.Create(S, TEncoding.UTF8, False);
   try
     SS.LoadFromStream(ResponseStream);
     // original: ResponseXML_v := S;   - so ResponseXML_v is jsut a string....
     ResponseXML_v:= SS.DataString; 
   finally
     SS.Destroy;
   end;
 end;
end;

But here UTF8 is sheer guessing and it would fail if ResponseStream is for example zip archive or a base64-encoded text or something other.

After obtaining the string you would have to parse it. But parsing the string and transferring it via net are different questions. You can get a list of XML parsers here : What is the fastest XML Parser available for Delphi?

I also suggest you trying to use SuperObject library. I am not sure it has support for all the XML features, but it has the common API for JSON and XML which should be a good thing.

Anyway, you can google through StackOverflow to see that Delphi stock DB-Express JSON parser has a number of bugs and limitations.

https://code.google.com/p/superobject/source/browse/superxmlparser.pas

Procedure TMyConnector.GenerateXML;
begin
 Response_v := nil;
 if ResponseStream <> nil then begin
    ResponseStream.Position := 0;
    Response_v:= XMLParseStream(ResponseStream)
 end;
end;

Procedure TMyConnector.GenerateJSON;
begin
 Response_v := nil;
 if ResponseStream <> nil then begin
    ResponseStream.Position := 0;
    Response_v:= TSuperObject.ParseStream(ResponseStream);
 end;
end;

https://superobject.googlecode.com/git/readme.html

However charset detection and convertion may still be your responsibility before the parsing.

So, "separating the concerns" and using "DRY principle" ("code re-use" part of it) we come to unifying the function into something like that:

{$ScopedEnums On}
type ResponseFormat = (XML, JSON)

function ParseResponse(const ResponseStream: TStream; 
           const Format: ResponseFormat): iSuperObject;
begin
 Result := nil;
 if ResponseStream = nil then exit;
 if ResponseStream.Size <= 0 then exit;

 ResponseStream.Position := 0;
 case Format of
   ResponseFormat.XML:  Result := XMLParseStream(ResponseStream);
   ResponseFormat.JSON: Result := TSuperObject.ParseStream(ResponseStream);
   else raise Exception.CreateFmt(
          'Unknown response format #%d, cannot parse it.',  [ Ord(ResponseFormat) ] );
 end;
end;

And now your responsibility is to call that function in uniform matter for both JSON or XML response and to ALWAYS provide it the proper input data stream.

Overall, you have two problems in front of you:

how to fetch the text file from HTTP server and convert its charset to the native Delphi string charset
how to parse Delphi string containing XML or JSON to an easy to use OO-API.

Those are different problems and you should deconstruct your situatio and your program to those pieces. Otherwise you entangle the thins to the wild mix that no one can understand (including you) and debug. You should be de-coupling and un-tangling, not the opposite.

You tell that you get the same request to the server, but the GenerateJSON function gets the stream and GenerateXML does not ? What it should mean ? only that there is some CRITICAL difference in the code, that you did not show. And that means you don;t understand what pieces of your code doing what, when and why. You should "divide and conquer". Separate your program to pieces. One piece would get the data stream from the server - and nothing else. Another part would parse the stream - and nothing else. Then you should debug them separately (and ask separate questions). There is no point to debug and rework the parsing, when you have nil stream. So... separate concerns. Make the function, fethcing the stream, and ensure it DOES work. For example by saving the stream to text file and checking its content with Text+Hex editor. After you complete that 1st task - go to the next one, do the parse of the received stream. But only after you ensured you always do get the stream.

Newline in HTTP should always be #13#10, regardless of platform. I think the OP is more stuck on how to read the content bytes (into a string), and assumes the rest will automagically work. — Marcus Adams, Nov 20 '13 at 17:36
@LIVESTUFF because you asked to compare the implementations of GetJSON - that did read it - and implementation of GetXML - that did not (before you edited the question and removed implementation at all). And then you wondered, why the former worked and the latter did not. What about nil - then show the same code that creates the stream, fills it with data, and then uniformly calls either GetXML or GetJSON. If the stream is nil in your case - that only means a difference in the code you omitted, the code that actually does the calls. — Arioch 'The, Nov 20 '13 at 17:54
@LIVESTUFF add the code that uniformly calls either GenerateXML or GenerateJSON to see that you actualyl create the same stream for them. You initial implementation did not read the ResponseStream not because it was nil - it would not read it anyway, there just was no single attempt to read it in your initial sources — Arioch 'The, Nov 20 '13 at 18:16
@LIVESTUFF then i sugegst you to start with reading to overall guidelines: http://en.wikipedia.org/wiki/Divide_and_conquer_algorithm and http://www.catb.org/esr/faqs/smart-questions.html - grasping that would help you to learn better. — Arioch 'The, Nov 21 '13 at 07:30
Now, your question is ABSOLUTELY unclear and vague. What input do you have ? no answer. Well, we may read your code and TRY TO GUESS that you have `ResponseStream:TStream` for an input - but why should we guess ? you should give us all the detail to completely understand the issue. And to make issue a narrow single-question case. Then what do you want as a result ? You initial code said that by "XML" you actually meant "string". That looks insane, but that was what you said. — Arioch 'The, Nov 21 '13 at 07:33
So back from copmmon things to the particular issues. "which of your code I should be using " - depends upon what do you want. In your question you implied that "getting XML" is the same as "getting string" to you (you did parsed JSON to some non-string objects, but you did not parsed XML to anything beyond string). Then - go with 1st code. // However if you would want to parse XML - then add some library to parse it from stream or from string. 2nd code is parsing both JSON and XML in a uniform way using the same library. It is an example of what you can do, just that. — Arioch 'The, Nov 21 '13 at 07:36
You're free to use other library for JSON and XML parsing and access, i chosen a well-known for JSON "SuperObject", but did not tried it for XML and do not know how good it is there. You are free to use ANY library for JSON and any library for XML as you would like to chose. // "As I stated, ResponseStream is always nil" - so make it not nil. Make it filled with data like you do for JSON. We cannot fix your code for you, twice so, when you hide it. Read http://sscce.org/ - you posted random unsorted pieces of code, with no relations between them. How can we GUESS what do you call if any ? — Arioch 'The, Nov 21 '13 at 07:38
I made yet another variant for you. Google about the mentioned concepts and answer for yourself why #3 variant is better that #2 — Arioch 'The, Nov 21 '13 at 07:49

Getting XML string from response stream using Indy's IDTCPClient

1 Answers1