How can I input XML data from an external web service into a scope field during a custom FAST ESP pipeline stage written in python?

Question

We have an external web service that we access during a custom pipeline stage in FAST. Our task is to get the XML that is output by this web service and store it into a scope field. At this point we have tried the following things:

1. Inputting the XML output by the web service directly into the scope field (i.e. using document.Set('xml', webserviceOutput), where xml is a scope field and 'webserviceOutput' is of a form akin to <?xml version="1.0" encoding="UTF-8" ?><root><element>...</element></root>). This did not work and the FAST pipeline complained in the logs that the data was of the wrong type (i.e. <type 'str'>).

2. We also tried inputting the XML output of the web service directly into a flat field and then using the XMLParser and XMLScopifier to convert the string into the right format expected by the scope field. This also did not work and the FAST pipeline complained that the XML data in our flat field was invalid. Upon closer inspection, we discovered that FAST escaped all of the angle brackets of the XML web service output into < and > before storing the string into the flat field. This seems to be the cause of the XMLParser not being able to convert the string into the right object expected by the XMLScopifier.

To determine what was going on when we first saw second error above, we ran a simple FQL query from the web front-end (i.e. http://localhost:15100/cgi-bin/xsearch?offset=0&hits=10&query=a) and looked at the source of the XML returned (e.g. ctrl+u on Firefox).

The angle brackets and the quotes in the XML value of the flat field of interest had all been escaped to < > and ". We initially thought this could be a product of the browser attempting to display the XML string correctly, but the same thing does not happen in the body field (i.e. <FIELD NAME="body">) that also contains tags in its string value (e.g. <sep/>convert the whole structure to <key>a</key> series of maps <sep/>create <key>a</key> dao that handles all interaction with<sep/>).

There is also the possibility that FAST is escaping the characters just before the results are sent to the browser, although I am not sure how we can determine if this is the case.

How can we solve this?

We are basically stuck on this at the moment and have absolutely no idea how to move forward from here, so any help or hints would be much appreciated!

Many thanks!