1

I'm getting an 'Unconvertable character' exception when sending a HTTPRequestHeader to HTTPRequest Node containing a special character.

Error in the ExceptionList when debugging:

RecoverableException
    File:CHARACTER:/jenkins/slot0/product-build/WMB/src/CommonServices/ImbConverter.cpp
    Line:INTEGER:733
    Function:CHARACTER:ImbConverterCPP::    
    Type:CHARACTER:
    Name:CHARACTER:
    Label:CHARACTER:
    Catalog:CHARACTER:BIPmsgs
    Severity:INTEGER:3
    Number:INTEGER:2136
    Text:CHARACTER:Unconvertable character
    Insert
            Type:INTEGER:5
            Text:CHARACTER:1920
    Insert
            Type:INTEGER:5
            Text:CHARACTER:4c0061006c0069006100192073002000420075007200690061006c002000460075006e006400200061006e006400200043006100720065002000450078007000 ...data truncated to first 64 chars
    Insert
            Type:INTEGER:2
            Text:CHARACTER:819

Snippet of my esql code:

SET OutputRoot.HTTPRequestHeader."Content-Type" = 'application/octet-stream';
SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = rInputDocData.*:DocumentName;
--Setting the content
SET OutputRoot.BLOB.BLOB = BASE64DECODE(rInputDocAttachment.*:AttachmentData64Binary);

The special character is coming in the field rInputDocData.*:DocumentName. Some of the values are:

Printing….PDF
Mark’s Agreement .pdf

Notice the … and ’ in the above two values which are not recognized as part of UTF-8.

Is there a way to convert these values to ISO 8859-1 in ESQL as doing the same conversion in Notepad++ leads to the values getting accepted?

I have tried the below steps but none have worked and I'm still getting the same error:

  1. Setting OutputRoot.Properties.CodedCharSetId to 1208 and OutputRoot.Properties.Encoding to 546 as the properties were when the request was received in Input.

  2. Setting OutputRoot.Properties.CodedCharSetId to 819.

  3. Setting Content-Type HTTPRequestHeader to 'application/octet-stream; charset=utf-8'.

  4. Setting Content-Type HTTPRequestHeader to 'application/octet-stream; charset=iso-8859-1'.

  5. Casting the HTTPRequestHeader 'X-IntegrationServer-Resource-Name' in the below ways:

    1. SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(rInputDocData.*:DocumentName AS CHARACTER CCSID 1208 ENCODING 546);
    2. SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(rInputDocData.*:DocumentName AS CHARACTER CCSID 819);
    3. SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(CAST(rInputDocData.*:DocumentName AS CHARACTER CCSID 1208 ENCODING 546) AS CHARACTER CCSID 819);

The source vendor have refused to handle/convert the special character so it is upon us to handle this in ACE. Any help would be much appreciated.

Goonerd
  • 15
  • 3

3 Answers3

2

HTTP headers should only use characters in the US-ASCII range. Some other characters may be tolerated, but web server implementations are not consistent in this area.

See the discussions here for more details: what characters are allowed in HTTP header values? One of the participants in that conversation is an author of the HTTP/1.1 specification, so I think we can rely on his input.

re:

Notice the … and ’ in the above two values which are not recognized as part of UTF-8.

UTF-8 can represent any Unicode character, so that statement cannot be true. If you look carefully at the error, the quoted CCSID is 819, which is the IBM CCSID for ISO-8859-1.

kimbert
  • 2,376
  • 1
  • 10
  • 20
  • Re the 2nd point, why is it trying to convert to 819 always? I've seen the same error (with 819 CCSID quoted) even when I've explicitly set the OutputRoot.Properties.CodedCharSetId to 1208. My assumption here is that the error is occurring because the broker is attempting to convert this character from Unicode to CCSID 819 and failing. – Goonerd May 06 '22 at 14:25
  • 1
    It is trying to convert to 819 because older versions of the HTTP specification would accept any character in that code page. But there is _no guarantee_ that a web server will accept any character outside of the US-ASCII (7-bit) range. You should stop trying to send headers that contain non-US-ASCII characters. – kimbert May 09 '22 at 10:43
0

It seems that the ... character is just not in the CCSID 819 charset. It's in the Windows version of Latin 1, CCSID 1252, but as the previous answer said, that may not work with the target Web server.

You can try these, but not sure any will work:

SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(ASBITSTREAM(rInputDocData.*:DocumentName CCSID InputRoot.Properties.CodedCharSetId) AS CHARACTER CCSID 819);

SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(ASBITSTREAM(rInputDocData.*:DocumentName CCSID InputRoot.Properties.CodedCharSetId) AS CHARACTER CCSID 1252);
Attila Repasi
  • 1,803
  • 10
  • 11
0

The below worked for me, but as the top comment had pointed out, it did go on to fail with an encoding error in the target server:

SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(CAST(rInputDocData.*:DocumentName AS BLOB CCSID InputRoot.Properties.CodedCharSetId) AS CHARACTER CCSID 819);
Goonerd
  • 15
  • 3