BizTalk FF Disassembler - Only Working With Very Small Files

Question

I have a BizTalk 2016 / VS 2015 Solution. There's a simple flat file schema that debatches each row by having a repeating record element have its "Max Occurs" property set to 0.

The "config" part of the schema can be seen below:

<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:annotation>
    <xs:appinfo>
      <schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
      <b:schemaInfo standard="Flat File" codepage="65001" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="true" early_terminate_optional_fields="true" allow_message_breakup_of_infix_root="true" compile_parse_tables="false" root_reference="Employee" />
    </xs:appinfo>
  </xs:annotation>
  <xs:element name="Employee">
    <xs:annotation>
      <xs:appinfo>
        <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xA" child_order="postfix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
        <b:properties>
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='NiNumber' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='OrgLeaveDate' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='PosLeaveDate' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
          <b:property distinguished="true" xpath="/*[local-name()='Employee' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='Data' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']/*[local-name()='PostStartDate' and namespace-uri()='http://LCC.Integration.Employees.Schemas.SAP.SapEmployee_FF']" />
        </b:properties>
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:annotation>
          <xs:appinfo>
            <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
          </xs:appinfo>
        </xs:annotation>
        <xs:element maxOccurs="1" name="Data">
          <xs:annotation>
            <xs:appinfo>
              <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0x9" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
            </xs:appinfo>
          </xs:annotation>
          <xs:complexType>

The input file is tab delimited with unix style line endings.

The only slightly unusual thing about the flat-file schema is that the last element "PostStartDate" has a micOccurs attribute value of 0. This is because there are a couple of providers of these document and one of the providers doesn't include a "PostStartDate" column.

I've setup a receive location that contains a FF disasembler. It has the DocumentSpecName property set to that of the schema. All other properties are as default.

The only subscriber to this receive is a send port that sends to a local folder.

If I drop a flat file with 3 rows then all works ok. However, if I extend with a few more rows (7 total in this example) then BizTalk suspends on the receive location with the error: "The Messaging Engine encountered an error during the processing of one or more inbound messages." In the event log, I see the error:

There was a failure executing the receive pipeline: "LCC.Integration.Common.Pipelines.RcvArchiveFF, LCC.Integration.Common.Pipelines, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ff3873c2d0a7e656" Source: "Flat file disassembler" Receive Port: "SharePoint.Employees.Receive.SAPExtract" URI: "D:\FileDrop\Employees\In\*.txt" Reason: Unexpected data found while looking for:
'\t'
The current definition being parsed is Employee. The stream offset where the error occured is 1813. The line number where the error occured is 8. The column where the error occured is 0.

Another thing I've noticed, if I remove the trailing LF from the last row in the file then it works - regardless of the number of rows.

I've made a screencast to demo the problem: Screencast of Problem

I've created a sample solution to demonstrate the problem. This is available at the following github repo: https://github.com/RobBowman/BizTalkFFProblem

Any ideas?

Can you check which lines get through when you enable RecoverableInterchangeProcessing on the receive pipeline? — Ruud, Sep 17 '18 at 17:43
@Ruud thanks for the suggestion - I'd completely forgotten about that option. When I enabled, then all of the records processed successfully. This solves the immediate problem but I'd love to know the underlying cause. I will keep experimenting and report back any discovery. — Rob Bowman, Sep 17 '18 at 18:44
Another thing I've noticed. If I remove the trailing LF from the last row in the file then it works - regardless of the number of rows — Rob Bowman, Sep 18 '18 at 09:48

r3verse · Answer 1 · 2018-09-18T14:04:01.703

0

Edit: I just tried your GitHub Sample in BizTalk 2013R2 and it's working fine without the change.

Edit2: Broken when >20 rows.

Edit3: I now realize that it's not really supported, the setting is: 'Allow Message Breakup At Infix Root'... your schema is not infix, it's postfix. Debatching FlatFiles on the disassember is only supported when infix. If possible, you should make your schema to support infix, then it'll work.

edited Sep 18 '18 at 14:04

answered Sep 18 '18 at 11:19

r3verse

1,000
8
19

I'm afraid that made no difference – Rob Bowman Sep 18 '18 at 12:38
Well i tried a few other times and it seems it not working when it's more that 20 rows here. So it's less rows for you. I cannot make any sense of it yet. I've had this issue before and i've seen it on the internet before; see: http://www.biztalkgurus.com/topic/suspended-message-strange-error-because-of-file-size/ . So it's definitely not your fault. – r3verse Sep 18 '18 at 13:51
Thanks r3verse, good to know you've been able to recreate it. – Rob Bowman Sep 18 '18 at 13:53
In my case the records are delimited by LF postfix - I would say that's the most common way, as described: https://learn.microsoft.com/en-us/biztalk/core/child-order-considerations – Rob Bowman Sep 18 '18 at 15:02
I guess you could try to make a trailing schema for just the final LF. That would trick it to being infix. – r3verse Sep 18 '18 at 18:58

BizTalk FF Disassembler - Only Working With Very Small Files

1 Answers1