I have a data source which generates XMLs in real time. While checking the Kinesis Firehose integration using kinesis agent for the real time analytics of this XML, I found that Kinesis agent considers each line in the source file as a single record. For example, Kinesis found 6 records in the given sample XML
<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>
Now, If I have to perform data transformation on the input XMLs using lambda. How do I do that since kinesis agent is overlooking the structure of the XML?
Here is the Kinesis agent log to refer:
2020-07-08 02:38:03.955+0000 (FileTailer[fh:POCKinesis:/var/log/cadabra_xml/*.xml].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:POCKinesis:/var/log/cadabra_xml/*.xml]: Tailer Progress: Tailer has parsed 6 records (236 bytes), transformed 0 records, skipped 0 records, and has successfully sent 6 records to destination.
2020-07-08 02:38:03.957+0000 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 6 records parsed (236 bytes), and 6 records sent successfully to destinations. Uptime: 870056ms
