1

I have a kinesis stream which has data with its original value as Base64 encoded . As Kinesis by default encodes all its data to Base64, I have to do a Base64decode to getdata from the Kinesis DataBlob . But since the originalData is also Base64 while decoding the originalData gets Corrupted/decoded and I donot want my Originaldata to get decoded. My code resides in a lambda which is triggered by an SQS and I cannot make Kinesis as the trigger . I have observed that this issue will not come if the trigger is Kinesis. Could you please suggest how can I extract the Original Data without decoding/Corrupting it ?

EDIT : Adding few more points . The SQS which is the trigger for this lambda01 is the Destination Queue for failed messages of another lambda02 that consumes the Kinesis . On failure the record details (ShardID and SequenceNumber of Kinesis) get pushed to the SQS and lambda01 will get these details do the getRecord from Kinesis based on this data .Lambda02 which is triggered by the Kinesis consumes the Kinesis but doesnot alter/push any records the Kinesis . Some Sample Data Sample Data pushed to Kinesis: { "Attribute1": "<Base64EncodedValue>", "Attribute2:"value1" }

nodejs Code used to decodeBase64

const decodedRecord= Buffer.from(recordFromKinesis.data, 'base64').toString();

I need the Attribute1 value to be intact.

Nephy
  • 11
  • 4
  • You don't provide enough information for people to answer you. – Parsifal Mar 17 '21 at 14:15
  • To start: how does the data get from Kinesis to SQS? Is there a Lambda that copies it? If yes, is that Lambda decoding the Base64 message? – Parsifal Mar 17 '21 at 14:16
  • You can Base64-encode data multiple times without causing corruption, _as long as you decode it an equal number of times._ I suspect that you are decoding one time too many. But without example data and code, it's impossible to say that for sure. – Parsifal Mar 17 '21 at 14:19
  • @Parsifal I have added few more points Hope it will help. – Nephy Mar 18 '21 at 06:13
  • Doesn't really help. Since your messages may contain sensitive data, I understand if you don't want to paste them here. But I recommend looking at the _actual_ messages and trying to decode their content. – Parsifal Mar 18 '21 at 12:51
  • Thanks @Parsifal . I found the issue , even though aws docs suggests `Buffer.from(record.kinesis.data, 'base64').toString('ascii');` . We probably do not require decode it like in this https://stackoverflow.com/questions/53998631/how-to-decode-data-from-kinesis-getrecords-in-javascript-sdk – Nephy Mar 23 '21 at 12:26

0 Answers0