How to get the pure Json string from DynamoDB stream new image?

Question

I've a Dynamodb table with streaming enabled. Also I've created a trigger for this table which calls an AWS Lambda function. Within this lambda function, I'm trying read the new image (Dynamodb item after the modification) from the Dynamodb stream and trying to get the pure json string out of it. My Question is how can i get the pure json string of the DynamoDB item that's been sent over the stream? I'm using the code snippet given below to get the new Image, but I've no clue how to get the json string out of it. Appreciate your help.

public class LambdaFunctionHandler implements RequestHandler<DynamodbEvent, Object> {

@Override
public Object handleRequest(DynamodbEvent input, Context context) {
    context.getLogger().log("Input: " + input);

    for (DynamodbStreamRecord record : input.getRecords()){

        context.getLogger().log(record.getEventID());
        context.getLogger().log(record.getEventName());
        context.getLogger().log(record.getDynamodb().toString());
        Map<String,AttributeValue> currentRecord = record.getDynamodb().getNewImage();

        //how to get the pure json string of the new image
        //..............................................
     }
     return "Successfully processed " + input.getRecords().size() + " records.";
}

}

My real intention here was to send the data from Dynamodb to elastic search server when an item is inserted / updated /Deleted in Dynamodb . I've managed to achieve that using the python lambda code template which is available on AWS user console (create new lambda function section). So without any concern about getting the pure json string out of the Dynamodb stream event, I've been managed to send data directly from Dynamodb stream to Amazon Elastic search service using the above mentioned code template. Hope this helps for someone. — Asanga Dewaguru, Jun 23 '16 at 04:55

Himanshu Parmar · Answer 1 · 2017-05-11T15:00:04.410

Below is the complete code for converting from Dynamo JSON to Standard JSON:

import com.amazonaws.services.dynamodbv2.document.Item;
import com.amazonaws.services.dynamodbv2.document.internal.InternalUtils;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.DynamodbEvent;
import com.amazonaws.services.lambda.runtime.events.DynamodbEvent.DynamodbStreamRecord;
import com.google.gson.Gson;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

/**
 * Main Lambda class to receive event stream, parse it to Survey
 * and process them.
 */
public class SurveyEventProcessor implements
        RequestHandler<DynamodbEvent, String> {

    private static final String INSERT = "INSERT";

    private static final String MODIFY = "MODIFY";

    public String handleRequest(DynamodbEvent ddbEvent, Context context) {

        List<Item> listOfItem = new ArrayList<>();
        List<Map<String, AttributeValue>> listOfMaps = null;
        for (DynamodbStreamRecord record : ddbEvent.getRecords()) {

            if (INSERT.equals(record.getEventName()) || MODIFY.equals(record.getEventName())) {
                listOfMaps = new ArrayList<Map<String, AttributeValue>>();
                listOfMaps.add(record.getDynamodb().getNewImage());
                listOfItem = InternalUtils.toItemList(listOfMaps);
            }

            System.out.println(listOfItem);
            try {
               // String json = new ObjectMapper().writeValueAsString(listOfItem.get(0));
                Gson gson = new Gson();
                Item item = listOfItem.get(0);

                String json = gson.toJson(item.asMap());
                System.out.println("JSON is ");
                System.out.println(json);
            }catch (Exception e){
                e.printStackTrace();
            }
        }


        return "Successfully processed " + ddbEvent.getRecords().size() + " records.";
    }
}

I am getting error as DynamodbEvent is returning com.amazonaws.services.lambda.runtime.events.models.dynamodb.AttributeValue where as the util is expecting com.amazonaws.services.dynamodbv2.model.AttributeValue. I am using aws-lambda-java-events:jar:3.7.0 and aws-java-sdk-dynamodb:jar:1.11.978. Any idea what versions are compatible @Himanshu Parmar — anuj, Mar 20 '21 at 01:04

Guru · Answer 2 · 2018-08-22T21:02:00.677

4

In c# you can convert newImage to pure json by use of DynamoDB Document class

using Amazon.DynamoDBv2.DocumentModel;

var streamRecord = dynamoEvent.Records.First();

var jsonResult=Document.FromAttributeMap(streamRecord.Dynamodb.NewImage).ToJson();

and if you want to go further ahead to convert json to object you can use Newtonsoft

using Newtonsoft.Json;

TModel model = JsonConvert.DeserializeObject(jsonResult);

edited Aug 22 '18 at 21:02

answered Aug 22 '18 at 20:56

Guru

155
2
6

3

This is a Java tagged question, so this C# answer is irrelevant in this case. – JNYRanger Mar 15 '19 at 21:48

score 3 · Answer 3 · answered Aug 08 '16 at 06:11

3

Found a way of doing it cleanly. Using InternalUtils from aws-java-sdk-dynamodb-1.11.15.jar

com.amazonaws.services.dynamodbv2.model.Record streamRecord = ((RecordAdapter) record).getInternalObject();
            // get order ready //
            OrderFinal order = Utils.mapO2Object(
                    InternalUtils.toSimpleMapValue(streamRecord.getDynamodb().getNewImage().get("document").getM()), 
                    OrderFinal.class );

answered Aug 08 '16 at 06:11

shailender arya

31
4

can you bring more detail about this? – Marckaraujo Feb 22 '17 at 19:41
Didnt understand the comment. What details do you want ? – shailender arya Mar 14 '17 at 08:59
2

Where does `Utils.mapO2Object` come from? Is that your own implementation? – EagleBeak Mar 15 '17 at 16:31
Utils.map2Object is our utility function which is a wrapper around jackson convertValue function. Find the function below `public static T map2Object(Map map, Class clazz) { Object t = null; t = OBJECT_MAPPER.convertValue(map, clazz); return (T) t; }` – shailender arya Mar 16 '17 at 06:02

score 3 · Answer 4 · edited Jan 23 '19 at 15:11

3

Just summarizing the answer of Himanshu Parmar:

Map<String, AttributeValue> newImage = record.getDynamodb().getNewImage();
List<Map<String, AttributeValue>> listOfMaps = new ArrayList<Map<String, AttributeValue>>();
listOfMaps.add(newImage);
List<Item> itemList = ItemUtils.toItemList(listOfMaps);
for (Item item : itemList) {
    String json = item.toJSON();
}

edited Jan 23 '19 at 15:11

Peter

5,556
3
23
38

answered Jun 01 '17 at 13:06

lost in binary

544
1
4
11

score 0 · Answer 5 · answered Jul 31 '21 at 18:10

For those stuck with a Map<String, ?> where objects are plain Map, but not Attributes value, you can do the following:

Map<String, AttributeValue> dynamoDbAttributes = 
    objectMapper.convertValue(dynamoDbMap, new TypeReference<Map<String, AttributeValue>>() {});

and then convert this DynamoDB Map into a plain Map (equivalent to the json originally pushed into DynamoDb):

asMap = InternalUtils.toSimpleMapValue(dynamoDbAttributes);

score 0 · Answer 6 · answered Mar 10 '22 at 11:14

For the ones facing issues with AttributeValue conversion refer the below code: https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-events-sdk-transformer/README.md

Map<String, AttributeValue> stringAttributeValueMap = DynamodbAttributeValueTransformer.toAttributeValueMapV1(dynamodb.getNewImage());

            List stringAttributeValueMapList = new ArrayList();
            stringAttributeValueMapList.add(stringAttributeValueMap);
            List<Item> listOfItem = InternalUtils.toItemList(stringAttributeValueMapList);
            Gson gson = new GsonBuilder().setPrettyPrinting().create();
            String updatedJSON = gson.toJson(listOfItem.get(0).asMap());

score 0 · Answer 7 · answered Nov 01 '22 at 19:15

Below is the Method which converts DynamoDB JSON to normal JSON

/**
 * Converts DynamoDB JSON to normal JSON.
 *
 * @param map Input map of String to AttributeValue.
 * @return Returns an ObjectNode containing the normal JSON.
 */

public JsonObject toJsonObject(final Map<String, AttributeValue> map) {
        final JsonNode result = mapToJsonObject(map);
        final ObjectNode objectNode = (ObjectNode) result;
        final ObjectMapper objectMapper = new ObjectMapper();
        String recordObjectString;
        try {
            recordObjectString = objectMapper.writeValueAsString(objectNode);
        } catch (JsonProcessingException e) {
            throw new RuntimeException(e);
        }
        final JsonParser jsonParser = new JsonParser();
        final JsonObject jsonObject = jsonParser.parse(recordObjectString)
                                                .getAsJsonObject();
        return jsonObject;
    }

So in your Case simple call below method like this

// here record is of type DynamodbStreamRecord

toJsonObject(record.getDynamodb().getNewImage());

score -4 · Answer 8 · answered Feb 22 '17 at 20:32

-4

This library do the job: dynamoDb-marshaler

var unmarshalJson = require('dynamodb-marshaler').unmarshalJson;

console.log('jsonItem Record: %j', unmarshalJson(record.dynamodb.NewImage));

answered Feb 22 '17 at 20:32

Marckaraujo

7,422
11
59
97

How to get the pure Json string from DynamoDB stream new image?

8 Answers8

Linked