Jackson @JsonAnySetter ignores values of duplicate key when used with Jackson ObjectMapper treeToValue method

Question

I am using the Jackson library to deserialize JSON. In the JSON I have a few custom fields whose values can be anything, so I am trying to use the @JsonAnySetter and @JsonAnyGetter to obtain the values. The fields and values within the JSON can be duplicated and I would like to obtain everything from the JSON and store it within the map. However, the direct Jackson derealization is storing the last value if there are duplicates in the key.

I have a large JSON file that has many events. I am reading the file event-by-event so that the whole JSON file is not stored within memory. After reading a single event, I check the type of event, based on which I assign it to a different POJO. Following is my sample JSON file consisting of 2 events.

[
  {
    "isA": "Type1",
    "name": "Test",
    "foo": "val1",
    "foo": "val2",
    "bar": "val3",
    "foo": {
      "myField": "Value1",
      "myField": "value2"
    }
  },
  {
    "isA": "Type2",
    "name": "Test1",
    "foo": "val1",
    "foo": "val2",
    "bar": "val3",
    "foo": {
      "myField": "Value1",
      "myField": "value2"
    }
  }
]

Following is the class that is used for deserialization: (I have many fields which are mapped directly during the deserialization and working correctly so omitted for simplicity). This is the class for the type1 event if it's type2

@JsonInclude(JsonInclude.Include.NON_NULL)
@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class Type1
{
    private String name;
    private Map<String, Object> userExtensions;
   
    @JsonAnyGetter
    public Map<String, Object> getUserExtensions() {
        return userExtensions;
    }
    
    @JsonAnySetter
    public void setUserExtensions(String key, Object value) {
        System.out.println("KEY : " + key + " VALUES : " + values);
    }
    
}

As you can observe from above I have a Map field that is used to populate the extensions using the JsonAnySetter. I have a method which will be called for the fields which cannot be directly searlized by Jackson.

I tried setting the System.out within the @JsonAnySetter method and I observed that Jackson does not get the duplicate field within this method:

  @JsonAnySetter
  public void setUserExtensions(String key, Object value) {
    System.out.println(" Key : " + key + " Value : " + value);
  }

I get only the last fields in this. For the above mentioned JSON I get only the last foo which has the value:

"foo" :{
  "myField" : "Value1"
  "myField" : "value2"
}

The first 2 foo with val1 and val2 does not even print within this method.

Following is my Main method which is actually the culprit as I am using the objectMapper.treeToValue of Jackson which does not support the duplicate fields.

public class Main
{
  public static void main (String[]args)
  {
    //File where the JSON is stored
    InputStream jsonStream = Main.class.getClassLoader().getResourceAsStream("InputEPCISEvents.json");
    final JsonFactory jsonFactory = new JsonFactory();
    final JsonParser jsonParser = jsonFactory.createParser (jsonStream);
    jsonParser.setCodec (new ObjectMapper ());
    final ObjectMapper objectMapper = new ObjectMapper();
    
    // Loop until the end of the events file
    while (jsonParser.nextToken() != JsonToken.END_ARRAY) {
        // Get the node
        final JsonNode jsonNode = jsonParser.readValueAsTree();
        // Get the eventType
        final String eventType = jsonNode.get("isA").toString().replaceAll("\"", "");
        
        switch (eventType) {
            case "Type1":
                final Type1 objInfo = objectMapper.treeToValue(jsonNode, Type1.class);
                break;
            case "Type2":
                final Type2 objInfo = objectMapper.treeToValue(jsonNode, Type2.class);
                break;
            default:
                System.out.println("None of the event Matches");
                break;
        }
        
    }
      
  }
}

I wanted to know how can I make Jackson read the duplicate key values so that I can handle them within the @JsonAnySetter method.

I wanted to know if there is a way to handle these scenarios directly using Jackson or I need to build my own custom deserialization. If yes, then how can I build one for only one field within the class?

PS: I tried many things available online but none worked hence posting the same. If found duplicate then I am really sorry.

I noticed that most of the code sample on Stack Overflow read JSON with the Jackson readValue method, but in my case I have a large JSON file that has many events so I am splitting them event by event and using the Jackson objectMapper.treeToValue method to map it to my class. Within this class, I try to map the fields of each event with the respective class fields and if no match found then I am expecting them to be populated using the @JsonAnySetter method.

Answer 1

I think I have a solution that may work for you, or at the very least move you a few steps forward. I created a local test using your files to demonstrate.

I figured out a way to ration about it by looking for how FAIL_ON_READING_DUP_TREE_KEY was implemented. Basically, the code that's responding to it is in JsonNodeDeserializer. If you see how this deserializer looks at _handleDuplicateField, it will basically just replace the newest node with the latest (the side effect you're seeing), and if you set the fail feature it will then conditionally throw an exception. However, by creating an extension of this deserializer class with an overriden method, we can basically implement the behavior of "merge the dupe" instead of "throw an error when you see one". The results of that are below.

The console logs from my test run yield this, demonstrating that "foos" contains all the independent strings and objects...

Since it did the merge at the JSON level instead of at the object level, it seems to have even inadvertently handled the inner myField use case by merging the multiple strings into a single arrayNode... I wasn't even trying to solve for myField at the time, but hey, there you go too!

I didn't handle Type2, because I figure this is enough to get you going.

Of course, you will have to create the file with your contents above in the test package as jsonmerge/jsonmerge.json

Console log results

Object Node Read with Dupes: {"isA":"Type1","name":"Test","foo":["val1","val2",{"myField":["Value1","value2"]}],"bar":"val3"}
Type 1 mapped after merge : Type1 [isA=Type1, name=Test, bar=val3, foos=[val1, val2, {myField=[Value1, value2]}]]
None of the event Matches
EOF

Test main code - JsonMerger.java

package jsonmerge;

import java.io.InputStream;

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.fasterxml.jackson.databind.node.ObjectNode;

/**
 * Project to resolve https://stackoverflow.com/questions/67413028/jackson-jsonanysetter-ignores-values-of-duplicate-key-when-used-with-jackson-ob
 *
 */
public class JsonMerger {

    public static void main(String[] args) throws Exception {

        // File where the JSON is stored
        InputStream jsonStream = JsonMerger.class.getClassLoader()
            .getResourceAsStream("jsonmerge/jsonmerge.json");
        final JsonFactory jsonFactory = new JsonFactory();
        final ObjectMapper objectMapper = new ObjectMapper();

        // Inject a deserializer that can handle/merge duplicate field declarations of whatever object(s) type into an arrayNode with those objects inside it
        SimpleModule module = new SimpleModule();
        module.addDeserializer(JsonNode.class, new JsonNodeDupeFieldHandlingDeserializer());
        objectMapper.registerModule(module);

        final JsonParser jsonParser = jsonFactory.createParser(jsonStream);
        jsonParser.setCodec(objectMapper);

        JsonToken nextToken = jsonParser.nextToken();

        // Loop until the end of the events file
        while (nextToken != JsonToken.END_ARRAY) {
            nextToken = jsonParser.nextToken();

            final JsonNode jsonNode = jsonParser.readValueAsTree();
            if (jsonNode == null || jsonNode.isNull()) {
                System.out.println("EOF");
                break;
            }

            // Get the eventType
            JsonNode getNode = jsonNode.get("isA");
            final String eventType = getNode
                .toString()
                .replaceAll("\"", "");

            switch (eventType) {
            case "Type1":
                final Object obj = objectMapper.treeToValue(jsonNode, JsonNodeDupeFieldHandlingDeserializer.class);
                ObjectNode objNode = (ObjectNode) obj;
                System.out.println();
                System.out.println();

                System.out.println("Object Node Read with Dupes: " + objNode);

                Type1 type1 = objectMapper.treeToValue(objNode, Type1.class);
                System.out.println("Type 1 mapped after merge : " + type1);

                break;
            default:
                System.out.println("None of the event Matches");
                break;
            }

        }

    }
}

Type1.java

package jsonmerge;

import java.util.Collection;
import java.util.LinkedList;
import java.util.List;

import com.fasterxml.jackson.annotation.JsonAnySetter;

public class Type1 {

    public String       isA;
    public String       name;
    public String       bar;
    public List<Object> foos;

    public Type1() {

        foos = new LinkedList<Object>();
    }

    /**
     * Inspired by https://stackoverflow.com/questions/61528937/deserialize-duplicate-keys-to-list-using-jackson
     * 
     * @param key
     * @param value
     */
    @JsonAnySetter
    public void fieldSetters(String key, Object value) {

        // System.out.println("key = " + key + " -> " + value.getClass() + ": " + value);
        // Handle duplicate "foo" fields to merge in strings/objects into a List<Object>
        if ("foo".equals(key) && value instanceof String) {
            foos.add((String) value);
        } else if ("foo".equals(key) && (value instanceof Collection<?>)) {
            foos.addAll((Collection<?>) value);
        }

    }

    @Override
    public String toString() {

        return "Type1 [isA=" + isA + ", name=" + name + ", bar=" + bar + ", foos=" + foos + "]";
    }

}

JsonNodeDupeFieldHandlingDeserializer

package jsonmerge;

import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ObjectNode;

/**
 * Handles JsonNodes by merging their contents if their values conflict into an arrayNode
 * 
 */
@JsonDeserialize(using = JsonNodeDupeFieldHandlingDeserializer.class)
public class JsonNodeDupeFieldHandlingDeserializer extends JsonNodeDeserializer {

    private static final long serialVersionUID = 1L;

    @Override
    protected void _handleDuplicateField(JsonParser p, DeserializationContext ctxt, JsonNodeFactory nodeFactory, String fieldName, ObjectNode objectNode, JsonNode oldValue, JsonNode newValue) throws JsonProcessingException {

        // When you encounter a duplicate field, instead of throwing an exception in this super-logic... (if the feature was set anyway)
//      super._handleDuplicateField(p, ctxt, nodeFactory, fieldName, objectNode, oldValue, newValue);

        // Merge the results into a common arrayNode
        // Note, this assumes that multiple values will combine into an array...
        // And *THAT* array will be used for future nodes to be tacked onto...
        // But if the FIRST thing to combine *IS* an array, then it will be the
        // initial array...
        // You could probably persist some way to track whether the initial array was encountered, but don't want to think about that now..
        ArrayNode asArrayValue = null;

        if (oldValue.isArray()) {
            asArrayValue = (ArrayNode) oldValue;
        } else {
            // If not, create as array for replacement and add initial value..
            asArrayValue = nodeFactory.arrayNode();
            asArrayValue.add(oldValue);
        }

        asArrayValue.add(newValue);
        objectNode.set(fieldName, asArrayValue);

    }

}

Answer 2

With this test data:

{
  "foo" : "val1",
  "foo" : "val2",
  "bar" : "val3",
  "foo" :{
    "myField" : "Value1",
    "myField" : "value2"
  }
}

This code:

import com.fasterxml.jackson.annotation.JsonAnySetter;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.File;

public class Main2 {

    public static void main(String[] args) throws Exception {
        File jsonFile = new File("test.json").getAbsoluteFile();

        ObjectMapper mapper = new ObjectMapper();
        MyObject myObject = mapper.readValue(jsonFile, MyObject.class);
    }
}

class MyObject {
    // attributes

    @JsonAnySetter
    public void manyFoos(String key, Object value) {
        System.out.println(" Key : " + key + " Value : " + value);
    }
}

Prints:

Key : foo Value : val1
Key : foo Value : val2
Key : bar Value : val3
Key : foo Value : {myField=value2}

The problem is that you have duplicate keys on multiple levels. With this code you can handle the duplicates on the outer level, and do as you want, but jackson deserializes the value part. In case of the third foo, the value is a map, and because of that, duplicates are lost.