Apply a mask on a JSON to keep only mandatory data

Question

I have an API which takes some potentially mandatory data to create a temporary session:

e.g.: at first, my POST /signup endpoint user need to send me the following data:

{
  "customer": {
    "age": 21,
    "ssn": "000 00 0000",
    "address": {
      "street": "Customer St.",
      "phone": "+66 444 333 222"
    }
  }
}

Let's call it JSON a.

On the other hand, I have some legal partners which requires some of these data, but not all of them:

e.g.:

{
  "customer": {
    "ssn": "The SSN is mandatory to register against XXX Company",
    "address": {
      "phone": "XXX Company will send a text message to validate your registration"
    }
  }
}

Let's call it JSON b.

Due to recent legal restrictions, in my information system, I have to keep only mandatory data for the user to carry with his chosen workflow.

Hence my question: is there a function (either built in Jackson or some other JSON handling library or an algorithm you would recommend) I could apply such that given JSON b and JSON a, it would output the following JSON:

{
  "customer": {
    "ssn": "000 00 0000",
    "address": {
      "phone": "+66 444 333 222"
    }
  }
}

---

Thinking a bit, I found something which might be a solution:

• merge JSON a with JSON b, on conflict take JSON b values, name the result JSON c
• make a diff (drop equals data) between JSON c and JSON a, on conflict take JSON a values

I know merging two JSON using Jackson can be done using com.fasterxml.jackson.databind.ObjectMapper#readerForUpdating, so my question could be reduced to: is there a way to make a diff between two JSON and give a conflict resolution function?

Answer 1

I'd suggest to use token/event/stream-based solution. The following is just an illustration using tiny parser/generator lib https://github.com/anatolygudkov/green-jelly (both Gson and Jackson also provide stream-oriented API):

import org.green.jelly.AppendableWriter;
import org.green.jelly.JsonEventPump;
import org.green.jelly.JsonNumber;
import org.green.jelly.JsonParser;

import java.io.StringWriter;
import java.io.Writer;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;

public class FilterMyJson {

    private static final String jsonToFilter = "{\n" +
            "  \"customer\": {\n" +
            "    \"age\": 21,\n" +
            "    \"ssn\": \"000 00 0000\",\n" +
            "    \"address\": {\n" +
            "      \"street\": \"Customer St.\",\n" +
            "      \"phone\": \"+66 444 333 222\"\n" +
            "    }\n" +
            "  }\n" +
            "}";

    public static void main(String[] args) {
        final StringWriter result = new StringWriter();

        final JsonParser parser = new JsonParser();
        parser.setListener(new MyJsonFilter(result, "age", "street"));
        parser.parse(jsonToFilter); // if you read a file with a buffer,
        // call parse() several times part by part in a loop until EOF
        parser.eoj(); // and then call .eoj()

        System.out.println(result);
    }

    static class MyJsonFilter extends JsonEventPump {
        private final Set<String> objectMembersToFilter;
        private boolean currentObjectMemberIsAllowed;

        MyJsonFilter(final Writer output, final String... objectMembersToFilter) {
            super(new AppendableWriter<>(output));
            this.objectMembersToFilter = new HashSet<>(Arrays.asList(objectMembersToFilter));
        }

        @Override
        public boolean onObjectMember(final CharSequence name) {
            currentObjectMemberIsAllowed =
                    !objectMembersToFilter.contains(name.toString());
            return super.onObjectMember(name);
        }

        @Override
        public boolean onStringValue(final CharSequence data) {
            if (!currentObjectMemberIsAllowed) {
                return true;
            }
            return super.onStringValue(data);
        }

        @Override
        public boolean onNumberValue(final JsonNumber number) {
            if (!currentObjectMemberIsAllowed) {
                return true;
            }
            return super.onNumberValue(number);
        }
    }
}

prints:

{
 "customer":
 {
  "ssn": "000 00 0000",
  "address":
  {
   "phone": "+66 444 333 222"
  }
 }
}

The code is quite simplified. For now it filters out only string and number scalars. No object hierarchy is supported. You may need to improve the code for some cases therefore.

Props of such type of solution:

• the file/data doesn't require to be loaded entirely into memory, you can process megs/gigs with no problems
• it works much more faster, especially for large files
• it's easy to implement any custom type/rule of transformation with this pattern. For example, it is easy to have your filter to be parametrized, you don't have to recompile the code each time your data structure is changed

Answer 2

Thanks to https://github.com/algesten/jsondiff which gave me the keywords to find a more maintained https://github.com/java-json-tools/json-patch (ability to use your own ObjectMapper), I have found my answer (see mask() method):

import static org.assertj.core.api.AssertionsForClassTypes.assertThat;

import com.fasterxml.jackson.annotation.JsonMerge;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.github.fge.jsonpatch.diff.JsonDiff;
import java.math.BigInteger;
import lombok.AccessLevel;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.SneakyThrows;
import lombok.ToString;
import org.junit.jupiter.api.Test;

class JsonPatchTest {

  private static final ObjectMapper mapper = new ObjectMapper();

  @Getter
  @Builder
  @ToString
  @EqualsAndHashCode
  @NoArgsConstructor(access = AccessLevel.PRIVATE)
  @AllArgsConstructor(access = AccessLevel.PRIVATE)
  public static class Data {

    @JsonMerge Customer customer;

    @Getter
    @Builder
    @ToString
    @EqualsAndHashCode
    @NoArgsConstructor(access = AccessLevel.PRIVATE)
    @AllArgsConstructor(access = AccessLevel.PRIVATE)
    public static class Customer {

      @JsonMerge BigInteger age;
      @JsonMerge String     ssn;
      @JsonMerge Address    address;

      @Getter
      @Builder
      @ToString
      @EqualsAndHashCode
      @NoArgsConstructor(access = AccessLevel.PRIVATE)
      @AllArgsConstructor(access = AccessLevel.PRIVATE)
      public static class Address {

        @JsonMerge String street;
        @JsonMerge String phone;
      }
    }

    @SneakyThrows
    Data merge(Data parent) {

      var originCopyAsString = mapper.writerFor(this.getClass()).writeValueAsString(this);

      var parentAsString = mapper.writerFor(this.getClass()).writeValueAsString(parent);
      var parentCopy     = mapper.readerFor(this.getClass()).readValue(parentAsString);

      var clone = mapper.readerForUpdating(parentCopy).readValue(originCopyAsString);

      return (Data) clone;
    }
  }

  @SneakyThrows
  @Test
  void mask() {

    final var diff = JsonDiff.asJsonPatch(mapper.readTree(jsonC()), mapper.readTree(jsonB()));

    final var masked = diff.apply(mapper.readTree(jsonA())).toPrettyString();

    assertThat(masked).isEqualToIgnoringWhitespace(masked());

  }

  private String jsonA() {
    return "{\n"
           + "  \"customer\": {\n"
           + "    \"age\": 21,\n"
           + "    \"ssn\": \"000 00 0000\",\n"
           + "    \"address\": {\n"
           + "      \"street\": \"Customer St.\",\n"
           + "      \"phone\": \"+66 444 333 222\"\n"
           + "    }\n"
           + "  }\n"
           + "}";
  }

  private String jsonB() {
    return "{\n"
           + "  \"customer\": {\n"
           + "    \"ssn\": \"The SSN is mandatory to register against XXX Company\",\n"
           + "    \"address\": {\n"
           + "      \"phone\": \"XXX Company will send a text message to validate your registration\"\n"
           + "    }\n"
           + "  }\n"
           + "}";
  }

  @SneakyThrows
  private String jsonC() {
    final Data dataA = mapper.readerFor(Data.class).readValue(jsonA());
    final Data dataB = mapper.readerFor(Data.class).readValue(jsonB());

    final Data merged = dataB.merge(dataA);

    return mapper.writerFor(Data.class).writeValueAsString(merged);
  }

  @SneakyThrows
  private String masked() {
    return "{\n"
           + "  \"customer\": {\n"
           + "    \"ssn\": \"000 00 0000\",\n"
           + "    \"address\": {\n"
           + "      \"phone\": \"+66 444 333 222\"\n"
           + "    }\n"
           + "  }\n"
           + "}";
  }

}