Apache Nifi 1.7.1 Flatten Json with delimiter

Question

I am using the Flatten Json processor with delimiter _ , this works fine in so far that it does flatten nested records. But I also noticed that it interferes with keys which already have "_" in them. For eg:

{
"first_name": "myfirstname",
"last_name":  "mylastname",
"address": {
            "billing": "mybilling",
            "shipping": "myshipping"
           }
}

flattened to:

{
"[\"first_name\"]": "myfirstname",
"[\"last_name\"]":  "mylastname",
"address_billing": "mybilling",
"address_shipping": "myshipping"
 }

while, 'address' record is flattened as expected, the keys first_name and last_name which have an _ character are also disturbed. Is there a way I can circumvent this? (perhaps a way to escape non record types?)

Answer 1

To workaround this issue you can use either

• JoltTransformJson processor

  (or)

• Keep the seperator(like '|','-'..etc) in FlattenJson which is not present in key names.

1. Using JoltTransformJson Processor:

Spec:

[
  {
    "operation": "shift",
    "spec": {
      "*": "&",
      "address": {
        "*": "address_&"
      }
    }
  }
]

Output:

{
  "first_name" : "myfirstname",
  "last_name" : "mylastname",
  "address_billing" : "mybilling",
  "address_shipping" : "myshipping"
}

2.FlattenJson Processor:

Output:

{
    "first_name": "myfirstname",
    "last_name": "mylastname",
    "address|billing": "mybilling",
    "address|shipping": "myshipping"
}

Answer 2

NiFi 1.7.1 uses an external library called json-flattener in version 0.5.0. This currently unconditionally checks for reserved characters and replaces them with the form you're seeing. The reason for this is so that a document can be unambiguously unflattened again.

There is a pull request that implements an option to disable this feature but it hasn't been merged since July 2017.

In short: There is nothing you can do within NiFi itself to circumvent this at the moment.