3

I am using EvaluateJsonPath processor in NiFi to specify a composite primary key for writing my JSON data into elasticsearch. I have to create an attribute called 'key' by concatenating two attributes, lets say 'attr1' and 'attr2'. In EvaluateJsonPath configuration, I added a property 'key' with value '${attr1}${attr2}' (I'm using this key to avoid redundancy in elasticsearch). But I'm getting an error like : 'key' is invalid because specified expression was not valid: ${attr1}${attr2}.

What is the proper syntax for concatenating 2 attributes in a json record using EvaluateJsonPath processor?

Skn
  • 140
  • 2
  • 2
  • 13

2 Answers2

8

I assume you mixed the term attributes with different contexts. I did that when I picked up NiFi for the first time and started working with JSON. To get things clarified, there are two types of attributes in this particular case:

  1. FlowFile attributes - These are the attributes that are sort of like metadata of the FlowFile, like filename, filesize, mime.type, etc. When you add an attribute using a processor like UpdateAttribute they get added to the metadata of the FlowFile i.e. FlowFile attributes not to the FlowFile content itself.
  2. JSON attributes - These are the fields that are in the FlowFile content. Ex: Name is an attribute here -> { "Name" : "Smith, John"}

Having said that, using expressions like ${projectName}, ${filename} are evaluated against the FlowFile attributes. This syntax is called NiFi Expression Language. they won't be evaluated against the content of the FlowFile. EvaluateJsonPath takes a JSON path expression like $.Name

So for example, lets assume you have the following JSON document in the FlowFile content:

{
    "attr1": "some value for attr1",
    "attr2": "some value for attr2"
}

In this case, you would have to configure the EvaluateJsonPath processor with new properties which look like the following:

attribute1 : $.attr1
attribute2 : $.attr2

Once done, flowfiles that pass through the EvaluateJsonPath will have the above two attributes added to the flowfile attributes which you can read using NiFi Expression like ${attribute1}. Then connect EvaluateJsonPath processor to UpdateRecord processor where you can add the combined composite key with the help of NiFi Expression Language to the FlowFile JSON content.

Useful Links

Sivaprasanna Sethuraman
  • 4,014
  • 5
  • 31
  • 60
  • Thanks for the answer. It solved the issue for me. I used EvaluateJsonPath processor for creating 2 flowfile attributes (set destination ='flowfile attribute' in configuration) with values same as of attr1 and attr2 that exist in flowfile content. Then using UpdateAttribute processor, I created another flowfile attribute as key = ${attr1}${attr2} ( concatenation with a special character like ${attr1}:${attr2} will also work ). I'm using this key in PutElasticsearchHttp processor as a primary key ( with index='upsert' ). Once again, thanks for the help :) – Skn Aug 14 '18 at 06:48
  • 1
    Can we concatenate two Json attributes into one (in `EvaluateJsonPath`)? Like `attribute3 : $.attr1+$.attr2`. – Amber Dec 14 '18 at 09:48
1

EvaluateJsonPath expects a JSONPath statement, it looks like in this case you want to take two attributes and create one from them using NiFi Expression Language. If that's the case, then UpdateAttribute should be what you want. If you're looking to insert the composite key into the content, then UpdateRecord should be what you want.

mattyb
  • 11,693
  • 15
  • 20
  • I don't want the new attribute to get added to the record. As you suggested, I'm using UpdateAttribute and added a property 'es_key' in it with value '${projectname}' where projectname is an attribute that exist in every JSON record I have. But the PutElasticsearchHttp after UpdateAttribute writes nothing to the index. – Skn Aug 13 '18 at 13:15
  • 2
    Matt's point is that `UpdateAttribute` will use EL to build the key you want and store it as an *attribute* on the flowfile without modifying the *content* of the record. If both attributes `attr1` and `attr2` are derived from the JSON content, then you would need the `EvaluateJsonPath` processor *first* to extract the JSON values into attributes, then use `UpdateAttribute` to combine them. – Andy Aug 13 '18 at 17:18