How to map nested JSON in Log-stash HTTP Output

Question

I am using Logstash to output JSON message to an API. I am using "mapping" attribute to map my message. See, following piece of my shipper configurations.

output {
    stdout { }
     http {
        url => "http://localhost:8087/messages"
        http_method => "post"
        format => "json"
        mapping => ["MessageId","654656","TimeStamp","2001-12-31T12:00:00","CorrelationId","986565","MessageType","%{log_MessageType}" ,"MessageTitle","%{log_MessageTitle}","Message","%{log_Message}"]
    }
}

This configuration is working fine and is producing following output:

{
  "MessageId": "654656",
  "TimeStamp": "2001-12-31T12:00:00",
  "CorrelationId": "986565",
  "MessageType": "INFO",
  "MessageTitle": "TestTittle",
  "Message": "Sample Message"
}

Input Log Entry:

TID: [0] [ESB] [2016-05-30 23:02:02,602]  INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} -  Configured Registry in 572ms {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService}

Grok Pattern :

TID:%{SPACE}\[%{INT:log_SourceSystemId}\]%{SPACE}\[%{DATA:log_ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:log_TimeStamp}\]%{SPACE}%{LOGLEVEL:log_MessageType}%{SPACE}{%{JAVACLASS:log_MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_Message}

Problem Statement:

I want following output through mapping of HTTP. I wanted a nested JSON type inside my message, how should I add that in the mapping tag.

Expected Output:

{
  "MessageId": "654656",
  "TimeStamp": "2001-12-31T12:00:00",
  "CorrelationId": "986565",
  "MessageType": "INFO",
  "MessageTitle": "TestTittle",
  "Message": "Sample Message",
  "MessageDetail": {
    "FieldA": "65656",
    "FieldB": "192.168.1.1",
    "FieldC": "sample value"

  }
}

I have tried few options but I am receiving errors.

omer, please also add input log and grok patteren. – Waqas Ali Razzaq Jun 02 '16 at 07:09 — Waqas Ali Razzaq, Jun 02 '16 at 07:09
I added the mentioned information as well. – omer khalid Jun 02 '16 at 08:31 — omer khalid, Jun 02 '16 at 08:31

Val · Accepted Answer · 2016-06-07T02:45:41.853

It is not possible to do this with the message mapping in the http output. That mapping can only create a single-level JSON.

What you can do, however, is to construct your JSON message before it reaches the http output using the mutate/add_field filter.

filter {
   grok {
       match => { "message" => "TID:%{SPACE}\[%{INT:SourceSystemId}\]%{SPACE}\[%{DATA:ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:log_TimeStamp}\]%{SPACE}%{LOGLEVEL:log_MessageType}%{SPACE}{%{JAVACLASS:log_MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_Message}" }
   }

   # add additional fields in your event here
   mutate {
      gsub => [
        "log_TimeStamp", "\s", "T",
        "log_TimeStamp", ",", "."
      ]
      add_field => {
        "MessageId" => "654656"
        "TimeStamp" => "%{log_TimeStamp}"
        "CorrelationId" => "986565"
        "MessageType" => "%{log_MessageType}"
        "MessageTitle" => "%{log_MessageTitle}"
        "Message" => "%{log_Message}"
        "[MessageDetail][FieldA]" => "65656"
        "[MessageDetail][FieldB]" => "192.168.1.1"
        "[MessageDetail][FieldC]" => "sample value"
      }
      remove_field => ["@version", "@timestamp", "host", "message", "SourceSystemId", "ProcessName", "log_TimeStamp", "log_MessageType", "log_MessageTitle", "log_Message"]
   }
}
output {
   stdout { codec => "rubydebug" }
   http {
      url => "http://localhost:8087/messages"
      http_method => "post"
      format => "json"
   }
}

You'll get exactly the JSON you expect posted to your HTTP endpoint

{
         "MessageId": "654656",
         "TimeStamp": "2016-05-30T23:02:02.602",
     "CorrelationId": "986565",
       "MessageType": "INFO",
      "MessageTitle": "org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService",
           "Message": "Configured Registry in 572ms {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService}",
     "MessageDetail": {
        "FieldA": "65656"
        "FieldB": "192.168.1.1"
        "FieldC": "sample value"
     }
}

Thanks for your answer, I have tried this, it is working still there are some fields are going to the API, which are not expected, so can you answer my two questions: 1. How to remove all the attributes and fields except the fields that we mentioned in the "mutate" filter? 2. Like "mapping" element of http output also ensures the sequence of the fields, does this mutate element also ensures the sequence of the fields, because the error I am getting in my api is that the fields sequence is not waht expected? Thanks in advance — omer khalid, Jun 06 '16 at 10:40
1) You can use the `mutate/remove_field` filter in order to remove unwanted fields. 2) By definition a JSON hash is not ordered, so there is no notion of sequence in a JSON document. What is the error you're getting from your API? — Val, Jun 06 '16 at 10:46
I've managed to get exactly the JSON you expect with the properties in the exact same order. See my updated answer. — Val, Jun 07 '16 at 02:46
If you have time, can u please look into this question too. http://stackoverflow.com/questions/37679108/unable-to-get-the-parse-value-out-of-multi-line-logs-in-logstash — omer khalid, Jun 07 '16 at 12:16

How to map nested JSON in Log-stash HTTP Output

1 Answers1