1
2019-04-01 10:57:35|[a1vx4d9r - ecaf-myself - N2PENL-ECFA0141.india.airtel.itm - servlet:/reserveNumbers/getAvailableNumbers/v1?httpMethodRestrict=POST ][[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO|com.airtel.common.aop.LogExecutionTime|EXECUTION TIME LOGGING METHODNAME: postResponseFromESB EXECUTION TIME: 472 ms

Above is the log format.

Please help in writing a grok filter in logstash that separates the fields as shown below:

timestamp: 2019-04-01 10:57:35
user_id:a1vx4d9r
project name: ecaf-myself
host_name: N2PENL
api_name:ECFA0141.india.airtel.itm - servlet:/reserveNumbers/getAvailableNumbers/v1?httpMethodRestrict=POST 
thread_id: [ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'
log_level: INFO
Method_name:postResponseFromESB 
Method_time:472
Harish
  • 1,433
  • 9
  • 21
Vibhu Garg
  • 67
  • 8

1 Answers1

2

Writing grok filters are easy with all the required patterns already defined here. You could use this app to try out your grok filter, and see if it is matching your log pattern: http://grokdebug.herokuapp.com/

For the above example, the below grok filter will work. You can check and fine tune it to fit your needs using the above two resources:

%{TIMESTAMP_ISO8601}.*\[%{WORD:user_id}* - %{NOTSPACE:project_name}* - %{WORD:host_name}*-%{DATA:api_name} \]%{GREEDYDATA:thread}\] %{WORD:log_level}.*METHODNAME: %{WORD:Method_name}.*EXECUTION TIME: %{INT:Method_time}
Harish
  • 1,433
  • 9
  • 21
  • Sir, in case suppose user_id is not present and log be like- 2019-04-01 10:57:35|[a1vx4d9r - - - servlet:/reserveNumbers/getAvailableNumbers/v1?httpMethodRestrict=POST ][[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO|com.airtel.common.aop.LogExecutionTime|EXECUTION TIME LOGGING METHODNAME: postResponseFromESB EXECUTION TIME: 472 ms . How to avoid grokparseFailure in case any field is not present and thus, we are not able to extract the same. – Vibhu Garg Apr 15 '19 at 11:03
  • if a field is absent for some entries, it means that the field is optional. Look at this question for handling such fields: [Logstash-optional-fields-in-logfile](https://stackoverflow.com/questions/30083719/logstash-optional-fields-in-logfile) – Harish Apr 16 '19 at 04:38
  • https://stackoverflow.com/questions/55990067/parsing-through-grok-filter - Please help in this answer. – Vibhu Garg May 06 '19 at 16:07