0

Some confusion here where I have to use filebeat multiline pattern to collec data. Question is how to use multiple pattern ? Here what i use now

multiline.pattern : '^Select'

So for above pattern we can see all word start from select will be match. So my question how about INSERT,UPDATE and DELETE word ?

Also one question can I use below pattern to indicate end of multiline match ?

multiline.flush_pattern: ';'

Any idea or help is highly appreciated

zz10
  • 67
  • 1
  • 10

1 Answers1

0

To your first question:

You can specify multiple words for the beginning of the message within a single regex. So if I understood you correctly, you want to include all log lines that start with Select, INSERT, UPDATE and DELETE. To achieve this you would define a group of valid values like so:

multiline.pattern : '^(Select|INSERT|UPDATE|DELETE)

The pipe-character ( | ) acts as an OR-Operator. Please note that by default regex is case sensitive. So e.g. messages that start with an uppercase SELECT would be ignored in the sample above.

To your second question:

Besides multiline.pattern you have to specify the settings multiline.match and multiline.negate:

  • multiline.match determines if the log lines before or after the pattern should be put into a single event.

  • multiline.negate determines if the following lines have to match the pattern.

So instead of specifying a particular end-character you tell Filebeat that every log line that matches the pattern AND is following that line should get aggregated UNTIL the following line matches again the pattern.

(See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html for a full reference and description).

Example:

Assuming your log file is structured as following:

Select foo from bar\n where baz = 1\n and id =4711;\n\n DELETE from bar\n where baz = null;\n\n INSERT ...

the following config should do the job:

multiline.pattern : '^(Select|INSERT|UPDATE|DELETE)' multiline.match: after multiline.negate: true

I hope I could help you.

apt-get_install_skill
  • 2,818
  • 10
  • 27
  • Hi @apt-get_install, thanks for your reply !! so before admit this is the best answer given here i want to ask one more question where how to exclude certain word/string ? if i set like this ^myword means myword will be ignore right ? – zz10 Mar 26 '20 at 06:16
  • Hi @zz10 thanks for your reply. I will answer your question soon, please be patient. – apt-get_install_skill Mar 26 '20 at 07:25
  • @zz10 if you want to exclude/ignore certain words/characters you should make use of the `exclude_lines` setting of the log input (see https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html). There you can specify another regex. This setting works also in conjunction with multiline settings since the multilines are first aggregated and then will be filtered by the exclude_lines setting. – apt-get_install_skill Mar 26 '20 at 07:57
  • @zz10 does this answer your question? – apt-get_install_skill Mar 26 '20 at 08:02
  • 1
    thanks for your reply, The using exclude lines is not possible for me as am doing the data collection for for multiple logs file as well using single filebeat configuration...Btw thanks so much and really appreciated ! – zz10 Mar 30 '20 at 03:13
  • @zz10 you're welcome, glad I could help you. You may want to take a look at this SO post regarding the exclusion of regex patterns: https://stackoverflow.com/questions/2078915/a-regular-expression-to-exclude-a-word-string . If you cant use exclude_lines then you can build the exclusion in the regex itself. – apt-get_install_skill Mar 30 '20 at 06:15