RegEx for replacing capture group content using sed

Question

Have inputs.conf files in multiple directories that needs to match and parse each stanza and modify the index= to index=secure. This are files type in inputs.conf and also do run the script to locate the inputs file in this dir (_GWAS_pr_linux_t1/local/inputs.conf) to modify the index

[monitor:///var/log/messages]
index=BDBD_np
sourcetype=syslog

[monitor:///var/log/cron]
index=NDNW_np
sourcetype=syslog

[monitor:///var/log/maillog]
index=BSKX_np
sourcetype=syslog

[monitor:///var/log/secure]
index=NDNDK_np
sourcetype=syslog

[monitor:///var/log/spooler]
index=DNWN_np
sourcetype=syslog

[monitor:///var/log/audit/audit.log]
index=XBJB_np
sourcetype=syslog


    sed -i -e 's/.*(?s)((\[monitor\:\/\/\/var\/log\/messages|secure\]).*?)(?:(?:\r*\n){2})' /index=secure *linux*/local/inputs.conf

############################
match each stanza and modify index name to index=windows
inputs file in this dir (_GWAS_pr_window_t1/local/inputs.conf)


    [WinEventLog://Application]
    checkpointInterval = 5
    current_only = 0
    disabled =0 
    start_from = oldest
    index = 

    [WinEventLog://Security]
    checkpointInterval = 5
    current_only = 0
    disabled =0 
    start_from = oldest
    index = 

    [WinEventLog://System]
    checkpointInterval = 5
    current_only = 0
    disabled =0 
    start_from = oldest
    index = 

    [WinEventLog://ForwardedEvents]
    checkpointInterval = 5
    current_only = 0
    disabled =0 
    start_from = oldest
    index = 

    [WinEventLog://Setup]
    checkpointInterval = 5
    current_only = 0
    disabled =0 
    start_from = oldest
    index = 

    sed -i -e 's/.*(?s)((\[WinEventLog:\/\/Application|Security|System|ForwardedEvents|Setup\]).*?)(?:(?:\r\n){2}) /index=window *window*/local/inputs.conf

Please edit your question and show us clear input along with the output you expect to see. — Tim Biegeleisen, May 31 '19 at 05:01
This what i am expected but getting error with the output [monitor:///var/log/messages] sourcetype=linux index=secure [monitor:///var/log/secure] sourcetype=linux index=secure — user10965163, May 31 '19 at 11:51

The fourth bird · Answer 1 · 2019-05-31T15:34:44.893

0

Instead of using sed which reads line by line, you might use perl with the -0 mode. You could capture for example this part in group 1:

[monitor:///var/log/messages]
sourcetype=linux
index=

Followed by matching power afterwards.

Then in the replacement you could referer to group 1 and add secure $1secure so that will replace the matched power.

(\[monitor:///var/log/(?:messages|secure)\](?:(?:\R.*){2})index=)power

Regex demo

perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure)\](?:(?:\R.*){2})index=)power#$1secure#g' inputs.conf

Result:

[monitor:///var/log/messages]
sourcetype=linux
index=secure

Update

If you want to match any of the listed words from the comments and replace those with what comes after index, you could use 2 capturing groups:

perl -0p -i.bak -e 's#(\[monitor:///var/log/(messages|secure|cron|spooler|audit\.log)\](?:(?:\R.*){2})index=).*#$1$2#g' inputs.conf

Pattern

(\[monitor:///var/log/(messages|secure|cron|spooler|audit\.log)\](?:(?:\R.*){2})index=).*

Regex demo

Result:

[monitor:///var/log/messages]
  sourcetype=linux
index=messages

[monitor:///var/log/secure]
  sourcetype=linux
index=secure

edited May 31 '19 at 15:34

answered May 31 '19 at 13:51

The fourth bird

154,723
16
55
70

This is awesome but some of the stanza index name will not be power, it has over hundreds of different index name so how do i match each stanza that startwith [monitor:///var/log and endwith messages, secure, cron, spooler and audit.log] and change the index name without modify any index name that do not reference to this stanzas [monitor:///var/log/messages|secure|cron|maillog|spooler|audit.log] sourcetype=linux or whatever index=whatever – user10965163 May 31 '19 at 15:21
@user10965163 Do you mean like this? https://regex101.com/r/MqBVFW/2 or if the index name can not contain a space https://regex101.com/r/MqBVFW/3 or If you want to match one of the words and replace that with the value of index: https://regex101.com/r/MqBVFW/4 – The fourth bird May 31 '19 at 15:27
This are all the stanza i want to match and modify the index name and some of index name i don`t know so i want the script to match this stanza and change any type of index name it has [monitor:///var/log/cron] sourcetype=linux_secure index= [monitor:///var/log/secure] sourcetype=linux_secure index=power [monitor:///var/log/messages] sourcetype=linux index=power [monitor:///var/log/spooler] sourcetype=syslog index = linux [monitor:///var/log/audit/audit.log] sourcetype=syslog index= [monitor:///var/log/maillog] sourcetype=syslog index= – user10965163 May 31 '19 at 15:40
@user10965163 Try it like this https://regex101.com/r/KTUlut/1 Now you can match all the variants and use your own replacement. – The fourth bird May 31 '19 at 15:53
1

Super great, so should it be like this perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*$1secure_linux inputs.conf – user10965163 May 31 '19 at 16:52
Getting error with this, perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*$1secure_linux inputs.conf – user10965163 May 31 '19 at 17:19
perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*#$1secure_linux#g' inputs.conf – user10965163 May 31 '19 at 17:22
Please all the directories is in this path: /tmp/apps$ and if i stand at the path and run the script it says No such file or directory. how do achieve this perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*#$1secure_linux_pr#g' *linux*/local/inputs.conf perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*#$1secure_linux_np#g' *linux*/local/inputs.conf _GWAS_pr_linux_t1/local/inputs.conf _GWAS_np_linux_t1/local/inputs.conf – user10965163 May 31 '19 at 17:48
I tested this on ubuntu an ran it from the same dir as where the file is: `perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*#$1secure_linux_pr#g' inputs.conf` – The fourth bird May 31 '19 at 19:16
the inputs.conf is in this dir deployment-apps# _GWAS_pr_linux_t1/local/inputs.conf, _GWAS_np_linux_t1/local/inputs.conf so if run the script it say No such file or directory deployment-apps# sed -E '/^\[/{h;b};G;s/^(index=)power\n.*(messages|secure)\]$/\1secure/;P;d' inputs.conf _GWAS_pr_linux_t1/local/inputs.conf _GWAS_pr_linux_t1/local/inputs.conf – user10965163 May 31 '19 at 19:43
could not recorgnized dir path to inputs.conf _GWAS_pr_linux_t1/local/inputs.conf _GWAS_pr_linux_t1/local/inputs.conf perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*#$1secure_linux_pr#g' *linux*/local/inputs.conf – user10965163 May 31 '19 at 20:02
In the comments you can format the code parts by using `backticks` to make it read a bit better. What is your os? Did you try running the code from the same dir as the file? You have perl installed right? – The fourth bird May 31 '19 at 20:09
i have about 100s of apps in this dir so i place the script at deployment-apps$ and run it and after i run, it says No such file or directory like example of apps in deployment-apps$ _GWAS_pr_linux_t1/local/inputs.conf _GWAS_pr_linux_t1/local/inputs.conf – user10965163 May 31 '19 at 20:19
please how do i achieved this – user10965163 May 31 '19 at 20:24
The code works great with single file but file and directory not working for me, still getting errors: perl -0p -i.bak -e 's#(\[monitor:///var/log/(?:messages|secure|cron|spooler|audit/audit\.log|maillog)\](?:(?:\R.*){2})index=).*#$1secure_linux_pr#g' *linux*/local/inputs.conf – user10965163 May 31 '19 at 22:25
Perhaps these pages [page1](https://stackoverflow.com/questions/19995368/perl-find-and-replace-specific-string-in-multiple-text-file) or [page2](https://stackoverflow.com/questions/31774499/apply-multiple-find-and-replace-regex-queries-on-multiple-txt-files-using-a-per) can be helpful – The fourth bird Jun 01 '19 at 09:54

score 0 · Answer 2 · answered May 31 '19 at 18:49

0

This might work for you (GNU sed):

sed -E '/^\[/{h;b};G;s/^(index=)power\n.*(messages|secure)\]$/\1secure/;P;d' file

This appends the first line of each stanza to every other line and using pattern matching, replaces power by secure.

answered May 31 '19 at 18:49

potong

55,640
6
51
83

the inputs.conf is in this dir deployment-apps# _GWAS_pr_linux_t1/local/inputs.conf, _GWAS_np_linux_t1/local/inputs.conf so if run the script it say No such file or directory deployment-apps# sed -E '/^\[/{h;b};G;s/^(index=)power\n.*(messages|secure)\]$/\1secure/;P;d' inputs.conf _GWAS_pr_linux_t1/local/inputs.conf _GWAS_pr_linux_t1/local/inputs.conf – user10965163 May 31 '19 at 19:43
so the script has to match each stanza and change the index name to index=secure_np or secure_pr – user10965163 May 31 '19 at 19:55
Some of the index name endwith np and pr so the script has to match each stanza and change the index name to index=secure_np or secure_pr – user10965163 May 31 '19 at 19:59
@user10965163 please amend the question to show before and after states (also include all edge cases), I find it very difficult to understand from your comments. – potong Jun 01 '19 at 09:50

RegEx for replacing capture group content using sed

2 Answers2