Sed dynamically search for string with and without '+' character

Question

So I have a configuration file that is parsed by various scripts, therefore the format can't be changed, although the contents can as long as the format is being strictly followed. This file by default contains strings such as

multiconfig:nmb-devel:nmb-trs-devel
multiconfig:nmb-deploy:nmb-trs-deploy
multiconfig:nmb-deploy:nmb-trs-deploy+

multiconfig:ijk-devel:ijk-trs-devel
multiconfig:ijk-deploy:ijk-trs-deploy
multiconfig:ijk-deploy:ijk-trs-deploy+

multiconfig:qrs-devel:qrs-trs-devel
multiconfig:qrs-deploy:qrs-trs-deploy
multiconfig:qrs-deploy:qrs-trs-deploy+

Currently, I have a script that parses these configurations (multiconfig:...) into an array, as well as an array of configurations to replace these original configurations. e.g.

disregard the following tail operation, the ACTUAL conf.txt contains another match to be skipped, which is what is successfully done.

TARGETS="multiconfig:new-devel:new-trs-devel multiconfig:newer-devel:newer-trs-devel multiconfig:newest-devel:newest-trs-devel"
NEW_TARGETS_ARR=( $TARGETS )
OLD_TARGETS_ARR=($(sed -n '/multiconfig/p' conf.txt | tail -n +2 | awk '!seen[$0]++'))

Note: Not the sed operation in question

These work fine, and result in proper arrays such as:

NEW_TARGETS_ARR: multiconfig:new-devel:new-trs-devel, multiconfig:newer-devel:newer-trs-devel, multiconfig:newest-devel:newest-trs-devel

OLD_TARGETS_ARR: multiconfig:nmb-devel:nmb-trs-devel, multiconfig:nmb-deploy:nmb-trs-deploy, multiconfig:nmb-deploy:nmb-trs-deploy+, multiconfig:ijk-devel:ijk-trs-devel, multiconfig:ijk-deploy:ijk-trs-deploy, multiconfig:ijk-deploy:ijk-trs-deploy+, multiconfig:qrs-devel:qrs-trs-devel, multiconfig:qrs-deploy:qrs-trs-deploy, multiconfig:qrs-deploy:qrs-trs-deploy+

My objective is to replace the old configurations by the new configurations.

My method of doing this right now is by looping through the OLD_TARGETS_ARR and replacing each OLD_TARGETS_ARR[index] with the NEW_TARGETS_ARR[index] as such:

for i in ${!OLD_TARGETS_ARR[@]}
do
        if [ "${NEW_TARGETS_ARR[$i]}" = "" ]; then
                NEW_TARGETS_ARR[$i]="[EMPTY]"
        fi


        echo $i
        echo "OLD TARGET $i: ${OLD_TARGETS_ARR[$i]}"
        echo "NEW TARGET $i: ${NEW_TARGETS_ARR[$i]}"
        echo "sed -i \"s/${OLD_TARGETS_ARR[$i]}/${NEW_TARGETS_ARR[$i]}/g\" conf.txt"
        sed -i "s/${OLD_TARGETS_ARR[$i]}/${NEW_TARGETS_ARR[$i]}/g" conf.txt
done

Now, theoretically what this should (or what I want) result in is as follows

conf.txt

multiconfig:new-devel:new-trs-devel
multiconfig:newer-devel:newer-trs-devel
multiconfig:newest-devel:newest-trs-devel

[EMPTY]
[EMPTY]
[EMPTY]

[EMPTY]
[EMPTY]
[EMPTY]

Although what this actually results in is as follows

conf.txt

multiconfig:new-devel:new-trs-devel
[EMPTY]
[EMPTY]+

[EMPTY]
[EMPTY]
[EMPTY]+

[EMPTY]
[EMPTY]
[EMPTY]+

While I haven't figured out why more entries than are necessary are getting replaced with '[EMPTY]', I have figured out that on the '+' on 'deploy+' is not getting parsed, and must be escaped in some way, shape, or form. Given that these sed operations are dynamic, I can't simply add a \ in front of the '+'. First, how could I ensure that the '+' is getting parsed as part of the string to search for in the sed operation?

Second, I probably am misunderstanding some basic part of sed, or my loop, although why is everything except for the first match getting replaced with '[EMPTY]'?

I appreciate any input anyone can give,

thank you all in advance!

score 1 · Answer 1 · answered Mar 16 '22 at 22:21

Assuming you are trying to match nmb, using your NEW_TARGETS array, you could try this implementation

#!/usr/bin/env bash

NEW_TARGETS=(multiconfig:new-devel:new-trs-devel multiconfig:newer-devel:newer-trs-devel multiconfig:newest-devel:newest-trs-devel)

i=0
while read -r line; do 
    sed "/^multiconfig:nmb/s/.*/${NEW_TARGETS[$i]}/;/new\|^$/! c\[EMPTY]" <<< $line
    i=$((i+1))
done < input_file

OUTPUT

multiconfig:new-devel:new-trs-devel
multiconfig:newer-devel:newer-trs-devel
multiconfig:newest-devel:newest-trs-devel

[EMPTY]
[EMPTY]
[EMPTY]

[EMPTY]
[EMPTY]
[EMPTY]

Fravadona · Answer 2 · 2022-03-17T13:02:08.833

The main problem of using sed for replacing string literals is that you need to escape those strings for them not to have any special meaning in this context (see for example Escape a string for a sed replace pattern).

An other problem, specific to your code, is that you're calling sed inside a shell loop. That is extremely slow and should be avoided when possible. You could work around that by building one sed command containing all the sub-commands (for example sed 's/old1/new1/;s/old2/new2/;...') but that's sub-optimal because each line will be tested by each sub-command, and also error prone (for example when new1 is the target of a replacement in a following sub-command).

With all that in mind, sed doesn't seem to be the best tool for the job, especially when you have an other standard tool like awk that can do it efficiently:

#!/bin/bash

NEW_TARGETS_ARR=(
    multiconfig:new-devel:new-trs-devel
    multiconfig:newer-devel:newer-trs-devel
    multiconfig:newest-devel:newest-trs-devel
)
OLD_TARGETS_ARR=(
    $(
        sed -n '/multiconfig/p' conf.txt |
        tail -n +2 |
        awk '!seen[$0]++'
    )
)
for (( i = ${#NEW_TARGETS_ARR[@]}; i < ${#OLD_TARGETS_ARR[@]}; i++))
do
     NEW_TARGETS_ARR[i]='[EMPTY]'
done

awk -v from="${OLD_TARGETS_ARR[*]}" -v to="${NEW_TARGETS_ARR[*]}" '
    BEGIN{
        fromCount = split(from,fromArr)
        toCount = split(to,toArr)
        for (i=1; i<=fromCount; i++)
            tr[fromArr[i]] = toArr[i]
    }
    $1 in tr { $1 = tr[$1] }
    1
' conf.txt

multiconfig:new-devel:new-trs-devel
multiconfig:newer-devel:newer-trs-devel
multiconfig:newest-devel:newest-trs-devel

[EMPTY]
[EMPTY]
[EMPTY]

[EMPTY]
[EMPTY]
[EMPTY]

Explanations

Once OLD_TARGETS_ARR and NEW_TARGETS_ARR are filled, I pass them as arguments to awk (as space delimited strings):

In the BEGIN { ... } block I re-split the string arguments into arrays and build a associative array for translating an old value to its new value.
$1 in tr { $1 = tr[$1] } means that if the first column of the current line is a target for replacement then I do the replacement.
1 prints the current line.

This actually comes very close to my desired solution, although as this question pertains to the sed method, could you provide an explanation of the logic? Thanks! — etfreima, Mar 17 '22 at 01:46
Thank you so much! I accepted this as the answer originally as I was able to make it work, although I was also able to make @potong's answer work, and as theirs is slightly more relevant to the direct question in that is uses sed. Ultimately I do agree awk is potentially the better tool here. Thank you for a great answer and great details! — etfreima, Mar 17 '22 at 18:09

score 1 · Accepted Answer · answered Mar 17 '22 at 11:24

This might work for you (GNU sed & bash):

sed -E '1{x;s/.*/echo '"${NEW_TARGETS_ARR[*]}"'/e;x}
        /multiconfig/!b;g;//!{s/.*/[EMPTY]/p;d};s/ .*//;x;s/\S+ ?//;x' file

Slurp the new targets into the sed hold space (space separated).

Focus only on lines that contain multiconfig.

Replace the current line by the new targets.

If the current line is empty, replace it with [EMPTY], print and then delete it.

Otherwise, remove all but the first target, swap to the hold space and prep for the next target by removing the first target and its following space, then swap back to the current line and print it.

Sed dynamically search for string with and without '+' character

3 Answers3

Explanations