bash : run sed -i multiple times

Question

I have a list of string to be replaced in files. However when I run the sed -i (change the file directly), only the last replacement works. I think the last one override all previous changes?

#!/bin/bash

# echo "searching text to be replaced from csv file.."

mapfile -t searchArray < search
mapfile -t replacementArray < replacement
for file in *.txt;
do
echo ">>>processing $file.."
for idx in "${!searchArray[@]}"; do
echo "@@replacing ${searchArray[idx]}"  
    if [[ ${searchArray[idx]} != "" && ${replacementArray[idx]} != "" ]]; then        
    ESCAPED_REPLACE=$(printf '%s\n' "${searchArray[idx]}" | sed -e 's/[\/&]/\\&/g')
    sed -i "s/$ESCAPED_REPLACE/${replacementArray[idx]}/g" "$file"             
    fi        
 done
done

Tried work on a sed file as this:

for idx in "${!searchArray[@]}"; do
    if [[ ${searchArray[idx]} != "" && ${replacementArray[idx]} != "" ]]; 
then        
        ESCAPED_REPLACE=$(printf '%s\n' "${searchArray[idx]}" | sed -e 
's/[\/&]/\\&/g')
        printf 's/%s/%s/g\n' "$ESCAPED_REPLACE" "${replacementArray[idx]}"
    fi 
done >script.sed

for file in *.txt;
do
sed -f script.sed "$file" >outputfile && mv outputfile "$file"  
done

But the search string has some special characters eg "${template(properties).activateDate}(pd -> \"#dateFormatCalc('$pd', 'long', 'dd MMM yyyy','-12Q')\")", how can I escape those special characters in *.sed file? I mean generate that *.sed file programmatically.

Edit: This is my search file (two rows):search string

${template(properties).activateDate}(pd -> \"#dateFormatCalc('$pd', 'long',  'dd MMM yyyy','-12Q')\")
${template(properties).activateDate}(pd -> \"#dateFormatCalc('$pd', 'long', 'dd MMM yyyy','-16')\")

This is my replacement file (two rows):

#dateCalc('dd-mm-yyyy','end','-12','Q')
#dateCalc('dd-mm-yyyy','end','-16','Q')

This is my test file (search and replace). It is a JSON file: code sample pic

   {
       "type": "TEXTBOX",
       "type": "TEXTBOX",
       "label": "${template(properties).activateDate}(pd -> \"#dateFormatCalc('$pd', 'long', 'dd MMM yyyy','-12Q')\")",
       "value": "${template(properties).activateDate}(pd -> \"#dateFormatCalc('$pd', 'long', 'dd MMM yyyy','-12Q')\")",
       "format": "PERCENTAGE",
       "displayTotal": false,
       "deactivated": false
   }

Expected result should be as this:

   {
       "type": "TEXTBOX",
       "type": "TEXTBOX",
       "label": "#dateCalc('dd-mm-yyyy','end','-12','Q')",
       "value": "#dateCalc('dd-mm-yyyy','end','-12','Q')",
       "format": "PERCENTAGE",
       "displayTotal": false,
       "deactivated": false
   }

@tripleee thanks. Yes I am working on sed file, but the search string in searchArray has some special charactors . How can I generate that by code ? the search string is something like this " ${template(properties).activateDate}(pd -> \"#dateFormatCalc('$pd', 'long', 'dd MMM yyyy','-12Q')\") " — ewan, Mar 05 '21 at 07:51
The symptom you describe seems hard to understand; your code certainly looks like it should perform all the substitutions, per se. Do you have DOS line feeds in your input files, by any chance? Or perhaps the problem is simply that you forgot to escape things which needed escaping (for example, an unescaped `$` does not match a literal `$`). — tripleee, Mar 05 '21 at 08:06
Please don't edit your question so that existing answers no longer make sense. Your original attempt tried to run `sed -i` in a loop but now my answer looks weird because you took out the code I was commenting on. — tripleee, Mar 05 '21 at 12:07
I have never seen a place where `paste -d /` did not join columns properly. Are you on some Busybox or DOS-based platform perhaps? Or does your input file contain invisible control characters (maybe DOS line feeds)? — tripleee, Mar 05 '21 at 12:17
If your `paste` really is broken, you can probably work around that with `paste <(sed ...) <(sed ...)` and have the subsequent `sed` script change the separating tab to a slash, too. — tripleee, Mar 05 '21 at 12:18
Are you trying to say that the search patterns should be manipulated to allow arbitrary runs of whitespace, including newlines, where the pattern has a single space? That's significantly harder, and probably beyond what you can easily do with `sed` (though an Awk solution might work nicely). — tripleee, Mar 05 '21 at 12:28
yes, it is hard, took me two days to write the script, In the end I handled those json files manually by searching and replacing in VS code which only took 10 minutes haha. but learned a lot as a bash newbie ... — ewan, Mar 05 '21 at 12:38

tripleee · Accepted Answer · 2021-03-05T09:22:09.813

0

Running sed -i on the same file multiple times is a horrible antipattern. Just do all the substitutions in the same sed script.

You need to escape many more metacharacters than you currently do. See Escape a string for a sed replace pattern for some details. Probably that's the reason many of your substitutions don't appear to work.

Just briefly, try this:

# Escape metacharacters from both files and paste them side by side
paste -d / <(sed -e 's/[]%\/$*.^[]/\\&/g' search) <(sed -e 's/[\/&]/\\&/g' replace) |
# Add necessary decorations to turn this into a valid sed script
sed 's%^%s/%;s%$%/g%' |
# Now pass the generated sed script to another sed instance
sed -f - -i *.txt

You can peel off stuff from the end of the pipeline to see what's being done at each step. I encourage you to do that in order to understand how this works. But just briefly, we turn pat.tern and repl&acement into pat\.tern/repl\&acement and then add the sed command around it so we get s/pat\.tern/repl\&acement/g; then feed that to sed -i.

As always, if your sed does not like -f -, try -f /dev/stdin or, in the absolutely worst case, save the generated file to a temporary file (don't forget to remove it when you are done).

edited Mar 05 '21 at 09:22

answered Mar 05 '21 at 08:05

tripleee

175,061
34
275
318

Untested; without sample data it's hard to know what to expect or test for. – tripleee Mar 05 '21 at 08:05
Updated sample data. Your explanation is very clear. Tried but blocked on the paste step; . Running : paste -d / <(sed -e 's/[]%\/$*.^[]/\\&/g' search) <(sed -e 's/[]%\/$*.^[]/\\&/g' replacement) , it gave wrong result... – ewan Mar 05 '21 at 11:49
Wrong how? Your sample JSON data has a line wrap which prevents `sed` from finding a match on either of the example search phrases, but the generated script looks fine as such. – tripleee Mar 05 '21 at 12:05
For illustration, here is a demo which works on the second search phrase but not the first, simply by trimming newlines from your JSON example; https://ideone.com/BpYDMW – tripleee Mar 05 '21 at 12:11
Updated the expected result . In the final result , there should not have string "${template(properties).activateDate}....*" – ewan Mar 05 '21 at 12:19
That's what I am saying, it works on the second instance but not the first because the input JSON has irregular spaces. – tripleee Mar 05 '21 at 12:22
oh that is bad , we cannot change the JSON format . So it is impossible to achieve that result in bash ? Still not understand why the last sed instance works , so even it reads from *.sed file , still running multiple instances behind? – ewan Mar 05 '21 at 12:27
What do you mean? It runs a single `sed` instance and it replaces all the matches it finds, but it doesn't find all of them because your search expression doesn't precisely match the text in the file. Of course this isn't impossible; but like I suggested elsewhere, perhaps this is becoming more challenging than your question reveals. Perhaps at this point (accept this answer, or post one of your own and accept that, and) ask a new question where you have your _actual_ requirements up front. – tripleee Mar 05 '21 at 12:34
If you just want to allow multiple spaces, not newlines, the fix is comfortably easy; extend the first `sed` script to also replace single spaces with space, space, star to match one or more spaces. – tripleee Mar 05 '21 at 12:37
Typically JSON files are either formatted for human consumption, or for easy machine manipulation as a single long line of text. Your example is neither. – tripleee Mar 05 '21 at 12:39
The JSON file is good , just I cannot edit it properly here , in the real JSON file, no line wrap – ewan Mar 05 '21 at 12:46
1

Pictures of text are almost always a bad idea. See https://meta.stackoverflow.com/questions/303812/discourage-screenshots-of-code-and-or-errors – tripleee Mar 05 '21 at 13:14
But if that picture is really genuine, the code works fine without any additional changes: https://ideone.com/YmDt7m – tripleee Mar 05 '21 at 13:18

bash : run sed -i multiple times

1 Answers1