sed command inside while loop is not working for ubuntu

Question

I have two files; the first includes patterns ( file.txt) that I want to search for in the second file (file.cfg).

Once the patter is found in "file.cfg" I want to remove it + whatever comes after it until next Hello that comes at the beginning of the line.

I have created the below script but its not working :

#! /bin/bash
cat file.txt | while read LINE; do
echo $LINE
    sed -i "/^$LINE$/,/^Hello/{//p;d;}" "file.cfg"
    sed -i "/^$LINE$/d" "file.cfg"
done

It was working fine yesterday on test files, Today I have modified the file's name, and It stopped working.

I am not sure If I changed something by mistake, but if I will use the below from Ubuntu command line, it works :

sed -i "/^Hello World$/,/^Hello/{//p;d;}" "file.cfg"

Also, I added echo in the loop, and I can see each line in "file.txt"

To provide further information, I will give an example of what I need to achieve with this code :

"file.txt" contains patterns I need to find a match in "file.cfg" once the pattern is found, I need to remove it, and anything comes after it until next Hello.

sed -i "/^$LINE$/,/^Hello/{//p;d;}" "file.cfg" -- > this line should remove anything in between.

sed -i "/^$LINE$/d" "file.cfg" --- > remove the pattern itself.

+++++++++

See the below example :

the File.cfg is divided into sections; each section starts with Hello

File.txt contains random sections name; I need a script to read the section's name from File.txt and see if it's available in file.cfg , then remove the section name and all of its contents

File.txt :

Hello World
Hello Mohammad
Hello Scripting

File.cfg :

Hellow xyz
a
b
c
Hello World
v
b
n
Hello stack
q
w
e

The final results should be :

Hellow xyz
a
b
c
Hello stack
q
w
e

Once the section name is found, I need to remove everything until the next "Hello" that comes at the beginning of a line ( new section ).

None of the lines start with Hello except the sections name.

Repeatedly rewriting the same file with `sed -i` in a loop is inefficient and error-prone. Probably try to refactor to create a single `sed` script from your input file. And try http://shellcheck.net/ to have some other common errors in your code pointed out to you. — tripleee, Aug 08 '21 at 18:33
Please read [why-is-using-a-shell-loop-to-process-text-considered-bad-practice](https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice) to understand some of the issues with your script, [is-it-possible-to-escape-regex-metacharacters-reliably-with-sed](https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed) for others and copy/paste it into http://shellcheck.net for more. [edit] your question to include concise, testable sample input and expected output so we can help you. — Ed Morton, Aug 08 '21 at 20:43
And don't use the word "pattern" when stating requirements for matching text as that's highly ambiguous, use regexp-or-string plus full-or-partial instead. See [how-do-i-find-the-text-that-matches-a-pattern](https://stackoverflow.com/questions/65621325/how-do-i-find-the-text-that-matches-a-pattern). — Ed Morton, Aug 08 '21 at 20:45
Thank you for the documents. Could you help me fix my code? I am still new to scripting, and I should make it work today. — Eng-Mohammed Kayed, Aug 08 '21 at 20:48
We can definitely help you fix your code but we need you to tell us what it should do and provide an example first. We currently don't even know if `$LINE` is supposed to be treated as a regexp or a string (e.g. should `a.c` in LINE match `abc` in the .cfg file or not?). Please [edit] your question to, at a minimum, replace "pattern" with regexp or string and add sample input and expected output that demonstrates what you need the script to do and we can copy/paste to test with. — Ed Morton, Aug 08 '21 at 20:52
@triplee took a guess at what you might want, and it looked like a reasonable guess to me, but apparently it's not what you want and you didn't say in what way it failed so there's no point anyone else taking another guess when you could just tell and show us. — Ed Morton, Aug 08 '21 at 20:54
Thank you in advance , I have added more details , Please let me know if its clear now — Eng-Mohammed Kayed, Aug 08 '21 at 21:12
What do you mean when you say that it's "not working"? Does it change the text at all? Does it give an error message? If you remove the `-i`, what happens? If you put `echo` in front of the sed commands in the script, what do you see? — Beta, Aug 09 '21 at 00:41

Ed Morton · Answer 1 · 2021-08-08T23:31:19.630

1

$ awk 'NR==FNR{names[$0]; next} $1=="Hello"{f=($0 in names)} !f' File.txt File.cfg
Hellow xyz
a
b
c
Hello stack
q
w
e

If you want to do "inplace" editing then just like GNU sed, which you're currently using, has -i, GNU awk has -i inplace but note that you're working with 2 input files so you need to write to both of them:

awk -i inplace 'NR==FNR{names[$0]; print; next} $1=="Hello"{f=($0 in names)} !f' File.txt File.cfg

or only activate inplace editing for the 2nd one, see the gawk man page for how to control that. IMHO just using a temp output file is simpler:

tmp=$(mktemp) &&
awk 'NR==FNR{names[$0]; next} $1=="Hello"{f=($0 in names)} !f' File.txt File.cfg > "$tmp" &&
mv -- "$tmp" File.cfg

edited Aug 08 '21 at 23:31

answered Aug 08 '21 at 21:59

Ed Morton

188,023
17
78
185

Thank you for the code, Actually, my file is large so I can't run the command and verify the result without creating a new file or overwriting the original one, I tried to add -i inplace but it cleared File.txt contents, any ideas? – Eng-Mohammed Kayed Aug 08 '21 at 22:36
Regarding `my file is large so...` - it's true of any command run on any size file that you can't `verify the result without creating a new file or overwriting the original one` so idk why you said that. Yes, if you ran the script as-is then `-i inplace` would clear `File.txt` because the script isn't printing anything while it's reading `File.txt` in the `NR==FNR` block. Change `next` to `print; next` if you want to use `-i inplace`. – Ed Morton Aug 08 '21 at 23:25
I added some more information to my answer about how to update File.cfg. – Ed Morton Aug 08 '21 at 23:31

Cole Tierney · Answer 2 · 2021-08-09T13:54:24.763

I like @tripleee's suggestion to create a sed script from the patterns file. It results in a single pass and sed making sed appeals to my sense of humor :)

The first step is to generate the sed script:

sed 's|.*|/^&$/, /^Hello/ {\n\t/^&$/ d\n\t/^Hello/! d\n}|' file.txt
/^Hello World$/, /^Hello/ {
    /^Hello World$/ d
    /^Hello/! d
}
/^Hello Mohammad$/, /^Hello/ {
    /^Hello Mohammad$/ d
    /^Hello/! d
}
/^Hello Scripting$/, /^Hello/ {
    /^Hello Scripting$/ d
    /^Hello/! d
}

In a nut shell, for each address range we want to delete everything except the ending pattern.

I'll generate the above sed using bash process substitution and treat it like a sed program file (or it could be put in a temp file):

#!/bin/bash

sed -f <(
    sed 's|.*|/^&$/, /^Hello/ {\n\t/^&$/ d\n\t/^Hello/! d\n}|' file.txt
) file.cfg

I left out the -i in place edit option for testing.

For non-destructive testing, compare the expected results with the output of the script:

diff expect <(./remove.sh) && echo ok

sed command inside while loop is not working for ubuntu

2 Answers2