Removing current line in bash during read

Question

I'm using bash to read a file and after doing opeation on particular line ,i need to delete that line from input file.

Can you please suggest some way to do so using sed or any other way ?

i've tried using sed command like this :-

#!/bin/sh

file=/Volumes/workplace/GeneratedRules.ion

while read line;do
printf "%s\n" "$line"
sed  '1d' $file
done <$file

my aim in this program is to read one line and then deleting it.

Input :-

AllIsWell
LetsHopeForBest
YouCanMakeIt

but the output , i got is more weird than i thought.

output :-

AllIsWell
LetsHopeForBest
YouCanMakeIt
LetsHopeForBest
LetsHopeForBest
YouCanMakeIt
YouCanMakeIt
LetsHopeForBest
YouCanMakeIt

but i need to output as :

AllIsWell 
LetsHopeForBest
YouCanMakeIt

as i want to delete line after reading it.

NOTE :- i have simplified my problem here . The actual usecase is :-

I need to perform some bunch of operation on line except reading that and the input file is too long and my operation got fails in some way in between .So i want those lines which i have read to be deleted so that if i start the process again , it will not start from the beginning but at the point where it got stuck.

please help.

Could you please show us what you've tried, also the input and the expected output so that people can direct you well! — User123, Oct 23 '20 at 03:59
The way you want to process `$file`, you want to end up with an empty `$file` after the loop completes. Why not remove or truncate the file after the loop is complete? — Jeff Holt, Oct 23 '20 at 04:17
my aim is to remove current line after reading , but not removing the whole file at once. — Shivanshu, Oct 23 '20 at 04:18
You haven't explained why you need to remove the line after reading. It is unreasonable to expect to remove a line from a file (i.e., a stream) and then read the next line. Without knowing more, all I can say is you need at least two files. If you need to fulfill ACID properties for some "system", then you need a database. — Jeff Holt, Oct 23 '20 at 04:21
If input and output should be the same, than `cat "$file"` is the solution. — ceving, Oct 23 '20 at 08:23
Note: The way you are reading the lines from a file in Bash is not correct. Without `\r` as an argument to `read` all `\backslashes` will be interpreted -- sometime with confounding results. You also should include `IFS= ` if you want to preserve leading spaces of the lines. [MORE HERE](http://mywiki.wooledge.org/BashFAQ/001) — dawg, Oct 27 '20 at 12:59
I would suggest starting over with your assumption that lines must be deleted from the input file. That assumption is making this very difficult to do without bugs. Perhaps keep a second file that tracks the lines you have read in one process and then delete those lines in a separate process. That make something super hard become trivial... — dawg, Oct 27 '20 at 13:02

Jeff Holt · Accepted Answer · 2020-10-27T12:42:26.490

You effectively said you want your process to be restartable. Depending upon how you define the successful completion of an iteration of your while loop, you should store a line number in a separate file, x, that indicates how many lines you have successfully processed. If the file doesn't exist, then assume you would start reading at line 1.

Otherwise, you would get the content of x into variable n and then you would start reading $file at line $n + 1.

How you start reading at a particular line depends on constraints we don't know yet.

One way you could do it is to use sed to put lines $n + 1 into a temporary file, remove $file and then move the temporary file to $file before your while loop begins.

There are other solutions but each one might not elegantly satisfy your constraints.

But you'll want to carefully consider what happens if some other process is modifying the content of $file and when it is changing the content. If it only changes the content before or after your bash script is running, then you're probably ok to continue down this path. Otherwise, you have a synchronization problem to solve.

dawg · Answer 2 · 2020-10-27T15:05:50.453

As stated in comments, there are many issues with altering the file you are currently reading from. Don't do it.

You could just keep track of which lines you have dealt with in the first loop (with a counter) then use sed to delete those lines after your first loop has processed them.

A simple example:

cd /tmp

echo 'Line 1
Line 2
Line 3
Line 4
Line 5' >file

echo "file before:"
cat file

cnt=1
while IFS= read -r line || [[ -n $line ]]; do 
    printf "'%s' processed\n" "$line"; 
    if [ "$cnt" -ge 3 ]; then
        break
    fi  
    let "cnt+=1"
done <file

sed -i '' "1,${cnt}d" file 

echo "file after:"
cat file

Prints:

file before:
Line 1
Line 2
Line 3
Line 4
Line 5
'Line 1' processed
'Line 2' processed
'Line 3' processed
file after:
Line 4
Line 5

Another method is to use something like awk, ruby or perl to 'slurp' the file and then feed that slurpped file line-by-line to your Bash while loop. The file can then be modified in your loop since the other process has already fully read and closed the file:

# Note: This is SLOWER and USES MORE MEMORY...
echo "file before:"
cat file

while IFS= read -r line; do 
    printf "'%s' processed\n" "$line"; 
    sed -i '' "1d" file
done < <(awk -v cnt=3 'NR>cnt{next} 
            {arr[NR]=$0}
        END { for(i=1;i<=cnt;i++) print arr[i] }' file)
echo "file after:"
cat file

# same output

Note:

Please make sure you polish up on how to use bash to read a stream line-by-line for less surprises.

Read HERE and HERE for more.

score -1 · Answer 3 · answered Oct 23 '20 at 12:27

-1

It needs a option -i, and if you need backup the file, just assign a suffix to the option. see the man sed

#!/bin/sh

file=/Volumes/workplace/GeneratedRules.ion

while read line;do
printf "%s\n" "$line"
sed -i '1d' $file
done <$file

answered Oct 23 '20 at 12:27

Blackdoor

922
1
6
12

Removing current line in bash during read

3 Answers3