Replace multiple lines in text file

Question

I have text files containing the text below (amongst other text)

DIFF_COEFF= 1.000e+07,1.000e+07,1.000e+07,1.000e+07,
1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,
1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,
1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,
1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,
1.000e+07,1.000e+07,1.000e+07,1.000e+07,1.000e+07,4.000e+05,

and I need to replace it with the following text:

DIFF_COEFF= 2.000e+07,2.000e+07,2.000e+07,2.000e+07,
2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,
2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,
2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,
2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,
2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,8.000e+05,

Each line above corresponds to a new line in the text file.

After some googling, I thought making use of Perl in the following might work, but it did not. I got the error message

Illegal division by zero at -e line 1, <> chunk 1

s_orig='DIFF_COEFF=*4.000e+05,'

s_new='DIFF_COEFF= 2.000e+07,2.000e+07,2.000e+07,2.000e+07,\n2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,\n2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,\n2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,\n2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,\n2.000e+07,2.000e+07,2.000e+07,2.000e+07,2.000e+07,8.000e+05,'

perl -0 -i -pe "s:\Q${s_orig}\E:${s_new}:/igs" file.txt

Does anyone here know the right way to do this?

Edit - some more details: the text after this block is "DIFF_COEFF_Q=" followed by the same set of numbers, so I need to search for and replace the specific lines shown. The text files are not very large in size.

(1) How is the "_text below_" (the `DIFF_COEFF` and the following 5 lines) distinguished from the "_other text_"? For example, what follows the last line with numbers? (2) Do you need to multiply each number by 2, as in the example, or to literally replace by text? — zdim, Dec 20 '17 at 17:32
Drop the `/` at the end of your replacement, i.e. `s:…:…:igs`, not `s:…:…:/igs`. — PerlDuck, Dec 20 '17 at 17:57
So you don't want to do calculations, you just want to literally replace? — simbabque, Dec 20 '17 at 18:38
Thanks for the comments. To respond to them in turn: - the text below the DIFF_COEFF lines is unfortunately very similar to the lines shown (it differs only in that it starts with DIFF_COEFF_Q) - I need to search and replace the specific lines shown. - I will usually be multiplying the numbers, so a way to get the result by doing a calculation would be helpful. But the more general method of replacing the text would be valuable for cases when I don't just want to multiply the numbers. - yes the list ends in a comma. ... — PeterW, Dec 20 '17 at 22:26
... - if I drop the slash, it gets rid of the error message, so thanks for pointing that out. But then the perl command does not change the file. - I will have to do this many times, so a scripted solution would be preferred to a text editor. — PeterW, Dec 20 '17 at 22:27
So you need to replace text from `DIFF_COEFF` to `DIFF_COEFF_Q`? That's good, it is unambigous. Where does the replacement text for each such passage (that is to be replaced) come from? I presume that there are multiple ones, and that it's not always the same text? — zdim, Dec 20 '17 at 23:15
Also: I suggest to edit your question and add those clarifications. — zdim, Dec 21 '17 at 00:00
*"the text below the `DIFF_COEFF` lines is unfortunately very similar to the lines shown"* You seem reluctant to reveal anything but the most trivial of information, and **zdim** is hinting about what may be useful. The data that you want to replace is only part of a file, and from what you've written that file may contain identical data in many formats. It is most unlikely that a simple (very long) string replacement will do what you want, and in general you must *parse* the data to extract its semantics, modify that data, and reencode the result. Anything less would be a stab in the dark. — Borodin, Dec 21 '17 at 17:24
Borodin - Why reluctant? I have answered everyone's questions as I understood them. I presume you don't want me to paste the whole text file here, so I said what I thought would be important. "The text below the DIFF_COEFF lines is unfortunately very similar to the lines shown (it differs only in that it starts with DIFF_COEFF_Q)" is unambiguous about what the text below is as far as I can see. This was sufficient for zdim to provide a solution. A long string replacement worked perfectly. The pieces of text I mentioned are not repeated, if that's what you're getting at (it's not clear). — PeterW, Dec 21 '17 at 23:19

zdim · Accepted Answer · 2017-12-24T01:55:30.957

Copy the file over to a new one, except that within the range of text between these markers drop the replacement text instead. Then move that file to replace the original, as it may be needed judging by the attempted perl -0 -i in the question.

Note that when changing a file we have to build new content and then replace the file. There are a few ways to do this and modules that make it easier, shown further below.

The code below uses the range operator and the fact that it returns the counter for lines within the range, 1 for the first and the number ending with E0 for the last. So we don't copy lines inside that region while we write the replacement text (and the post-region-end marker) on the last line.

I consider the region of interest to end right before DIFF_COEFF_Q= line, per the question edit.

use warnings;
use strict;
use feature 'say';
use File::Copy 'move';

my $replacement = "replacement text";

my $file     = 'input.txt';
my $out_file = 'new_' . $file;

open my $fh_out, '>', $out_file or die "Can't open $out_file: $!";
open my $fh,     '<', $file     or die "Can't open $file: $!";

while (<$fh>) 
{
    if (my $range_cnt = /^\s*DIFF_COEFF\s*=/ .. /^\s*DIFF_COEFF_Q\s*=/) #/
    {
        if ($range_cnt =~ /E0$/)
        {
            print $fh_out $replacement;  # may need a newline
            print $fh_out $_;         
        }
    }   
    else { 
        print $fh_out $_; 
    }
}
close $fh     or die "Can't close $file: $!";      # don't overwrite original
close $fh_out or die "Can't close $out_file: $!";  # if there are problems

#move $out_file, $file or die "Can't move $file to $out_file: $!";

Uncomment the move line once this has been tested well enough on your actual files, if you want to replace the original. You may or may not need a newline after $replacement, depending on it.

An alternative is to use flags for entering/leaving that range. But this won't be cleaner since there are two distinct actions, to stop copying when entering the range and write replacement when leaving. Thus multiple flags need be set and checked, what may end up messier.

If the files can't ever be huge it is simpler to read and process the file in memory. Then open the same file for writing and dump the new content

my $text = do {  # slurp file into a scalar
    local $/; 
    open my $fh, '<', $file or die "Can't open $file: $!"; 
    <$fh> 
};

$text =~ s/^\s*DIFF_COEFF\s*=.*?(\n\s*DIFF_COEFF_Q)/$replacement$1/ms;

# Change $out_file to $file to overwrite
open my $fh_out, '>', $out_file or die "Can't open $out_file: $!";
print $fh_out $text;

Here /m modifier is for multiline mode in which we can use ^ for the beginning of a line (not the whole string), what is helpful here. The /s makes . match a newline, too. Also note that we can slurp a file with Path::Tiny as simply as: my $text = path($file)->slurp;

Another option is to use Path::Tiny, which in newer versions has edit and edit_lines methods

use Path::Tiny;
                      # NOTE: edits $file in place (changes it)
path($file)->edit( 
    sub { s/DIFF_COEFF=.*?(\n\s*DIFF_COEFF_Q)/$replacement$1/s } 
);

For more on this see, for example, this post and this post and this post.

The first and last way change the inode number of the file. See this post if that is a problem.

@PeterW Added code for the other ways that were mentioned. – zdim Dec 22 '17 at 09:12 — zdim, Dec 22 '17 at 09:12

score 1 · Answer 2 · edited Dec 21 '17 at 16:43

It's an interesting error that you've made and I can see what has led you to make it. But I don't think I've ever seen anyone else make the same mistake :-)

Your substitution statement is this:

s:\Q${s_orig}\E:${s_new}:/igs

So you've decided to use : as the delimiter of the substitution operator. But you want to use the options i, g and s and everywhere you've seen people talk about options on a substitution operator, they talk about using / to introduce the options. So you've added /igs to your substitution operator.

But what you've missed (and I completely understand why) is that the / that comes before the options is actually the closing delimiter of the standard, s/.../.../, version of the substitution operator. If you change the delimiter (as you have done) then your altered closing delimiter is all you need.

In your case, Perl doesn't expect the / as it has already seen the closing delimiter. It, therefore, decides that the / is a division operator and tries to divide the result of your substitution by igs. It interprets igs as zero and you get your error.

The fix is to remove that / so:

s:\Q${s_orig}\E:${s_new}:/igs

becomes:

s:\Q${s_orig}\E:${s_new}:igs

Thanks Dave. You're right about my silly mistake. However, if I remove the final '/', the result is that I now don't get the error message, but no change is made to the file. — PeterW, Dec 21 '17 at 09:38
@PeterW: I think we've answered your question. I suggest you accept an answer here and post a new question with more details about exactly what you're trying to do. — Dave Cross, Dec 21 '17 at 09:53

Replace multiple lines in text file

2 Answers2