how to replace a block of text with new block of text?

Question

As the question title specifies , i have to replace a block to text in a file with a new block of text

I have searched all over for this thing but every solution i ever found was just too specific to the question. Isn't it possible to create a function which is flexible/reusable ?

To be very specific i need something which has options like

1) File ( where changes are to be done )
2) Exiting block of text
3) New block of text
( 2nd & 3 option could be either as manually pasted text or cat $somefile)

whereby i could change these 3 and use the script for all cases of text block replacement , i am sure it will help many other people too

As for an example , currently i need to replace the below block of text with one at bottom and say the file is $HOME/block.txt . Although i need the solution which is easily reusable/flexible as mentioned above

- name: Set default_volumes variable
  set_fact:
    default_volumes:
      - "/opt/lidarr:/config"
      - "/opt/scripts:/scripts"
      - "/mnt:/mnt"
      - "/mnt/unionfs/Media/Music:/music"

- name: Set default_volumes variable
  set_fact:
    default_volumes:
      - "/opt/lidarr:/config"
      - "/opt/scripts:/scripts"
      - "/mnt:/mnt"
      - "/mnt/unionfs/Media/Music:/music"
      - "/mnt/unionfs/downloads/lidarr:/downloads-amd"

PS / while replacement i need the spacing and indentation to be preserved.

A naive solution would be to a) read the file into a variable b) do a replacement with `s{\$original_text}{$replacement_text}s` c) save the changed variable into a file. It is trivial to make a very short function which does exactly this given a file name, original text and replacement text. — Steffen Ullrich, Jun 13 '21 at 08:36
@SteffenUllrich did you mean `{\Q$original_text}` instead of `{\$original_text}` — Sundeep, Jun 13 '21 at 08:40
This looks like YAML. So the best approach would be to a) read the YAML into a Perl data structure (using [YAML.pm](https://metacpan.org/pod/YAML)), b) change the Perl data structure in whatever way you need, c) convert the data structure back to Perl and write in to a file. — Dave Cross, Jun 13 '21 at 13:00
This is [YAML](https://en.wikipedia.org/wiki/YAML) and is best processed using libraries for that data format — zdim, Jun 13 '21 at 19:39

ikegami · Answer 1 · 2021-06-13T21:02:15.667

Your data is serialized using YAML. You should treat it as such.

Using yq

yq eval '
   .[0].set_fact.default_volumes +=
      [ "/mnt/unionfs/downloads/lidarr:/downloads-amd" ]
'

yq doesn't natively support in-place editing, but you can use sponge to achieve the same thing.

yq eval '
   .[0].set_fact.default_volumes +=
      [ "/mnt/unionfs/downloads/lidarr:/downloads-amd" ]
' a.yaml | sponge a.yaml

Using Perl

perl -MYAML -0777ne'
   my $d = Load($_);
   push @{ $d->[0]{set_fact}{default_volumes} },
      "/mnt/unionfs/downloads/lidarr:/downloads-amd";
   print Dump($d);
'

As per specifying file to process to Perl one-liner, editing in place would look like this:

perl -i -MYAML -0777ne'
   my $d = Load($_);
   push @{ $d->[0]{set_fact}{default_volumes} },
      "/mnt/unionfs/downloads/lidarr:/downloads-amd";
   print Dump($d);
' file.yaml

Ed Morton · Accepted Answer · 2021-06-13T13:33:36.753

Using GNU awk for multi-char RS and ARGIND, this will work for any chars in your old or new text including regexp metachars, delimiters, quotes, and backreferences as it's just doing literal string search and replace:

awk -v RS='^$' -v ORS= '
    ARGIND==1 { old=$0; next }
    ARGIND==2 { new=$0; next }
    s=index($0,old) {
        $0 = substr($0,1,s-1) new substr($0,s+length(old))
    }
1' old new file

or you can do the same using any awk in any shell on every Unix box with:

awk -v ORS= '
    { rec = (FNR>1 ? rec RS : "") $0 }
    FILENAME==ARGV[1] { old=rec; next }
    FILENAME==ARGV[2] { new=rec; next }
    END {
        $0 = rec
        if ( s=index($0,old) ) {
            $0 = substr($0,1,s-1) new substr($0,s+length(old))
        }
        print
    }
' old new file

For example:

$ head old new file
==> old <==
- name: Set default_volumes variable
  set_fact:
    default_volumes:
      - "/opt/lidarr:/config"
      - "/opt/scripts:/scripts"
      - "/mnt:/mnt"
      - "/mnt/unionfs/Media/Music:/music"

==> new <==
- name: Set default_volumes variable
  set_fact:
    default_volumes:
      - "/opt/lidarr:/config"
      - "/opt/scripts:/scripts"
      - "/mnt:/mnt"
      - "/mnt/unionfs/Media/Music:/music"
      - "/mnt/unionfs/downloads/lidarr:/downloads-amd"

==> file <==
foo
- name: Set default_volumes variable
  set_fact:
    default_volumes:
      - "/opt/lidarr:/config"
      - "/opt/scripts:/scripts"
      - "/mnt:/mnt"
      - "/mnt/unionfs/Media/Music:/music"
bar

$ awk -v RS='^$' -v ORS= 'ARGIND==1{old=$0; next} ARGIND==2{new=$0; next} s=index($0,old){ $0=substr($0,1,s-1) new substr($0,s+length(old))} 1' old new file
foo
- name: Set default_volumes variable
  set_fact:
    default_volumes:
      - "/opt/lidarr:/config"
      - "/opt/scripts:/scripts"
      - "/mnt:/mnt"
      - "/mnt/unionfs/Media/Music:/music"
      - "/mnt/unionfs/downloads/lidarr:/downloads-amd"
bar

This is really working great and exactly what i needed .Just one final edit request - Instead of printing the final result at end , can you make it edit the file in place ? Thanks — Sachin, Jun 13 '21 at 15:19
No Unix text manipulation tool (awk, sed, perl, ruby, etc.) **actually** edits a file inplace. Some have a `-i` or similar option to tell them to create a temp file in the background but they're still using a temp file. You can do exactly the same manually for any command that does or doesn't have a `-i` option by using `tmp=$(mktemp) && cmd 'script' file > "$tmp" && mv -- "$tmp" file`. GNU awk has `-i inplace` just like GNU sed has `-i` but then you'd need to add `print`s to recreate the `old` and `new` file or otherwise add complexity so I'd recommend you just create the temp file manually. — Ed Morton, Jun 13 '21 at 15:25
Re "*You can do exactly the same manually for any command that does or doesn't have a `-i` option*", Or use `some_cmd a.yaml | sponge a.yaml` — ikegami, Jun 13 '21 at 21:03

mattb · Answer 3 · 2021-06-13T09:36:23.263

For a task like this, you could just use existing commands rather than reinventing the wheel:

sed '/some text to change/,/with indentation/d; /a bit more/r new_file' your_file

I used two example files:

# original file
some original text to keep
a bit more
some text to remove
    - with indentation
rest of original text
is kept

and:

# replacement text
SOME TEXT TO ADD
    - WITH DIFFERENT INDENTATION
        - ANOTHER LEVEL

Then the command works by first deleting lines between and including two lines matching patterns:

sed '/some text to change/,/with indentation/d;

Then reading the replacement text from some other file, using a pattern matching just where the old text used to start:

/a bit more/r new_file' your_file

To yield the result:

some original text to keep
a bit more
SOME TEXT TO ADD
    - WITH DIFFERENT INDENTATION
        - ANOTHER LEVEL
rest of original text
is kept

Edit

The above is better than my original way:

sed '/a bit more/q' your_file > composite; cat new_file >> composite; sed -n '/rest of original text/,/$/p' your_file >> composite

sed isn't a good choice for this as you'd have to escape every regexp metachar, `/`, and `'` in the text you're trying to replace (see [is-it-possible-to-escape-regex-metacharacters-reliably-with-sed](https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed)) and a range expressions is a bad approach as you can't assume that `/foo/,/bar/` uniquely identifies the multi-line block of text between foo and bar that's to be replaced. — Ed Morton, Jun 13 '21 at 13:25

how to replace a block of text with new block of text?

3 Answers3

Edit