2

In the following configuration file 

/etc/fine-tune.conf

We have duplicate lines as 

clean_history_in_os=true

we want to delete all the lines that include clean_history_in_os=true except the first matched line in the file

what I did until now is that

  sed  -i '/clean_history_in_os=true/d' /etc/fine-tune.conf

but the problem is that sed delete all "clean_history_in_os=true" lines

I will happy to get ideas to solve this issue ,

King David
  • 500
  • 1
  • 7
  • 20
  • You want to delete all duplicates or just `clean_history_in_os`? – Joaquin Aug 06 '18 at 17:59
  • only according to the requested line ( in this example I want to delete the duplicate line - clean_history_in_os=true ) , but this is example , it could be different line – King David Aug 06 '18 at 18:01
  • `awk '!a[$0]++' file` – Cyrus Aug 06 '18 at 18:22
  • @Cyrus not good option because - its delete all duplicate lines and what we want is only to delete specific duplicate line , second its delete also the empty lines – King David Aug 06 '18 at 18:25
  • I posted a Perl answer even though the awk one by anybhava is good, because you specifically asked -- but if you wanted Perl or sed then why tag the question with awk as well? – zdim Aug 06 '18 at 18:27
  • yes you right , I tag awk because I was thinking that awk can also update the file as ( sed -i ) , now I see that option isnt in awk and I not have awk gnu , so I will delete this tag – King David Aug 07 '18 at 06:13
  • @zdim , your perl answer is really excellent , I need to check this option on my machine to see if all options can be supported – King David Aug 07 '18 at 06:16

2 Answers2

3

With Perl

perl -i -ne'next if /clean_history_in_os=true/ && ++$ok > 1; print' file

This increments the counter when on that line and if > 1 it skips the line, otherwise prints

The question came up of how to pass the pattern to Perl if we have it as a shell variable. Below I assume that the shell variable $VAR contains the string clean_history...

In all this a shell variable's value is directly used as a pattern in a regex. If it's the literal string from the question then the code below goes as given. However, if there may be special characters they should be escaped; so you may want to precede the pattern with \Q when used in regex. As a general note, one should take care to not use input from the shell to run code (say under /e).

  • Pass it as an argument, which is then available in @ARGV

      perl -i -ne'
      BEGIN { $qr=shift; }; 
      next if /$qr/ && ++$ok > 1; print
  ' "$VAR" file

    where the BEGIN block runs in the BEGIN phase, before runtime (so not for the following iterations). In it shift removes the first element from @ARGV, which in the above invocation is the value in $VAR, first interpolated by shell. Then the filename file remains in @ARGV, so available for processing under -n (file is opened and its lines iterated over)

  • Use the -s switch, which enables command-line switches for the program

      perl -i -s -ne'next if /$qr/ && ++$ok > 1; print' -- -qr="$VAR" file

    The -- (after the one-line program under '') marks the start of arguments for the program; then -qr introduces a variable $qr into the program, with a value assigned to it as above (with just -qr the variable $qr gets value 1, so is a flag).

    Any such options must come before possible filenames, and they are removed from @ARGV so the program can then normally process the submitted files.

  • Export the bash variable, making it an environment variable which can then be accessed in the Perl program via %ENV hash

      export VAR="clean_history..."
  perl -i -ne'next if /$ENV{VAR}/ && ++$ok > 1; print' file

or, if $VAR is used only in this one-liner, can use the shorter (what must be on one line)

    VAR="clean_history..."  perl -i -ne'...' file

I would rather recommend either of the first two options, over this one.

These are ways to pass input to a Perl program entered entirely on the command-line (one-liner), without STDIN or files. With a script better use a library, in the first place Getopt::Long.

A refinement of the question given in a comment specifies that if the phrase clean_... starts with a # then that line should be skipped altogether. It's simplest to separately test for that

next if /#$qr/; next if /$qr/ && ++$ok > 1; print

or, relying on short-circuiting

next if /#$qr/ || (/$qr/ && ++$ok > 1); print

The first version is less error prone and probably clearer.

zdim
  • 64,580
  • 5
  • 52
  • 81
2

You can use this awk to delete all matching lines except the first one:

awk '!(/clean_history_in_os=true/ && n++)' file

To save file in place you can use this gnu awk command:

awk -i inplace '!(/clean_history_in_os=true/ && n++)' file

otherwise use temporary file as:

awk '!(/clean_history_in_os=true/ && n++)' file > $$.tmp && mv $$.tmp file

Here is one sed solution to do the same:

sed -i -n '0,/clean_history_in_os=true/p;/clean_history_in_os=true/!p' file
anubhava
  • 761,203
  • 64
  • 569
  • 643
  • I prefer to work with sed or perl , because I can edit the file with ( -i ) option , with awk I not able to edit the file as sed -i , or perl -i , – King David Aug 06 '18 at 18:08
  • 1
    in `awk` just output to a temp file and `mv` the temp file: `awk '$0=="clean_history_in_os=true"{c[$0]++} c[$0]<2; ' conf > tmp && mv tmp conf` – justaguy Aug 06 '18 at 18:09
  • yes I know thx but I prefer to avoid that , because some time I need to delete around 10 duplicate lines so this approach is not so clean in case we have a lot of duplicate lines – King David Aug 06 '18 at 18:10
  • 1
    If using `gnu awk`, you can use `-i inplace` – anubhava Aug 06 '18 at 18:10
  • I guess in my case we not have this option , ( awk -i inplace ) , so I will happy to get solution with sed / perl one liner – King David Aug 06 '18 at 18:11
  • 1
    Using the range command in sed i.e. `1,/../` will not work if the regexp is on the first line. GNU sed has a specific `0,/../` range will take care of this problem. An alternative solution is `sed '/clean_history_in_os=true/{x;/./{x;d};g;x}' file` which keeps a counter in the hold space. – potong Aug 06 '18 at 19:27
  • @Ed I think your solution is the best , but still this isnt update the file itself – King David Aug 06 '18 at 23:44
  • 1
    You've already been told in the comments how to update the original file, just like with any other command: `awk 'script' file > tmp && mv tmp file`. Having said that `sed -i` is GNU sed only so it's hard to believe you have GNU sed and perl but you don't have or can't get GNU awk (for `awk -i inplace`). – Ed Morton Aug 06 '18 at 23:56
  • 1
    What's the error with `sed` command? I have tested my updated `sed` answer and it worked fine. @EdMorton: `awk '!(/clean_history_in_os=true/ && n++)'` is really smart – anubhava Aug 07 '18 at 10:30