Updating a specific field with sed

Question

I'm trying to update a specific field on a specific line with the sed command in Bourne Shell.

Lets say I have a file TopScorer.txt

Player:Games:Goals:Assists
Salah:9:9:3
Kane:10:8:4

And I need to update the 3rd Column (Goals) of a player, I tried this command and it works unless Games and Goals have the same value then it updates the first one

player="Salah"
NewGoals="10"
OldGoals=$(awk -F':' '$1=="'$player'"' TopScorer.txt | cut -d':' -f3)
sed -i '/^'$player'/ s/'$OldGoals'/'$NewGoals'/' TopScorer.txt

Output> Salah:10:9:3 instead of Salah:9:10:3

Is there any solution? Should I use delimiters and $3==... to specify that field? I also tried the option /2 for second occurrence but it's not very convenient in my case.

Your quoting is pretty broken. The variables should be in double quotes and the `sed` script might as well be too. Code with quoting errors will generally seem to work fine as long as you work with simple tokens, then blow up in interesting ways when you start working on real-world data. See further https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable — tripleee, Jan 10 '19 at 04:37
Maybe have a look at `sqlite` for this type of thing - it's a zero-configuration database that needs no server running - just a single binary executable. — Mark Setchell, Jan 10 '19 at 07:23

score 4 · Answer 1 · answered Jan 10 '19 at 04:03

4

You can just do this with awk alone and not with sed. Also note that awk has an internal syntax to import variables from the shell. So your code just becomes

awk -F: -v pl="$player" -v goals="$NewGoals" 
    'BEGIN { OFS = FS } $1 == pl { $3= goals }1' TopScorer.txt

The -F: sets the input de-limiter as : and the part involving -v imports your shell variables to the context of awk. The BEGIN { OFS = FS } sets the output field separator to the same as input. Then we do the match using the imported variables and update $3 to the required value.

To make the modifications in-place, use a temporary file

awk -F: -v pl="$player" -v goals="$NewGoals" 
   'BEGIN { OFS = FS } $1 == pl { $3= goals }1' TopScorer.txt > tmpfile && mv tmpfile TopScorer.txt

answered Jan 10 '19 at 04:03

Inian

80,270
14
142
161

1

I got something similar `awk -F':' 'BEGIN {OFS=FS} /^'$player'/ {$3='$NewGoals'}1' TopScorer.txt` before redirection and move. – David C. Rankin Jan 10 '19 at 04:19
1

@DavidC.Rankin that introduces security vulnerabilities and leaves the shell variables open for word splitting, globbing, etc. since `$player` (and `$NewGoals`) is unquoted in the shell (should be `/^'"$player"'/` etc.) and also creates cryptic errors based on the contents of the shell variables since you're making their expansion part of the actual awk script text before awk interprets it and that is not fixable. Don't do it. Use `-v` or similar to set awk variables from the shell variable values and then use those awk variables in the script as Inian is doing. – Ed Morton Jan 10 '19 at 12:15
1

@EdMorton thank you. That is the finer grained detail that left me questioning how best to handle the variable expansion within the `awk` command. I see the word-splitting issue, but is the concern about the content of the shell variables, the potential unknown content? Because, it seems if you know the content that will be expanded, the use of `-v` or shell should be the same? But if the `-v` is to protect against spurious content in the expansions, then that makes sense as a general approach. – David C. Rankin Jan 10 '19 at 18:11
@DavidC.Rankin yes, the concern is about potential unknown content or content that's known but produces unexpected behavior. It's the same argument as for why you should always quote shell variables unless you have a specific purpose in mind that requires you to not do so and fully understand all the implications and consequences. – Ed Morton Jan 11 '19 at 18:19
2

Try `foo=$'hello\nworld'; awk -v foo="$foo" 'BEGIN{print foo}'` and you'll get the expected output. Now try changing the awk command to `awk 'BEGIN{print "'$foo'"}'` or anything else that's letting the shell variable expand to become part of the awk script and watch it fail in various ways. There are just SO many ways it can fail (often cryptically and/or silently) given SO many values of the shell variable. – Ed Morton Jan 11 '19 at 18:29
1

That is a very good example of the problem. Learning has occurred. Thank you. – David C. Rankin Jan 11 '19 at 22:10

score 2 · Answer 2 · answered Jan 10 '19 at 09:14

You can try it with Perl

$ player="Salah"
$ NewGoals="10"
$ perl -F: -lane "\$F[2]=$NewGoals  if ( \$F[0] eq $player ) ; print join(':',@F) " TopScorer.txt
Player:Games:Goals:Assists
Salah:9:10:3
Kane:10:8:4
$

or export them and call Perl one-liner within single quotes

$ export NewGoals="10"
$ export player="Salah"
$  perl -F: -lane '$F[2]=$ENV{NewGoals} if $F[0] eq $ENV{player} ; print join(":",@F) ' TopScorer.txt
Player:Games:Goals:Assists
Salah:9:10:3
Kane:10:8:4
$

Note that Perl has -i switch and you can do the replacement in-place, so

$ perl -i.bak -F: -lane '$F[2]=$ENV{NewGoals} if $F[0] eq $ENV{player} ; print join(":",@F) ' TopScorer.txt
$ cat TopScorer.txt
Player:Games:Goals:Assists
Salah:9:10:3
Kane:10:8:4
$

score 2 · Accepted Answer · answered Jan 10 '19 at 10:18

This might work for you (GNU sed):

(player=Salah;newGoals=10;sed -i "/^$name/s/[^:]*/$newGoals/3" /tmp/file)

Use a sub shell so as not to pollute the current shell (...). Use sed and pattern matching to match the first field of each record to the variable player and replace the third field of the matching record with the contents of newGoals.

P.S. If the variables are needed in further processes the sub shell is not necessary i.e. remove the ( and )

score 1 · Answer 4 · answered Jan 10 '19 at 04:03

1

This will work .

With the first part of sed , i try to match a full line that math the player, and i keep all fields i want to keep by using \( .

The second part , i rebuild the line with some constants and the value of \1 and the value of \2

player="Salah"
NewGoals="10"
sed "s/^$player:\([^:]*\):[^:]*:\([^:]*\)\$/$player:\1:$NewGoals:\2/"

answered Jan 10 '19 at 04:03

EchoMike444

1,513
1
9
8

You still end up with a race condition between reading the file to obtain the old goals and updating it with a calculated new value. The Awk script in the other answer solves this more elegantly. – tripleee Jan 10 '19 at 04:39

RavinderSingh13 · Answer 5 · 2019-01-10T05:02:15.860

Could you please try following once. Advantage of this approach is that I am not hard coding field for Goals. This program will look for header's field wherever Goal is present(eg--> 4th or 5th any field), it will change for that specific column only.

1st Solution: When you need to make changes to all occurrences of player name then use following.

NewGoals=10
awk -v newgoals="$NewGoals" 'BEGIN{FS=OFS=":"} FNR==1{for(i=1;i<=NF;i++){if($i=="Goals"){field=i}}} FNR>1{if($1=="Salah"){$field=newgoals}} 1' Input_file

2nd Solution: In case you want to change a specific player's goals value to specific row only then try following.

NewGoals=10
awk -v newgoals="$NewGoals" 'BEGIN{FS=OFS=":"} FNR==1{for(i=1;i<=NF;i++){if($i=="Goals"){field=i}}} FNR>1{if($1=="Salah" && FNR==2){$field=newgoals}} 1' Input_file

Above will make changes only for row 2, you coud change it by changing FNR==2 in 2nd condition where FNR refers row number inawk. In case you want to save output into Input_file itself then you could append > temp_file && mv temp_file Input_file to above codes.

Updating a specific field with sed

5 Answers5