Bash remove duplicate words from file (regex: sed,awk)

Question

I have read other questions that were very similar to mine. However, when I tried the suggested codes, it did not work

So I have a file with the following content

COMP 232:MATH. FOR COMPUTER SCIENCE:3 credits summer:Lect AA:-T-J--- (10:45-13:15):SGW H-521:KRATOCHVIL, VACLAV Tut AE:-T-J--- (13:45-15:25):SGW H-521 fall:Lect Q:-T-J--- (13:15-14:30):SGW H-820:GRAHNE, GOSTA Tut QA:-T----- (16:15-17:55):SGW H-607 Tut QB:---J--- (14:45-16:25):SGW H-619 fall:Lect R:-T-J--- (13:15-14:30):SGW H-937:NARAYANAN, LATA Tut RA:-T----- (16:15-17:55):SGW H-1070 Tut RB:-T----- (16:15-17:55):SGW GN-M100 fall:Lect DD:--W---- (17:45-20:15):SGW FG-C070:DOEDEL, EUSEBIUS Tut DA:--W---- (20:30-22:10):SGW H-429 Tut DB:--W---- (20:30-22:10):SGW H-619 winter:Lect S:-T-J--- (13:15-14:30):SGW H-561:BERGLER, SABINE Tut SA:-T----- (14:45-16:25):SGW H-920 winter:Lect NN:---J--- (17:45-20:15):SGW H-520:GRAHNE, GOSTA Tut NA:---J--- (20:30-22:10):SGW H-520 Tut NB:-T----- (20:30-22:10):SGW H-607

and I would like my output to be this

COMP 232:MATH. FOR COMPUTER SCIENCE:3 credits summer:Lect AA:-T-J--- (10:45-13:15):SGW H-521:KRATOCHVIL, VACLAV Tut AE:-T-J--- (13:45-15:25):SGW H-521 fall:Lect Q:-T-J--- (13:15-14:30):SGW H-820:GRAHNE, GOSTA Tut QA:-T----- (16:15-17:55):SGW H-607 Tut QB:---J--- (14:45-16:25):SGW H-619 Lect R:-T-J--- (13:15-14:30):SGW H-937:NARAYANAN, LATA Tut RA:-T----- (16:15-17:55):SGW H-1070 Tut RB:-T----- (16:15-17:55):SGW GN-M100 Lect DD:--W---- (17:45-20:15):SGW FG-C070:DOEDEL, EUSEBIUS Tut DA:--W---- (20:30-22:10):SGW H-429 Tut DB:--W---- (20:30-22:10):SGW H-619 winter:Lect S:-T-J--- (13:15-14:30):SGW H-561:BERGLER, SABINE Tut SA:-T----- (14:45-16:25):SGW H-920 Lect NN:---J--- (17:45-20:15):SGW H-520:GRAHNE, GOSTA Tut NA:---J--- (20:30-22:10):SGW H-520 Tut NB:-T----- (20:30-22:10):SGW H-607

I have tried the following codes, but none produced my desired output

sed -e 's/\b\([a-z]\+\)[ ,\n]\1/\1/g' temp.txt

sort -u temp.txt

score 1 · Answer 1 · edited May 23 '17 at 11:49

1

GNU awk 4.1

#!/usr/bin/awk -f
@include "join"
{
  split($0, foo, ":")
  if (seen[foo[1]]++)
    $0 = "   " join(foo, 2, length(foo), ":")
}
1

awk save modifications in place

edited May 23 '17 at 11:49

Community

1
1

answered Nov 10 '14 at 01:14

Zombo

1
62
391
407

I'm putting this in my `test.sh` bash code but the final closing bracket `}` is highlighted **red**. And also instead of `foo` I put my filename? – Matt Nov 10 '14 at 01:37

score 1 · Accepted Answer · answered Nov 10 '14 at 09:58

Looks lengthy a bit, but works fine:

prev="dummy"
while read line
do
   tag=$(echo $line | grep -Eo "^[a-zA-Z]+:")
   if [ "$tag" == "$prev" ]
   then
     echo "  ${line#$tag}"
   else
     if [ "$tag" != "" ] 
     then
        prev="$tag"
     else
        echo -n "  "
   fi
   echo "$line"
fi
done < /path/to/file

Bash remove duplicate words from file (regex: sed,awk)

2 Answers2