16

This question already addresses how to remove duplicate lines, but enforces that the list is sorted first.

I would like to perform the remove contiguous duplicate lines step (i.e. uniq) without first sorting them.

Example before:

Foo
Foo
Bar
Bar

Example after:

Foo
Bar
Community
  • 1
  • 1
davetapley
  • 17,000
  • 12
  • 60
  • 86

6 Answers6

28

Just found the solution here. The following regex works correctly:

g/^\(.*\)$\n\1$/d
davetapley
  • 17,000
  • 12
  • 60
  • 86
22
:%!uniq

if you're on a unix system, or a system that has the uniq program

John Weldon
  • 39,849
  • 11
  • 94
  • 127
14

If you want to remove non-contiguous duplicates you could use

:g/^\(.*\)\ze\n\%(.*\n\)*\1$/d

(which will remove all but the last copy of a line)

which would change

Foo
Bar
Foo
Bar
Foo
Baz
Foo
Quux

to

Bar
Baz
Foo
Quux

If you want to remove all but the first copy, try

:g/^/m0
:g/^\(.*\)\ze\n\%(.*\n\)*\1$/d
:g/^/m0

which would change

Foo
Bar
Foo
Bar
Foo
Baz
Foo
Quux

to

Foo
Bar
Baz
Quux
rampion
  • 87,131
  • 49
  • 199
  • 315
3

If you just want to remove contiguous duplicate lines, just use uniq without sorting anything.

:%!uniq
Juliano
  • 39,173
  • 13
  • 67
  • 73
0
:%s/^\(.*\)\(\n\1\)\+$/\1/ge

this is my answer for you

cn8341
  • 129
  • 7
-3

I know this is old, but it's worth mentioning the following also works if you don't mind sorting as well (I know the OP wanted to avoid it):

:sort u
OJ.
  • 28,944
  • 5
  • 56
  • 71