2

today I was trying to remove duplicate lines on a simple text file, something like:

input (list.txt):

hello
hello
try

output (list.txt):
try

i was trying with notepad++ to remove duplicate rows and remove the remaining one but nothing. is there a software o some function for do this with notepad++?

thanks.

nabasoki
  • 57
  • 2
  • 7
  • If you have the relative plugin available, [this might help](https://stackoverflow.com/questions/3958350/removing-duplicate-rows-in-notepad) – Paul T. Dec 31 '17 at 04:46
  • @PaulT. i tried with TextFX but remove and keep 1 row i need to remove all rows – nabasoki Dec 31 '17 at 04:48
  • Oh, I see. Are there only single words, or can there more than one word duplicated across lines? ... and is there only a few lines, or are we talking hundred of lines? – Paul T. Dec 31 '17 at 04:55

2 Answers2

9

Assuming the file is sorted, to have all duplicate lines together.

  • Ctrl+H
  • Find what: ^(.+(?:\R|$))\1+
  • Replace with: LEAVE EMPTY
  • check Wrap around
  • check Regular expression
  • DO NOT CHECK . matches newline
  • Replace all

Explanation:

^           : beginning of line
  (         : start group 1
    .+      : 1 or more any character but newline
    (?:     : start non capture group
      \R    : any kind of linebreak
     |      : OR
      $     : end of line
    )       : end group
  )         : end group 1
  \1+       : back-reference to group 1, may appear 1 or more times

Result for given example:

try
Toto
  • 89,455
  • 62
  • 89
  • 125
  • 4
    If you want to leave a single occurrence of the lines, put `\1` in 'Replace with'. – Gabriel Luci Jul 25 '18 at 12:41
  • @GabrielLuci has an important point. I copied the regex and used it and figured out I needed the \1 in the replacement field, then came back and saw his comment. I think this is a common enough variation of the technique that including it in the solution makes it easier for future visitors. But it is a terrific elegant solution. (I used to used TEXTFX's sort/unique option) – RufusVS Oct 18 '18 at 16:57
0

you can do it with php by exploding each line to an array then using the array_unique to get rid of duplicate values then implode the array using \n as a seperator. It can be done in php with 6 lines of code or less readfile explode file unique_array file implode file write file close file return file

Matthew Lagerwey
  • 120
  • 6
  • 1
    It would be better (an improved answer) to show the actual PHP code that you describe as an example, as the OP might not know PHP. – Paul T. Dec 31 '17 at 04:45