Removing duplicate rows in Notepad++

Question

Is it possible to remove duplicated rows in Notepad++, leaving only a single occurrence of a line?

score 803 · Answer 1 · edited Dec 12 '19 at 00:01

803

Since Notepad++ Version 6 you can use this regex in the search and replace dialogue:

^(.*?)$\s+?^(?=.*^\1$)

and replace with nothing. This leaves from all duplicate rows the last occurrence in the file.

No sorting is needed for that and the duplicate rows can be anywhere in the file!

You need to check the options "Regular expression" and ". matches newline":

Notepad++ Replace dialogue

^ matches the start of the line.
(.*?) matches any characters 0 or more times, but as few as possible (It matches exactly on row, this is needed because of the ". matches newline" option). The matched row is stored, because of the brackets around and accessible using \1
$ matches the end of the line.
\s+?^ this part matches all whitespace characters (newlines!) till the start of the next row ==> This removes the newlines after the matched row, so that no empty row is there after the replacement.
(?=.*^\1$) this is a positive lookahead assertion. This is the important part in this regex, a row is only matched (and removed), when there is exactly the same row following somewhere else in the file.

edited Dec 12 '19 at 00:01

qyb2zm302

6,458
2
17
17

answered Apr 30 '13 at 06:27

stema

90,351
20
107
135

1

This one is better indeed than the other regex. No need for multiple passes to eliminate all duplicates. – Benny Jun 20 '13 at 03:55
15

oh, this one is brilliant, it even deletes empty rows, i'm macroing it this very moment :) – Aprillion Jun 28 '13 at 16:14
1

Great to learn. Precise explanation too! Thanks to both the raiser and reply-er! – SarjanWebDev Oct 29 '13 at 01:50
75

It just removes ALL lines in a file in some cases. – SerG Feb 20 '14 at 13:56
3

Is there any way to remove the LAST occurrence? This matches all but the last one... – Cullub Sep 23 '14 at 11:18
38

In my case where this solution removed all lines, unchecking the `. matches newline` did the trick. – Kuitsi Dec 01 '15 at 07:34
2

**Perfect!** I was using Notepad++ on a locked-down system with no internet access. No way to download plugins, so this was better for me. – ADTC Jan 06 '16 at 05:47
4

@SerG In some cases it didn't work for me also, but when I removed "matches newline" it did :) – Davidenko Dec 19 '16 at 08:50
In my case, ALL lines were removed, as happened with SerG. But, when I left unchecked "matches newline", it worked perfectly, as happened with Davidenko. – aldemarcalazans Jun 21 '18 at 23:09
2

If you adjust the capture group a little bit you can fix the side effect of deleting the file: ^([^\r\n]*)$\s+?^(?=.*^\1$) – Nick Kuznia Sep 30 '18 at 05:16
1

only found 1 line – user924 Oct 24 '18 at 18:38
2

For me it worked correctly only after sorting out lines with the native ordering function (Menu Edit → Line Operations → Sort Lines Lexicographically Ascending / Descending) – Mariano Paniga Oct 29 '18 at 11:03
1

Perhaps add some statement about the actual performance? It sounds like it must at least have quadratic performance (both memory and execution). What is the actual number of lines for which it takes more than 1 second to execute? – Peter Mortensen Nov 19 '18 at 17:22
1

For those who want to keep the first occurrence and delete the rest, reverse the lines first https://superuser.com/questions/331098/flip-or-reverse-line-order-in-notepad and then use the regex above and then reverse again. – Shayan Sep 21 '19 at 14:45
perfect for quick&dirty editing with "lists" copied into notepad++ – bucky Nov 04 '20 at 13:07
for being more cautionary, I use Find Next, and remove the duplicate manually. – Noor Hossain Jan 18 '21 at 19:40

Colin Pickard · Accepted Answer · 2020-08-12T13:21:50.090

792

Notepad++ with the TextFX plugin can do this, provided you wanted to sort by line, and remove the duplicate lines at the same time.

To install the TextFX in the latest release of Notepad++ you need to download it from here: https://sourceforge.net/projects/npp-plugins/files/TextFX

The TextFX plugin used to be included in older versions of Notepad++, or be possible to add from the menu by going to Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX -> Install. In some cases it may also be called TextFX Characters, but this is the same thing.

The check boxes and buttons required will now appear in the menu under: TextFX -> TextFX Tools.

Make sure "sort outputs only unique..." is checked. Next, select a block of text (Ctrl+A to select the entire document). Finally, click "sort lines case sensitive" or "sort lines case insensitive"

menu layout in n++

edited Aug 12 '20 at 13:21

answered Oct 18 '10 at 10:46

Colin Pickard

45,724
13
98
148

39

Incredibly powerful plugin, despite its "age". Hope they will NEVER remove that one from the standard NPP plugin offer. The guy who thought about all the features in this plug-in, was kind of a "visionary". – GeertVc Sep 01 '14 at 09:32
Note that this method does not give any kind of warning if the file is read-only. My file was sorted anyway, so it seemed that the tool had worked, until I spotted a duplicate. Quite frustrating until I tried @stema's search & replace method, which did warn me. – JV. Dec 04 '14 at 12:44
8

More powerful than excel. – Vasu Apr 22 '15 at 21:58
2

Textpad does it with one key - F9 hoping NP++ can also allow hotkey for this operation. – prash Apr 12 '16 at 09:11
@GeertVc: was that sarcasm, zynism or something? There's no TextFX plugin in my installation – Thomas Weller Jun 01 '16 at 10:35
NO I don't want to sort anything – ACV Dec 20 '16 at 10:01
11

What about Notepad++ x64 version? Plugin TextFX x64-version not exists – Geograph Jan 14 '18 at 15:22
i am not able to check the `Sort outputs only` option. What to do? – th3pirat3 Mar 09 '18 at 18:18
28

TextFx is not in the 64 bit version. – Rhyous Mar 23 '18 at 00:27
6

@Geograph And there will be no 64 bit plugin of TextFx see this [note](http://docs.notepad-plus-plus.org/index.php/TextFX%27s_Future). Therefore it would be good to know if there is an alternative plugin providing sort and duplicate removal. – Robert Oct 23 '18 at 13:32
The only downside is that it sorts the lines.. I don't wanna change the order of my lines. – Shayan Sep 21 '19 at 14:31
TextFX has been phased out - from *[TextFX's Future](http://docs.notepad-plus-plus.org/index.php/TextFX%27s_Future)*: *"... bid farewell to an aging workhorse that has served the community well."* (though the link is broken now) – Peter Mortensen Oct 02 '19 at 20:38
And for safety, you might want to remember to UNcheck "sort outputs only unique..." when you're done, so that a month from now, after you've forgotten all this, it does not happen unexpectedly, silently – john v kumpf Oct 15 '19 at 16:32
"You will need the TextFX plugin" ... then really it's TextFX doing it, rather than Notepad++ doing it. – Stewart Oct 17 '19 at 14:56
@GeertVc I don't see is in the Available tabs for Plugins Manager so I believe it was removed from the standard NPP plugin. – John Odom Aug 10 '20 at 20:11
@JohnOdom: That is correct. And I anticipated on that by saving it locally on my system. But you still can find it here: https://sourceforge.net/projects/npp-plugins/files/TextFX/. I would suggest: if you want to use it, take your copy and also save it locally. It's more than worth it. And I know, it isn't working on the 64 bit version of NPP but hey... I don't care... I'm very satisfied with the 32 bit version. As long as I have the capability to install TextFX, that's almost the only thing that matters to me... – GeertVc Aug 12 '20 at 05:25
@Stewart: That's all about NPP. It's also not NPP who does the real text editing and manipulation. It's Scintilla which is the real workhorse and doing the heavy work in the background. NPP is merely, and I say this with the utmost respect for the guy who made NPP, a UI on top of lots of other "helper" stuff, like Scintilla, like the many plugins,... I'm using NPP now for already 15+ years or so and since CodeWright (R.I.P.) it's the best one I found out there **for my purposes** (beware: I'm not saying it's **the** best one out there...). – GeertVc Aug 12 '20 at 05:30
@GeertVc I did tried installing the latest version from there but I got an error saying that it is not supported by the latest version of Notepad++, so I went with stema's answer instead that works for me. – John Odom Aug 12 '20 at 13:15
Thanks for the update @GeertVc, I have updated the answer with the link you posted. – Colin Pickard Aug 12 '20 at 13:22
@JohnOdom: what latest version of NPP? 64 bit? Then the answer is: no it's not supported. I'm currently using NPP 7.8.9 - 32 bit (the latest version of NPP AFAICS, related to Hong Kong even...) and TextFX is working like a charm... – GeertVc Aug 13 '20 at 15:07
@GeertVc. It's 64-bit, so that would explain it. – John Odom Aug 13 '20 at 19:13
Saved a lot of time. Thanks for the pictorial demo. – Chandra Shekhar Feb 15 '21 at 08:20
1

Although this is the accepted answer, it's not applicable to more recent versions of Notepad++. See this answer instead: https://stackoverflow.com/a/58549356/1681788 – Mark Barnes Mar 04 '21 at 11:00
1

64 bit version of TextFX https://github.com/HQJaTu/NPPTextFX/blob/VS2017-x64/bin/x64/NppTextFX.dll – Urbley Mar 02 '22 at 17:47

score 109 · Answer 3 · edited Oct 02 '19 at 20:43

109

If the rows are immediately after each other then you can use a regex replace:

Search Pattern: ^(.*\r?\n)(\1)+

Replace with: \1

edited Oct 02 '19 at 20:43

Peter Mortensen

30,738
21
105
131

answered Oct 18 '10 at 10:53

Grant Peters

7,691
3
45
57

1

Maybe others have had luck with this, but for me ^(.*\n)\1 results in "Cant find the text" – arkon Apr 28 '12 at 18:18
4

@b1naryatr0phy make sure you have "Search Mode" set to "Regular expression", I also updated the pattern so that it can handle windows style line endings – Grant Peters May 01 '12 at 13:25
3

notepad++ has a light regex engine, it dosen't permit advanced functios, not even the "? or \r\n" as it only works on a single line and you use $ for the \r\n characters – Stefan Rogin May 25 '12 at 16:39
3

this eliminates one by one. You must repeat it many times. I wonder why \n+ -> \n does not work (thought it reports many replacements) – Val Sep 06 '12 at 09:48
2

@Val, if you make the back-reference part of the match a group with 1-or-more matches required, the pattern will match N contiguous duplicate lines at a time: `^(.*\r?\n)(\1)+` – Kenigmatic Apr 29 '16 at 23:05
If you want to find unique characters in a file first use this: https://superuser.com/questions/1088622/notepad-split-line-after-given-number-of-characters then sort the rows and then use this answer. A bit long winded but it works. – wobblycogs Jun 01 '20 at 11:58

score 83 · Answer 4 · answered Aug 04 '21 at 09:24

83

As of Notepad++ version 8.1, there is a specific command to do precisely what this popular question asks. On can remove duplicated rows in a text file with the menu command Edit > Line Operations > Remove Duplicate Lines.

There is no need to install a plugin (as the currently accepted answer suggests), or sort the lines beforehand, or use the regex syntax in the Replace dialogue as other answers suggested.

answered Aug 04 '21 at 09:24

divenex

15,176
9
55
55

4

This is perfect solution and saving my lots of time to go to Excel and perform Einstein based operations. – Dnyaneshwar Jadhav Sep 27 '21 at 12:42
4

This works perfectly out of the box. Should be the accepted answer IMHO – TimothyHeyden Apr 19 '22 at 11:53
2

Excellent answer and easiest approach. It worked perfectly for me. – prashant thakre May 03 '22 at 07:16
3

This should be marked as the right solution, as the other is outdated – jhe Aug 17 '22 at 08:52

score 49 · Answer 5 · edited Dec 13 '19 at 17:46

49

In version 7.8, you can accomplish this without any plugins - Edit -> Line Operations -> Remove Consecutive Duplicate Lines. You will have to sort the file to place duplicate lines in consecutive order before this works, but it does work like a charm.

Sorting options are available under Edit -> Line Operations -> Sort By ...

edited Dec 13 '19 at 17:46

PeterK

1,697
10
20

answered Oct 24 '19 at 21:28

dr.nixon

657
5
11

10

There's also now an option for Edit -> Line Operations -> Remove Duplicate Lines which eliminates the need to sort. – Mark Barnes Mar 04 '21 at 10:50
1

Whats the difference between "Remove Duplicate Lines" and "Remove Consecutive Duplicate Lines" – user3304007 Apr 24 '21 at 19:05
3

First option should remove all but one of each matching line in a document (so a, a, b, a, c would become a, b, c). Second option should only remove lines that are repeated immediately after a matching line (a, a, b, a, c would become a, b, a, c). – dr.nixon Apr 26 '21 at 15:48

score 40 · Answer 6 · edited Oct 02 '19 at 20:46

40

Notepad++

-> Replace window

Ensure that in Search mode you have selected the Regular expression radio button

Find what:

^(.*)(\r?\n\1)+$

Replace with:

$1

Before:

and we think there

and we think there

single line

Is it possible to

Is it possible to

After:

and we think there

single line

Is it possible to

edited Oct 02 '19 at 20:46

Peter Mortensen

30,738
21
105
131

answered May 20 '17 at 22:21

blueberry0xff

3,707
30
18

3

Isn't the file required to be sorted for this to work? – Peter Mortensen Nov 19 '18 at 17:24

score 39 · Answer 7 · edited Nov 19 '18 at 16:42

39

If you don't care about row order (which I don't think you do), then you can use a Linux/FreeBSD/Mac OS X/Cygwin box and do:

$ cat yourfile | sort | uniq > yourfile_nodups

Then open the file again in Notepad++.

edited Nov 19 '18 at 16:42

Peter Mortensen

30,738
21
105
131

answered Oct 18 '10 at 10:46

Pablo Santa Cruz

176,835
32
241
292

4

Doesn't work on Windows 7. `'cat' is not recognized as an internal or external command, operable program or batch file.` – Iain Samuel McLean Elder Dec 11 '14 at 16:50
2

@Iain Elder: cat is a standard Unix utility, which is why this answer specifies that it works on linux, FreeBSD, and MacOSX. The answer also suggests Cygwyn: This is a windows program that gives you a unix style shell, and with it, cat. Long story short (too late!): Win 7 needs Cygwin to do this. – Travis Clark Jan 14 '15 at 16:14
16

In windows you have powershell: `cat yourfile | sort -Unique` – Elazar Aug 05 '15 at 11:32
11

These are good examples of "the gratuitous use of cat". Forget about the cat utility and just use file redirection thusly: sort **< yourfile** | uniq > yourfile_nodups – scott8035 May 16 '16 at 18:57
1

@scott8035, I agree that cat is of no use for running that command, but I find it often helpful to start with cat when figuring out a long sequence of non-obvious commands, like cat file | sed ... | sed ... | sed ... and so on. So I'd say that there might be reasons for using cat. Of course cat can be removed at the end, but some are too lazy for that. – FORTRAN Sep 14 '17 at 06:56
You can install bash now on Windows 10, just search "Ubuntu" in Microsoft Store and follow the instructions in the Description. – Patronaut Feb 23 '18 at 14:46

score 18 · Answer 8 · edited May 23 '17 at 12:02

The latter versions of Notepad++ do not apparently include the TextFX plugin at all. In order to use the plugin for sorting/eliminating duplicates, the plugin must be either downloaded and installed (more involved) or added using the plugin manager.

A) Easy way (as described here).

Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX Characters -> Install

B) More involved way, if another version is needed or the easy way does not work.

Download the plugin from SourceForge:

http://downloads.sourceforge.net/project/npp-plugins/TextFX/TextFX%20v0.26/TextFX.v0.26.unicode.bin.zip
Open the zip file and extract NppTextFX.dll
Place NppTextFX.dll in the Notepad++ plugins directory, such as:
C:\Program Files\Notepad++\plugins
Start Notepad++, and TextFX will be one of the file menu items (as seen in Answer #1 above by Colin Pickard)

After installing the TextFX plugin, follow the instructions in Answer #1 to sort and remove duplicates.

Also, consider setting up a keyboard shortcut using Settings > Shorcut mapper if you use this command frequently or want to replicate a keyboard shortcut, such as F9 in TextPad for sorting.

In notepad++ 7.6, the plug-in should be added to `C:\Users\\AppData\Local\Notepad++\plugins\NppTextFX` . Other than that this still works fine. — P_W999, Dec 18 '18 at 07:34

score 17 · Answer 9 · answered Jun 25 '20 at 09:06

As of now, it's possible to remove all consecutive duplicate lines with Notepad in-built functionality. Sort the lines first:

Edit > Line Operations > "Sort lines lexicographically",

then

Edit > Line Operations > "Remove Consecutive Duplicate Lines".

The regex solution suggested above didn't remove all duplicate lines for me, but just the consecutive ones as well.

Donald · Answer 10 · 2018-07-18T03:23:54.543

8

You may need a plugin to do this. You can try the command line cc.ddl(delete duplicate lines) of ConyEdit. It is a cross-editor plugin for the text editors, including Notepad++.

With ConyEdit running in background, follow the steps below:

enter the command line cc.ddl at the end of the text.
copy the text and the command line.
paste, then you will see what you want.

Example

edited Jul 18 '18 at 03:23

answered Jul 07 '18 at 14:53

Donald

131
1
4

score 6 · Answer 11 · edited Oct 02 '19 at 20:45

6

Search for the regular expression: \b(\w+)\b([\w\W]*)\b\1\b

Replace it with: $1$2

Hit the Replace button until there are no more matches for the regular expression in your file.

edited Oct 02 '19 at 20:45

Peter Mortensen

30,738
21
105
131

answered May 22 '14 at 13:05

Hesham Eraqi

2,444
4
23
45

Created a test file to try this, but the regular expression did not work reliably to get the job done. – RockPaperLz- Mask it or Casket Mar 20 '16 at 05:29
Would you please provide an example that fails so I can improve my answer? – Hesham Eraqi Aug 03 '19 at 00:07
1

Remove duplicates leaving also the original row number position of other text, I like this solution – Jack Rock Jan 16 '21 at 18:18

score 6 · Answer 12 · answered Jul 27 '22 at 08:26

Notepad++ has builtin operations:

Edit -> Line Operations -> Sort Lines...
Edit -> Line Operations -> Remove Duplicate Lines

Maybe it works with just Remove Duplicate Lines, but I needed to see that the operations work by seeing that sorting works.

If it doesn't work, problem could be with different ending of lines, which I encountered now. You can check it with View -> Show Symbol -> Show End of Line. Replace it to have it same.

Click on Search > Replace (or Ctrl + H)
Find what:  \r\n
Replace with: \n
Search Mode: select Extended (\n, \r,...)
Replace All

Manohar Reddy Poreddy · Answer 13 · 2018-09-11T13:45:28.130

4

None worked for me.

A solution is:

Replace

^(.*)\s+(\r?\n\1\s+)+$

with

\1

edited Sep 11 '18 at 13:45

answered Mar 24 '15 at 00:13

Manohar Reddy Poreddy

25,399
9
157
140

Created a test file to try this, but the regular expression did not work reliably to get the job done. – RockPaperLz- Mask it or Casket Mar 20 '16 at 05:28
For all my data, it worked fine.I forgot what my solution was. Add more details where it failed so that other people might improve this regex. – Manohar Reddy Poreddy Mar 20 '16 at 05:46
I created a file so each line had a integer between 0-999 on it, in random order, sometimes with duplicates. It didn't remove most of the duplicates, and didn't remove any duplicates there were not sequential. – RockPaperLz- Mask it or Casket Mar 20 '16 at 05:58
1

Please do provide 2 examples for working and for not-working ones. It will help someone. – Manohar Reddy Poreddy Mar 20 '16 at 09:01
the only one that worked for me (npp 7.3). thanks :-) – Sickboy Nov 22 '17 at 15:49
1

why `^(.*)\s+(\r?\n\1\s+)+$` and not `^(.*)\s*(\r?\n\1\s*)+$` ? – Mark Ch Sep 11 '18 at 13:19
hey, I lost the context of this regex I wrote long back, but the difference you point is either 1 or more characters vs 0 or more characters, but if I wrote it + instead of * that must mean that I tried * then came to a + solution, so the answer must be correct to the question asked. – Manohar Reddy Poreddy Sep 11 '18 at 13:44

αғsнιη · Answer 14 · 2022-09-18T16:13:30.427

4

Whether the file is sorted or not, you can use below regex to remove duplicates in anywhere occurred in your file.

Find what: ^([^\r]*[^\n])(.*?)\r?\n\1$
Replace with: \1\2
Search Mode:

"Regular expression"
Check the ". matches newline" option

Click "Replace All" as many time as possible (or press and hold Alt+A shortcut key) until you see "0 occurrences were replaced"

edited Sep 18 '22 at 16:13

answered Jan 13 '21 at 07:19

αғsнιη

2,627
2
25
38

I like this because you're not forced to sort the contents of the file first. It also can be used on any text editor that supports Perl regex. – foundationer May 26 '21 at 19:22

score 2 · Answer 15 · edited Oct 02 '19 at 20:48

The plugin manager is currently unavailable (does not come with the distribution) for Notepad++. You must install it manually (https://github.com/bruderstein/nppPluginManager/releases) and even if you do, a lot of the plugins are not available anymore (no TextFX) plugin.

Maybe there is another plugin which contains the required functionality. Other than that, the only way to do it in Notepad++ is to use some special regex for matching and then replacing (Ctrl + F → Replace tab).

Although there are many functionalities available via Edit menu item (trimming, removing empty lines, sorting, converting EOL) there is no "unique" operation available.

If you have Windows 10 then you can enable Bash (just type Ubuntu in Microsoft Store and follow the instructions in the description to install it) and use cat your_file.txt | sort | uniq > your_file_edited.txt. Of course you must be in the same working directory as "your_file.txt" or refer to it via its path.

score 0 · Answer 16 · answered Mar 28 '22 at 22:51

Extending the top answer, you can also use a 2nd lookahead to find rows that are almost duplicates of other rows.

^(\s*(<PackageReference Include=".*" Version=).*)$\s+?^(?=.*^\2.*$)

Here I'm after multiple references to the same <PackageReference Include=".*" string, regardless of its version.

Test data

<PackageReference Include="Package1" Version="2.2.1" />

    <PackageReference Include="Package1" Version="2.2.1" /> // Match
<PackageReference Include="Package1" Version="2.2.2" />

<PackageReference Include="Package2" Version="5.1" /> // Match
<PackageReference Include="Package2" Version="5.2" />

<PackageReference Include="Package3" Version="2.2.1" /> // No match
<PackageReference Include="Package4" Version="2.2.1" />

See a breakdown of what the regex terms mean and try with your own data on this regex101 share.

score -1 · Answer 17 · answered Jan 21 '21 at 10:57

-1

Difficult to do this in NPP. Better way is following:

Download cygwin utility, it is simple Linux terminal under windows. It allow to execute any Linux command in Windows. And you have sort -u there.

answered Jan 21 '21 at 10:57

haykp

435
12
29

2

Why is it difficult? Have you seen other answers? What's wrong with them? – Toto Jan 21 '21 at 15:11

Removing duplicate rows in Notepad++

17 Answers17

Linked

Related