1

I want to remove duplicated lines from a text file but keeping only 1st occurence using windows batch scripting.

I tried many ways but it's long and not efficient.

would you please help ?

user212051
  • 93
  • 1
  • 2
  • 11
  • 2
    Show your work so far. – SiKing Mar 24 '15 at 19:55
  • Yeah, this is the sort of thing GnuWin32 or UnixUtils would really help with. You could just `uniq infile > outfile` (or, actually, I think `sort infile | uniq > outfile` probably). This'll be a bit of a pain and inefficient with pure batch -- although it certainly is possible. – rojo Mar 24 '15 at 20:23
  • I've tried uniq but it's not supported .. Do you know a way to make it available ? – user212051 Mar 24 '15 at 20:25
  • http://unxutils.sourceforge.net/ – rojo Mar 24 '15 at 20:28

1 Answers1

1

The Batch file below do what you want:

@echo off
setlocal EnableDelayedExpansion
set "prevLine="
for /F "delims=" %%a in (theFile.txt) do (
   if "%%a" neq "!prevLine!" (
      echo %%a
      set "prevLine=%%a"
   )
)

If you need a more efficient method, try this Batch-JScript hybrid script that is developed as a filter, that is, similar to Unix uniq program. Save it with .bat extension, like uniq.bat:

@if (@CodeSection == @Batch) @then

@CScript //nologo //E:JScript "%~F0" & goto :EOF

@end

var line, prevLine = "";
while ( ! WScript.Stdin.AtEndOfStream ) {
   line = WScript.Stdin.ReadLine();
   if ( line != prevLine ) {
      WScript.Stdout.WriteLine(line);
      prevLine = line;
   }
}

Both programs were copied from this post.

Community
  • 1
  • 1
Aacini
  • 65,180
  • 12
  • 72
  • 108
  • This doesn't work if the file contains exclamation marks. :( – zreptil Feb 07 '19 at 08:18
  • @zreptil: I suppose that with "This" you are referring to the pure Batch version... – Aacini Feb 07 '19 at 13:01
  • @Aacini Yes, i wanted to edit my comment, but i recognized this ten minutes too late ^^ It only is a problem with the pure batch version. The other version works without any problem :) – zreptil Feb 07 '19 at 19:58