how to get a unique string from a text file using Batch script

Question

I have a text file that contains the following information:

-host A -P 1
-host A -P 2

-host B -P 1
-host B -P 2
-host B -P 3

-host C -P 1
-host C -P 2

-host A -P 3

Now from a Batch script, I want to extract these hostnames uniquely. For example, after fetching my array should look like this: [A B C]

I was able to get these hostnames with this below command:

setlocal ENABLEDELAYEDEXPANSION
set /A f=0
if exist "%TEXTFILE%" for /F usebackq^ delims^=^ eol^= %%I in (%TEXTFILE%) do for %%J in (%%I) do (
    if /I "%%~J" == "-host" (
        set /A f=1
    ) else (
        if !f!==1 (
            echo %%J
            set /A f=0
        ) 
    )
)

I am getting the following output:

A
A
B
B
B
C
C
A

But I only want unique names here. How should we get the unique names from the text file?

Related: [Batch to remove duplicate rows from text file](https://stackoverflow.com/q/11689689) — aschipfl, Mar 02 '21 at 13:56

Stephan · Accepted Answer · 2021-03-02T13:49:34.093

1

@echo off
setlocal
for /f "tokens=2" %%a in (test.txt) do set "_%%a=."
for /f "delims=_=" %%a in ('set _') do echo %%a

Output:

A
B
C

The first for sets a variable for each name (we don't care for the content). The second for lists those variables. For that to happen, all variables have to start with the same character, and no other variables may start with that letter. The set command will even sort them alphabetically for you.

@echo off
setlocal enabledelayedexpansion
set "list=["
for /f "tokens=2" %%a in (test.txt) do set "_%%a=."
for /f "delims=_=" %%a in ('set _') do set "list=!list! %%a"
set "list=%list:[ =[%]"
echo %list%

Output:

[A B C]

edited Mar 02 '21 at 13:49

answered Mar 02 '21 at 13:44

Stephan

53,940
10
58
91

it says "Environment variable _ not defined" – Gaurav Sahu Mar 02 '21 at 13:49
then it didn't read the file (correctly). – Stephan Mar 02 '21 at 13:50
It is reading the file correctly in the code I have posted in Question. Then how come now it is not able to read it? – Gaurav Sahu Mar 02 '21 at 13:54
What's the encoding of the file? – Stephan Mar 02 '21 at 13:56
2

@GauravSahu - You remembered to replace `test.txt` with your file, right? – SomethingDark Mar 02 '21 at 14:06
It's UTF-8 @Stephan – Gaurav Sahu Mar 02 '21 at 14:15
works fine for me with UTF-8 - both with or without BOM. – Stephan Mar 02 '21 at 15:21
wait - what's the exact content of `%TEXTFILE%`? If it has surrounding quotes (bad practice by the way), add `usebackq` – Stephan Mar 02 '21 at 16:09

score 1 · Answer 2 · answered Mar 03 '21 at 02:11

What follows is a rather complex looking batch file, written only for use on Windows 10. The code uses the new, but undocumented, /unique option of the sort command only in that OS version. I should mention here that the sort command is case insensitive so, a and A, for instance, are not unique.

It attempts to get all trimmed content between the line leading -host word and any following -P* word. I did it this way to ensure that strings containing spaces would still be captured. Whitespace is disallowed in hostnames, so you should not need this extra functionality, however, for generic use, it may be useful to you. In addition, should you wish to modify this for other uses later, it does not currently capture substrings beginning with a hyphen, which are also disallowed in hostnames, (you would need to modify the findstr sub-match [^-] on line 8 to allow for such strings). Finally, should your case insensitive line leading word not be -host, you can change that on line 8, and if your second hyphen leading word doesn't begin with the case insensitive character pair -P, you could modify that on line 11.

The intended output should be a variable %array%, which using the example you've posted, should look like this:

"A","B","C"

If you'd prefer not to have the commas, change ^, on line 37 to . Also, if, you dont need the enclosing doublequotes, change "%%I" on line 35 to %%I, and "!$:~1!" on line 37 to !$:~1!.

It should also create individual variables, for each array indexed item, in the format %array[#]%, where # is the index number, zero based, (you could adjust that by changing Set "$=-1" on line 17 to Set "$=0" and array[0] on line 28 to array[1]). Using your posted example the current code should produce the following:

%array[0]% - expanding to string value A
%array[1]% - expanding to string value B
%array[2]% - expanding to string value C

Here is the batch file example, please remember to adjust the value of %TEXTFILE% to the full, (recommended), or relative, path of your source file on line 3:

@Echo Off
SetLocal EnableExtensions DisableDelayedExpansion
Set "TEXTFILE=myfile.txt"
Set "$T=%TEMP%\$.lst"
If Not Exist "%TEXTFILE%" GoTo :EOF
For /F "Delims==" %%G In ('"(Set array) 2>NUL"') Do Set "%%G="
(   For /F Delims^=^ EOL^= %%G In ('Type "%TEXTFILE%" ^|
     %SystemRoot%\System32\findstr.exe /RIC:"^-host[ ][ ]*[^-]"') Do (
        Set "$=%%G"
        SetLocal EnableDelayedExpansion
        For /F "Delims=&" %%H In ("!$: -P=&:!") Do (
            EndLocal
            For /F "Tokens=1,*" %%I In ("%%H") Do Echo %%J
        )
    )
) 1>"%$T%"
Set "$=-1"
For /F Delims^=^ EOL^= %%G In (
    '%SystemRoot%\System32\sort.exe /Unique "%$T%" ^& Del "%$T%"'
) Do (
    Set /A $ += 1
    SetLocal EnableDelayedExpansion
    For %%H In (!$!) Do (
        EndLocal
        Set "array[%%H]=%%G"
    )
)
If Not Defined array[0] GoTo :EOF
For /F "Tokens=1,* Delims=]" %%G In ('"(Set array[) 2>NUL"') Do (
    Set "$=%%H"
    SetLocal EnableDelayedExpansion
    If Not Defined array (
        For /F Delims^=^ EOL^= %%I In ("!$:~1!") Do (
            EndLocal
            Set "array="%%I""
        )
    ) Else For /F UseBackQ^ Delims^=^ EOL^= %%I In ('!array!^,"!$:~1!"') Do (
        EndLocal
        Set "array=%%I"
    )
)
For %%G In (TEXTFILE $T $) Do Set "%%G="

Set array & Pause

The last line is included just to ensure that you can see the resultant variables with their values. After testing, you can replace that with your own code, as needed.

how to get a unique string from a text file using Batch script

2 Answers2