Batch write text from file into local variable error

Question

My scenario is the following:

I have a huge file with 700.000+ lines. I have to work with this, I name this file now trc.txt

The structure of each line of this file is like so:

20958 191014 07:43:57.08 CCComRPC DCMSGCFW_E PID:00000864.00001F40 Data:23
< PREP_FIXED::Process 0

I have a seconde file, I call it classID.txt with 300 lines. Each line have the following structure:

ID_Key;ClassName
720;ComEFM
721;CCComRPC
725;ComSSL
730;WOSA-CRD
731;WOSA-PIN

The aim is now to check my trc.txt how often a specific Class can be found.

The different possible class names are stored in the classID.txt and the name can be found in the fourth element from the left in each line inside the trc.txt.

My procedure right now was to save the different possible ClassNames inside a list-variable. For this I used this for-loop (oriented by this post)

set trcClasses=
for /f "tokens=2 delims=;"  %%i in (classID.txt) do set trcClasses=!trcClasses!,%%i

This seems to work perfectly.

Now to cope with my aim, I thought to iterate through my big-list trc.txt line by line and check each time if one element of the trcClasses occur. If this is so, to count, I implement a simple counter which then increments by one and for that I am using the following code:

for /f "tokens=4 delims= "  %%t in (trc.txt) do (
set "dataRow=%%~t"
set "break="
    for %%l in (%trcClasses%) do if not defined break (
        if not "!dataRow:%%l=!"=="!dataRow!" (
            set /a kumSum%%l+=1
            set "break=1"
        )
    )
)

I then return my values with this:

for%%l in (%trcClasses%) do (
    if (!kumSum%%l! NEQ 0) echo %%l !kumSum%%l!
)

First problem: Console have problems with some items in the classID.txt. I receive something like this:

Error: Division durch Null.
Missing operator

In my opinion this is caused by some of the names inside classID.txt like WOSA-PTR or TCP/IP

The bigger problem: Running the code takes approx. 12 minutes!

Any suggestions would be appreciated.

Answer 1

You didn't specify your desired output format, so I had to guess. Reading each line with a for /f loop is slow, so it's better to process the 300 line file than the 700000 line file (the caching system of modern computers will help a lot).

@echo off
setlocal
for /f "skip=1 tokens=2 delims=;" %%a in (classID.txt) do (
  <nul set /p "=%%a;"
  <trc.txt find /c "%%a"
)

I added skip=1 to skip the header line in classID.txt.
The downside is, you have to read the big file ~300 times, but that still should be faster than processing it line by line (I'd appreciate some feedback about speed comparison of the two methods)

Output with your example files:

ComEFM;0
CCComRPC;1
ComSSL;0
WOSA-CRD;0
WOSA-PIN;0

PS: I assume you want to have the result in a file. Don't write it line by line (>> out.txt inside the for loop. That takes ages because the file has to be opened, read until "end of file", appended and closed again for each single line. Instead redirect the whole loop at once:

(for /f "skip=1 tokens=2 delims=;" %%a in (classID.txt) do (
  <nul set /p "=%%a;"
  <trc.txt find /c "%%a"
))>out.txt