Inexplicable behavior caused by addition of for-loop inside else case of a function while trying to fix nested delayed expansion problem

Question

Foreword

I'm new to batch and have been doing it for about 2 weeks. My function isn't following what I would consider normal program flow rules when I add a specific for loop to it. When the for loop is removed, the function follows what I expect program flow to be, but the output still isn't what I want. The root problem is delayed expansion variables inside delayed expansion variables.

I've checked the parentheses multiple times and I can't find anything unbalanced about them in the full code near the bottom. Program flow wise, this makes no sense, the addition of a for loop affects statements before the for loop. The only thing I can think of is that some weird error is happening where the code simply doesn't execute the full loop and reaches my pause statement right after the loop.

What I'm actually trying to do

I am trying to delimit a one line string by ; with an unknown/variable number of tokens and other possible delimiters in the string (like commas) that I want to delimit LATER. This can't be done with the for loops that batch has premade, FOR /F can only do lines and or set numbers of tokens and the regular for loop delimits by semi colons AND commas at the same time. I want to delimit FIRST by semicolons and THEN by commas later seperately. To do this I'm trying to make my own simple split a line for loop with labels. I ran into a problem involving delayed expansion of my variables. Example here:

Not full code, but code up to the delayed expansion problem

setlocal EnableDelayedExpansion
set "TestArgument=%~1"
set /a "CurrentIndex=0"
set /a "DelimIndex=0"
set /a "LoopCount=0"
:ForEach
IF "!TestArgument:~%CurrentIndex%,1!"=="" (goto :ForEachEnd & echo End of list)
echo testing semicolon
IF NOT "!TestArgument:~%CurrentIndex%,1!"==";" (SET /a CurrentIndex=%CurrentIndex%+1 & goto :ForEach & echo found non ;) ELSE (
    set /a "LengthToSearch=%CurrentIndex%-%DelimIndex%"
    SET "currentArg=!TestArgument:~%DelimIndex%,!LengthToSearch!!

What I want from this specific part of the function

Example inputs:

set "ICMPv4RuleTest="Enable Echo Ping Request ICMPv4;Protocol,ICMPv4;Enabled,Yes;LocalIP,Any;RemoteIP,Any;Direction,In;Action,Allow""

Example desired outputs: a For loop (the labels of the above) where each iteration takes one string delimited by ; so the first iteration would take the Enable Echo Ping Request ICMPv4 value. The whole point of the above is to take the input and substring it in a way so that each iteration is delimited by the semicolon

The code above, specifically the last line causes the problem because (I think) the program reads this as delay expanding !TestArgument:~%DelimIndex%,! (which would error out) then !! (which is nothing) leaving only LengthToSearch as a raw string. What I want is to have the LengthToSearch value be expanded before the substring operation, then expand/run/do the expansion of the substring operation and set that to CurrentArg (a single line string delimited by a ;). I can't use the % signs with LengthtoSearch since it seems to be in the same scope as the substring operation however I can use % signs with DelimIndex (because thats out of scope? I don't know).

What I've tried

I've tried variations of the substring line (last line) including:

SET "currentArg=!TestArgument:~%DelimIndex%,%LengthToSearch%! prints nothing.
SET "currentArg=!TestArgument:~%DelimIndex%,!!LengthToSearch!!! prints 31 (the literal argument of DelimIndex).
SET "currentArg=%TestArgument:~%DelimIndex%,!!LengthToSearch!!% prints nothing.
SET "currentArg=%TestArgument:~%DelimIndex%,!LengthToSearch!% prints nothing.

I've tried some others but they definitely aren't right. The way I think it should be was the

SET "currentArg=!TestArgument:~%DelimIndex%,!LengthToSearch!!.

way, but that just has the aforementioned problems. I did some research on here and found out that someone else had somethign similar to my problem with string search and replace and they used a FOR loop to insert the delayed expansion values into a delayed expansion substring operation as the for loop parameter. I thought I could use the same methodology so I tried this (big change to the last line):

setlocal EnableDelayedExpansion
set "TestArgument=%~1"
set /a "CurrentIndex=0"
set /a "DelimIndex=0"
set /a "LoopCount=0"
:ForEach
IF "!TestArgument:~%CurrentIndex%,1!"=="" (goto :ForEachEnd & echo End of list)
echo testing semicolon
IF NOT "!TestArgument:~%CurrentIndex%,1!"==";" (SET /a CurrentIndex=%CurrentIndex%+1 & goto :ForEach & echo found non ;) ELSE (
    set /a "LengthToSearch=%CurrentIndex%-%DelimIndex%"
    FOR /F "tokens=*" %%A IN ("!LengthToSearch!") DO (SET "currentArg=!TestArgument:~%DelimIndex%,%%A!)

And thats when things got REALLY weird. For some reason, the addition of the forloop causes the whole ELSE case to not execute. When I run my code, the

IF NOT "!TestArgument:~%CurrentIndex%,1!"==";"

Only "runs" once (it doesn't really run all of what it should) and then the loop literally breaks. I don't mean it goes to the end of the loop it simply ignores the whole rest of the function entirely. I've checked parenthesis multiple times in the full code HERE:

Full Code of function

:FirewallRuleTestFunc
echo got to function
setlocal EnableDelayedExpansion
set "TestArgument=%~1"
ECHO PARAM WAS "%~1"
echo TestArgument was "%TestArgument%"
set /a "CurrentIndex=0"
set /a "DelimIndex=0"
set /a "LoopCount=0"
echo trying foreach
:ForEach
echo got to foreach
IF "!TestArgument:~%CurrentIndex%,1!"=="" (goto :ForEachEnd & echo End of list)
echo testing semicolon
IF NOT "!TestArgument:~%CurrentIndex%,1!"==";" (SET /a CurrentIndex=%CurrentIndex%+1 & goto :ForEach & echo found non ;) ELSE (
    echo WHY IS THIS NOT BEING REACHED.
    echo CurrentIndex is %CurrentIndex%
    echo startposition was first: %DelimIndex%
    set /a "LengthToSearch=%CurrentIndex%-%DelimIndex%"
    echo length to search was: !LengthToSearch!
    echo length to search was: %LengthToSearch%
    echo startposition was: %DelimIndex%
    FOR /F "tokens=*" %%A IN ("!LengthToSearch!") DO (SET "currentArg=!TestArgument:~%DelimIndex%,%%A!)
    echo loop of ; Arg was: !currentArg!
    set /a DelimIndex=!CurrentIndex!  
    echo Delim index was: %DelimIndex%
    IF !LoopCount! NEQ 0 (
        echo afterfirstloop
        FOR /F "tokens=1,2 delims=," %%A IN ('netsh advfirewall firewall show rule name^="!RuleName!" ^| FIND "%%A"') DO (
            echo loop of , A was '%%A'
            SET "FoundArgument=%%B"
            SET "FirewallRuleExists=%%A"
            IF "!FirewallRuleExists!"=="" (echo TEST FAILED firewall rule with name "!RuleName!" NOT FOUND)
            IF "!FoundArgument!"=="Yes" (echo Test Passed) ELSE (echo Test Failed was "!FoundArgument!")
        )
    ) ELSE (SET "RuleName=!currentArg!" & set /a "LoopCount=%LoopCount%+1" & echo firstloop done)
    echo got past if else
)
:ForEachEnd
echo end of THING
endlocal
echo SOMETHING
goto:EOF

And none of the echos in either the ELSE case of the second IF (IF NOT "!TestArgument:~%CurrentIndex%,1!"==";"), nor any other echos besides the ones before the second IF ever actually happen. This makes absolutely no sense to me and I can't comprehend why it ignores program flow when the for loop is added and program flow works when the for loop is commented out. All I want to do is have a for loop that loops once per a delimiter that I can specify. My only guess is there is some error happening causing it to go directly to EOF. right after my function is called there is a pause because the rest of my program should be irrelevant. And I'm currently working on just this part. The inputs to the function are like specified below:

Example of function call:

set "ICMPv4RuleTest="Enable Echo Ping Request ICMPv4;Protocol,ICMPv4;Enabled,Yes;LocalIP,Any;RemoteIP,Any;Direction,In;Action,Allow""
call :FirewallRuleTestFunc %ICMPv4RuleTest%

I am beyond frustrated and I just need some help. I'll keep trying various things but this is the last part of my program I need to complete and I just want to get something stable, working and efficient and not have to do what I did previously, use line columns and magic numbers to check firewall settings by token. (I'm testing a script to configure some settings so a company network can be set up).

That's not the preferred [mcve] :-( BTW you know this `set /a` syntax: `set /a "CurrentIndex=DelimIndex=LoopCount=0"` or `SET /a CurrentIndex+=1` ? — , Oct 26 '18 at 14:51
I strongly suggest you to be much more _concise_ in your questions. A simple question don't requires several "chapters"; just describe the facts in a short and simple way and add _one small example_ as a fully working program that show the behavior. Do not write your _opinions_. — Aacini, Oct 27 '18 at 13:24

score 1 · Answer 1 · edited Jun 20 '20 at 09:12

What was wrong

ALRIGHT I found out the problem. I have/had a typo in my FOR loop statement. The line:

FOR /F "tokens=*" %%A IN ("!LengthToSearch!") DO (SET "currentArg=!TestArgument:~%DelimIndex%,%%A!)

In my file that was causing problems had unbalanced quotes ("") in the SET call. This cased a crash of the program thus, not doing the rest of the function and causing me a great deal of confusion. I found it by trying this

FOR /F "tokens=*" %%A IN ("!LengthToSearch!") DO (echo %%A)

and program flow MAGICALLY worked as expected. So I then re-investigated my for loop and discovered the missing quotation.

what about nested delayed expnasion variables?

The for loop methodology suggested (somewhere, can't refind it through google) did indeed solve the nested delayed expansion problem. I don't know how deep you can nest variables. but regardless, doing:

FOR /F "tokens=*" %%A IN ("!LengthToSearch!") DO (SET "currentArg=!TestArgument:~%DelimIndex%,%%A!)

instead of:

SET "currentArg=!TestArgument:~%DelimIndex%,!LengthToSearch!!

Actually works as expected! It splits the string with both indexing values subbed in properly. I can't remember where it was mentioned but I recall the explanation as being this: FOR loop expansion happens before delayed variable expansion but after the initial expansion of % signs. Thus I can put the for loop arguments in a delayed expansion operation and it will work. I don't why exactly why it works, but thats kindof how I understand it. I'd appreciate any additional insights and I will edit this answer if I find out more information on why this problem happens and how I just solved it.

Just for the record, I HATE silent errors like this. I really wish there was better error handling because batch scripting is powerful but needlessly hard because there is little to no good error checking. If only notepad++ showed unbalanced quotations in variables. But for future reference, at least this solves any issues with trying to use nested delayed expansion variables. The solution is simply to use for loop variables as the intermediary.

If anyone finds a good reference to an explanation for the delayed variables expansion behavior and for loop behavior I'd be happy to make required edits so people don't have this problem again

I would suggest you read [How does the Windows Command Interpreter (CMD.EXE) parse scripts?](https://stackoverflow.com/questions/4094699/how-does-the-windows-command-interpreter-cmd-exe-parse-scripts) — Squashman, Oct 26 '18 at 15:19

score 1 · Accepted Answer · answered Oct 27 '18 at 13:47

If you want to split a string at semicolons, you may use this simple method:

@echo off
setlocal

set "ICMPv4RuleTest=Enable Echo Ping RequestICMPv4;Protocol,ICMPv4;Enabled,Yes;LocalIP,Any;RemoteIP,Any;Direction,In;Action,Allow"

echo Original string: "%ICMPv4RuleTest%"
echo/

echo Split string at semicolons:
for %%a in ("%ICMPv4RuleTest:;=" "%") do echo %%a

Output:

Original string: "Enable Echo Ping Request ICMPv4;Protocol,ICMPv4;Enabled,Yes;Lo
calIP,Any;RemoteIP,Any;Direction,In;Action,Allow"

Split string at semicolons:
"Enable Echo Ping Request ICMPv4"
"Protocol,ICMPv4"
"Enabled,Yes"
"LocalIP,Any"
"RemoteIP,Any"
"Direction,In"
"Action,Allow"

I saw that you like to do multiple tests and experiments, so I leave the explanation of this simple method to you... ;)

If you want to know how to combine the expansion of one variable to use it in a further expansion of another variable, then I suggest you to read this answer.