Array of file names inside a for loop with delayed expansion (windows batch file)

Question

Background

I am attempting to sync files from my hard drive to my MEGA account (https://mega.nz/), a free encrypted online data storage service. Unfortunately, MEGA has not released any Windows command line utilities and their current Windows sync client (https://mega.nz/#sync) needs a lot of work. It deleted all my local files one time when I was attempting to upload. I was able to find some third-party utilities on GitHub called Megatools (https://megatools.megous.com/) that are severely lacking in their functionality. For example: the copy function does not give you the option of overwriting files on the MEGA server with newer versions from the hard drive. As such, I have sought to expand Megatools by writing a Windows Batch File program.

Windows Batch Sync Program

The program is pretty simple: it scans the local sync folder using FORFILES (http://ss64.com/nt/forfiles.html) and then compares properties such as file name, relative path, size, and date modified against a listing of remote files. The remote files are quickly populated using the Megatools megals.exe function. Properties are stored in arrays. Please note that I am aware of the fact that Windows batch files do not treat arrays like other programming languages do, that to the compiler it is simply another variable.

Problem

Because FORFILES functions as a basic for loop, and I'm using indexed arrays to store information, I need to enable Delayed Expansion for the indexing to work as desired. I really don't understand Delayed Expansion very well. I've tried researching it a lot, but it doesn't make any sense to me. Since the syntax for Delayed Expansion involves encapsulating variables with !var! instead of %var% and my file names may contain a ! symbol, I'm at a loss as to what to do.

I researched the problem and apparently the solution is to temporarily disable Delayed Expansion using SETLOCAL DisableDelayedExpansion and ENDLOCAL statements, but that involves limiting the scope of any variables created using the SET statement to lines between SETLOCAL and ENDLOCAL.

The one solution I found to this problem was to append ENDLOCAL with & SET newvar=%oldvar% but newvar remains empty whenever I try to run the program!

Code (abridged)

@ECHO off
SETLOCAL EnableDelayedExpansion

REM ...
REM (Parameter processing code, irrelevant)
REM ...

SET localpathempty=UNSET
SET numlocal=UNSET
CALL:islocalpathempty
IF "%localpathempty%"=="FALSE" (
    ECHO Items found in local directory.  Gathering file names...
    SETLOCAL DisableDelayedExpansion
    FOR /F "tokens=*" %%a IN ('FORFILES /S /P "%localpath%." /C "cmd /c echo @FILE"') DO (
        SET /A "j+=1"
        SET "line=%%~a"
        SETLOCAL EnableDelayedExpansion
        SET localfilename[!j!]=!line!
        ECHO Array Index inside 'LOCAL': !j!
        ENDLOCAL
        ECHO Array Index outside 'LOCAL': %j%
    )
    ENDLOCAL & SET numlocal=%j%
)
IF "%localpathempty%"=="TRUE" (
    ECHO WARNING: LOCAL PATH %localpath% IS EMPTY.  PROCESSING SKIPPED.
)
ECHO Final Index: %j%
ECHO Total Items: %numlocal%

REM ...
REM (Further processing of local files and remote files)
REM ...

ENDLOCAL

GOTO :eof

REM ...
REM (functions)
REM ...

Results

Items found in local directory.  Gathering file names...
Array Index inside 'LOCAL': 1
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 2
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 3
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 4
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 5
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 6
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 7
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 8
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 9
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 10
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 11
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 12
Array Index outside 'LOCAL': 
Array Index inside 'LOCAL': 13
Array Index outside 'LOCAL': 
Final Index: 
Total Items:

Question

As you can see, the index that is set before the second SETLOCAL EnableDelayedExpansion statement does not survive that block of code. I need to be able to record that final index as well as the file name. I did not get so far as to try and pass the file name past the second SETLOCAL EnableDelayedExpansion and ENDLOCAL statements because the index doesn't even survive! Any help would be greatly appreciated with this problem. I realize my syntax may have some issues but I've done my best. The index disappearing is a huge mystery to me.

PS: If you're wondering why I'm starting with Delayed Expansion enabled instead of disabled, the reason is because the vast majority of the code that occurs later (not shown here) requires it to be enabled.

You still need delayed expansion: `ECHO Array Index outside 'LOCAL': !j!` because `j` is still inside a code-block (`do ( ...... )`) — Stephan, May 06 '16 at 20:31

score 0 · Answer 1 · edited May 23 '17 at 12:07

The string value of an environment variable referenced with %name of variable% is first inserted into a command line before the command line is processed at all.

So on a line with

echo Path extensions are: %PATHEXT%

the Windows command processor first replaces %PATHEXT% by the string value of the environment variable PATHEXT and THEN interprets the line which means calls internal function echo which writes the string not containing anymore %PATHEXT% to standard stream stdout.

A block starting with ( and ending with ) is interpreted by Windows command processor as a command line spanning over multiple lines. Therefore the Windows command processor on finding ( being beginning of a block searches for the matching ). Then all occurrences of %name of variable% in current line and in this block are replaced by the current string values of the referenced variables.

This variable expansion before processing the lines at all results very often in having unexpected behavior if a variable is defined or modified within such a block. The entire block was once parsed by command processor before executing any command in the block and therefore updates on environment variables during processing the lines are not taken into account anymore on executing the lines of the block.

That's the reason why delayed expansion is necessary whenever a variable is defined or modified within a block starting with ( and ending with ) and the current variable value is evaluated also within this block.

Note: Loop variables are not environment variables. Loop variables are always "expanded delayed".

This substitution of %name of variable% by current string value of the variable can be easily seen on running a batch file with @ECHO OFF removed from first line of batch file from within a command prompt window because then Windows command processor outputs what is really processed on each line.

But setlocal EnableDelayedExpansion does not just enable delayed expansion giving ! now a special meaning anywhere (including echo Hello!), it additionally makes a copy of current environment table and pushes also current state of delayed expansion, state of usage of extensions, and current directory on stack. Later when endlocal is explicitly or implicitly called, the current environment table is discarded and the state of delayed expansion, the state of usage of extensions, and the current directory are restored from stack.

In other words setlocal creates always really a complete local environment.

A simple solution on working with blocks avoiding confusions is very often enabling delayed expansion at top of the batch file using setlocal EnableDelayedExpansion, explicitly or implicitly restore previous environment at bottom of the batch file with endlocal, and reference environment variables (not loop variables or parameters passed to batch file or a subroutine) using !...! instead of %...%. It is always safe with delayed expansion enabled to use !...! instead of %...% even if the referenced variable is not modified within a block.

However, batch files can be always written without using a block at all by using GOTO and CALL. Therefore it is very often possible to write batch files also without the need to use delayed expansion.

Here is your code rewritten for not using a block.

@ECHO off
REM ...
REM (Parameter processing code, irrelevant)
REM ...

SET LocalPathEmpty=UNSET
SET NumLocal=UNSET
SET FileIndex=0
CALL :IsLocalPathEmpty

IF "%LocalPathEmpty%"=="TRUE"  GOTO NoLocalPath
IF "%LocalPathEmpty%"=="FALSE" GOTO ProcessFiles
REM There is something wrong with subroutine IsLocalPathEmpty.
GOTO :EOF

:ProcessFiles
ECHO Items found in local directory.  Gathering file names...
FOR /R "%LocalPath%" %%a IN (*) DO CALL :AddLocalFile "%%~a"
GOTO OutputSummary

:NoLocalPath
ECHO WARNING: LOCAL PATH %LocalPath% IS EMPTY. PROCESSING SKIPPED.

:OutputSummary
ECHO Final Index: %FileIndex%
ECHO Total Items: %NumLocal%

REM ...
REM (Further processing of local files and remote files)
REM ...

GOTO :EOF

REM ...
REM (functions)
REM ...

:AddLocalFile
SET /A FileIndex+=1
SET "LocalFileName[%FileIndex%]=%~1"
EXIT /B

Please note that I could not test this code on syntax errors because it is not executable.

set /? executed from within a command prompt window outputs multiple pages of a help explaining the difference of normal and delayed environment variable expansion on an IF and a FOR block.

See also answers on Echoing a URL in Batch and why is my cd %myVar% being ignored?

I replaced forfiles also by a simple for loop as this command can also list recursively all files in a specified directory. Run for /? in a command prompt window for help and details on for /R syntax.