Preserve a variable across a DisableDelayedExpansion ENDLOCAL

Question

@echo off
setlocal EnableDelayedExpansion
set /a N=0
for /f "tokens=* delims=" %%g in ('dir !FOLDERPATH! /b') do (
setlocal DisableDelayedExpansion
  set "item=%%g"
endlocal
  set /a N+=1
REM next line loses exclamation marks. replacing %%g with %%item%% gives error: not defined $$ variable
  call set "$$%%N%%=%%g"
  )
set "ind=%N%"
REM View the results
echo !ind!
for /f "tokens=* delims=" %%i in ('set $$') do echo %%i
pause
EXIT /b

I read many solutions to my problem (stackoverflow.com/questions/3262287; stackoverflow.com/questions/28682268; stackoverflow.com/questions/29869394; stackoverflow.com/questions/3262287) and am still baffled. Any help appreciated

I have a folder with mp3 filenames containing exclamation marks (and percentage signs and ampersands). The above subroutine is supposed to fill an array=$$ with these filenames. It gets called 2000 times, each time for a different folder.

I want to use EnableDelayedExpansion as the master setlocal (twice as fast). In my entire batch program this is only line (set "item=%%g") where I need DisableDelayedExpansion. I need to know how to efficiently pass this variable (item) across the DisableDelayedExpansion endlocal boundary (and into the $$ set). Alternatively I guess I could fill the $$set within the Disabled environment, and then I need to pass the set across the boundary.

I'm looking for speed. I can use Disabled for the entire subroutine, but this doubles my processing time (for very few exclamation marks and percentage signs).

---

Based on the responses I see it might be good to include the entire batch script (simplified). This script takes 28 minutes to run with my database on my puny machine. It handles exclamation marks, percentage signs, ampersands and anything else you can throw at it.

@echo off
chcp 1254>nul
setlocal DisableDelayedExpansion
set /p COUNT=Select Desired Number of Random Episodes per Album:
for /d %%f in (H:\itunes\Podcasts\*) do (
  set buffer="%%f"
  set /a ind = 0
  call:Set$$Variables
setlocal EnableDelayedExpansion
  if !COUNT! LEQ !ind! ( set DCOUNT=!COUNT! ) ELSE set DCOUNT=!ind!
  for /l %%g in (1, 1, !DCOUNT!) do (
    call:GenerateUniqueRandomNumber
    for %%N in (!num!) do echo !$$%%N!>>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8"
    )
endlocal
  )
pause
Exit /b

:Set$$Variables
set /a N = 0
for /f "tokens=* delims=" %%g in ('dir %buffer% /b') do (
  set "item=%%g"
  set /a N+=1
  call set "$$%%N%%=%%item%%"
  )
set "ind=%N%"
EXIT /b

:GenerateUniqueRandomNumber
:nextone
set /a "num = (((!random! & 1) * 1073741824) + (!random! * 32768) + !random!) %% !ind! + 1"
for %%N in (!num!) do (
  if !RN%%N!==1 (
  goto:nextone
  )
  set "RN%%N=1"
)
EXIT /b

A simple change to the following and it runs in 13 minutes. But it doesn't handle exclamation marks (bangs=!).

@echo off
chcp 1254>nul
setlocal EnableDelayedExpansion
set /p COUNT=Select Desired Number of Random Episodes per Album:
for /d %%f in (H:\itunes\Podcasts\*) do (\
setlocal
  set buffer="%%f"
  set /a ind = 0
  call:Set$$Variables
  if !COUNT! LEQ !ind! ( set DCOUNT=!COUNT! ) ELSE set DCOUNT=!ind!
  for /l %%g in (1, 1, !DCOUNT!) do (
    call:GenerateUniqueRandomNumber
    for %%N in (!num!) do echo !$$%%N!>>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8"
    )
endlocal
  )
pause
Exit /b

:Set$$Variables
set /a N = 0
for /f "tokens=* delims=" %%g in ('dir %buffer% /b') do (
  set "item=%%g"
  set /a N+=1
  call set "$$%%N%%=%%item%%"
  )
set "ind=%N%"
EXIT /b

:GenerateUniqueRandomNumber
:nextone
set /a "num = (((!random! & 1) * 1073741824) + (!random! * 32768) + !random!) %% !ind! + 1"
for %%N in (!num!) do (
  if !RN%%N!==1 (
  goto:nextone
  )
  set "RN%%N=1"
)
EXIT /b

More than double the processing time to handle a very few filenames containing exclamation marks. The resource - time hog is the subroutine Set$$Variables. So my hope is someone out there can find a happy middle ground between these two times: 13 minutes vs. 28 minutes. It seems to me there must be a way to handle the dozen or so affected files in the 15 minute difference.

Answer 1

Best if you truly understand what is slowing down your script.

The following all contribute to unacceptable performance:

• Excessive CALLs in a loop
• Randomly selecting a number until you get one that has not been selected yet
• Redirection in append mode for each file - best to redirect only once

Enabling and disabling delayed expansion normally does not contribute much to bad performance (unless you have a massively large environment space)

My code below uses the FINDSTR technique to quickly build the array, without needing delayed expansion.

I then can enable delayed expansion just once for each folder. I never have to pass any values across the endlocal "barrier"

I guarantee that each random operation selects an available file by building a list of possible numbers, fixed width of 4 with leading spaces. The list must fit in a single variable with max length of ~8190 bytes, so this solution supports up to ~2040 files per folder. Each random number specifies which position to take from the list, and then the value is extracted and the count decremented.

I enclose the entire outer loop in an extra set of parentheses so that I only need to redirect once.

I'm pretty sure this code will be significantly faster than even your 2nd code that does not support ! etc.

benham1.bat

@echo off
chcp 1254>nul
setlocal DisableDelayedExpansion
set /p "maxCnt=Select Desired Number of Random Episodes per Album:"

pushd "H:\itunes\Podcasts"
>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8" (
  for /d %%F in (*) do (
    pushd "%%F"
    set "fileCnt=0"
    for /f "delims=: tokens=1,2" %%A in ('dir /b /a-d 2^>nul^|findstr /n "^"') do (
      set "$$%%A=%%B"
      set "fileCnt=%%A"
    )
    setlocal enableDelayedExpansion
    set "nums="
    for /l %%N in (1 1 !fileCnt!) do (
      set "n=   %%N"
      set "nums=!nums!!n:~-4!"
    )
    if !fileCnt! lss !maxCnt! (set "cnt=!fileCnt!") else set "cnt=!maxCnt!"
    for /l %%N in (1 1 !cnt!) do (
      set /a "pos=(!random!%%fileCnt)*4, next=pos+4, fileCnt-=1"
      for /f "tokens=1,2" %%A in ("!pos! !next!") do (
        for /f %%N in ("!nums:~%%A,4!") do echo !$$%%N!
        set "nums=!nums:~0,%%A!!nums:~%%B!"
      )
    )
    endlocal
    popd
  )
)
popd

Update and solution

It really bothered me that this code behaved so poorly in RKO's hands (see his comment). So I did some tests of my own to figure out what is happening.

I ran the code against 500 folders with 100 files in each folder, and I asked for an output of 50 files for each folder. I modified the loop to print out the name of each folder to stderr so I could monitor progress. As expected, each folder was processed very quickly at the beginning. But as the program progressed, each folder became slower than the previous one, in a non-linear fashion. By the time it reached the end, each folder was painfully slow.

It took ~3 minutes to process 500 folders. I then doubled the number of folders, and the time exploded to ~18 minutes. Very nasty.

Way back when, a group of us at DosTips tried to investigate how cmd.exe manages the environment space, and how large environments impact performance. See Why does SET performance degrade as environment size grows?

We determined that ENDLOCAL does not actually release allocated memory, which causes SET performance to degrade as the number of SET operations accumulates. This problem has lots of SET operations, so it makes sense that it becomes slow.

But RKO has code with lots of inefficiencies that is performing better than mine. In the absence of environment size issues, his code should be much slower. So somehow his code must not be accumulating memory like mine. So I went on a quest to isolate the memory allocation for each folder from all the rest.

My first attempt was to put all the memory allocation for a single folder within a new cmd.exe process. And it worked! Processing 500 folders now took ~1 minute, and doubling to 1000 folders basically doubled the time to ~2 minutes!

benham2.bat

@echo off
if "%~1" equ ":processFolder" (
  pushd "%folder%"
  set "fileCnt=0"
  for /f "delims=: tokens=1,2" %%A in ('dir /b /a-d 2^>nul^|findstr /n "^"') do (
    set "$$%%A=%%B"
    set "fileCnt=%%A"
  )
  setlocal enableDelayedExpansion
  set "nums="
  for /l %%N in (1 1 !fileCnt!) do (
    set "n=   %%N"
    set "nums=!nums!!n:~-4!"
  )
  if !fileCnt! lss !maxCnt! (set "cnt=!fileCnt!") else set "cnt=!maxCnt!"
  for /l %%N in (1 1 !cnt!) do (
    set /a "pos=(!random!%%fileCnt)*4, next=pos+4, fileCnt-=1"
    for /f "tokens=1,2" %%A in ("!pos! !next!") do (
      for /f %%N in ("!nums:~%%A,4!") do echo !$$%%N!
      set "nums=!nums:~0,%%A!!nums:~%%B!"
    )
  )
  popd
  exit
)

setlocal DisableDelayedExpansion
set /p "maxCnt=Select Desired Number of Random Episodes per Album:"

pushd "H:\itunes\Podcasts"
>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8" (
  for /d %%F in (*) do (
    set "folder=%%F"
    cmd /v:off /c ^""%~f0" :processFolder^"
  )
)
popd
exit /b

But then I realized that I didn't have to restart my console for each test of my original benham1 code - when the batch script terminated, the memory seemed to have reset because the next run would start out just as fast as the prior one.

So I thought, why not simply CALL a :subroutine instead of initiating a new cmd.exe. This worked about the same, just a little bit better!

benham3.bat

@echo off
setlocal DisableDelayedExpansion
set /p "maxCnt=Select Desired Number of Random Episodes per Album:"

pushd "H:\itunes\Podcasts"
>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8" (
  for /d %%F in (*) do (
    set "folder=%%F"
    call :go
  )
)
popd
exit /b

:go
setlocal
cd %folder%
set "fileCnt=0"
for /f "delims=: tokens=1,2" %%A in ('dir /b /a-d 2^>nul^|findstr /n "^"') do (
  set "$$%%A=%%B"
  set "fileCnt=%%A"
)
setlocal enableDelayedExpansion
set "nums="
for /l %%N in (1 1 !fileCnt!) do (
  set "n=   %%N"
  set "nums=!nums!!n:~-4!"
)
if !fileCnt! lss !maxCnt! (set "cnt=!fileCnt!") else set "cnt=!maxCnt!"
for /l %%N in (1 1 !cnt!) do (
  set /a "pos=(!random!%%fileCnt)*4, next=pos+4, fileCnt-=1"
  for /f "tokens=1,2" %%A in ("!pos! !next!") do (
    for /f %%N in ("!nums:~%%A,4!") do echo !$$%%N!
    set "nums=!nums:~0,%%A!!nums:~%%B!"
  )
)
exit /b

Another Update

I substituted Aacini's superior Array based method for guaranteeing each random operation selects a unique file name in place of my string based method. It yields slightly better performance:

benham-aacini.bat

@echo off
setlocal DisableDelayedExpansion
set /p "maxCnt=Select Desired Number of Random Episodes per Album:"

pushd "H:\itunes\Podcasts"    
>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8" (
  for /d %%F in (*) do (
    set "folder=%%F"
    call :go
  )
)
popd
exit /b

:go
setlocal
cd "%folder%"
set "fileCnt=0"
for /f "delims=: tokens=1,2" %%A in ('dir /b /a-d 2^>nul^|findstr /n "^"') do (
  set "$$%%A=%%B"
  set /a "fileCnt=%%A, RN%%A=%%A"
)
setlocal enableDelayedExpansion
if !fileCnt! lss !maxCnt! (set end=1) else set /a "end=fileCnt-maxCnt+1"
for /l %%N in (!fileCnt! -1 !end!) do (
  set /a "ran=!random!%%%%N+1"
  set /a "num=RN!ran!, RN!ran!=RN%%N
  for %%N in (!num!) do echo !cd!\!$$%%N!
)
exit /b

Here is a summary of the timings of each version:

folders  | benham1  benham2   benham3   benham-aacini
---------+-------------------------------------------
    500  |    2:49     0:53      0:43       0:41
   1000  |   17:48     1:56      1:44       1:34

So our original thinking at DosTips that the environment never shrinks is wrong. But I haven't had time to fully test and determine exactly when it shrinks.

Regardless, I think I finally have a version that is truly faster than your current 13 minute code :-)

Yet Another Update (Assume few collisions)

Multiple random selections from the complete set of files might result in duplicates. My code assumes that the number of requested files might be a large portion of a large list, in which case you can expect to get many duplicates.

So all of my previous solutions did some extra bookkeeping to guarantee that each random operation results in a unique file.

But RKO seems to typically select just a few files from each folder. So the chance of collision is small. His code just randomly selects a file, and then if the file has been selected before, it loops back and tries again until it finds a new file. But since the chance of collision is small, the retry rarely happens. This method has significantly less bookkeeping. The result is that his random selection algorithm is faster as long as the number requested remains small.

So I have adopted my code to use a slightly modified version of RKO's selection method. I expect this to be the fastest yet for a small request counts.

benham4.bat

@echo off
setlocal DisableDelayedExpansion
set /p "maxCnt=Select Desired Number of Random Episodes per Album:"

pushd "H:\itunes\Podcasts"    
>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8" (
  for /d %%F in (*) do (
    set "folder=%%F"
    call :selectRandom
  )
)
popd
exit /b

:selectRandom
setlocal
cd "%folder%"
set "fileCnt=0"
for /f "delims=: tokens=1,2" %%A in ('dir /b /a-d 2^>nul^|findstr /n "^"') do (
  set "$$%%A=%%B"
  set "fileCnt=%%A"
)
setlocal enableDelayedExpansion
if !fileCnt! lss !maxCnt! (set /a end=fileCnt) else set /a end=maxCnt
for /l %%N in (1 1 !end!) do (
  set /a "N=!random!%%fileCnt+1"
  if not defined $$!N! call :tryAgain
  for %%N in (!N!) do echo !folder!\!$$%%N!
  set "$$!N!="
)
exit /b

:tryAgain
set /a "N=!random!%%fileCnt+1"
if not defined $$!N! goto :tryAgain
exit /b

Answer 2

There isn't any good way to transer many variables out of the scope (over the endlocal barrier).

But when you transfer one by one than it works.

@echo off
setlocal
set "folderPath=C:\temp"
call :func
set $$
exit /b

:func
setlocal EnableDelayedExpansion
FOR /F "delims=" %%F in ("!FOLDERPATH!") DO (
    endlocal
    setlocal DisableDelayedExpansion
    set /a Counter=0
    for /f "tokens=* delims=" %%g in ('dir %%F /b') do (        
        set "item=%%g"
        setlocal EnableDelayedExpansion
        FOR /F "tokens=1,*" %%C in ("!Counter! !item!") DO (

            endlocal
            endlocal
            set "$$%%C=%%D"
            setlocal DisableDelayedExpansion
            set /a Counter=%%C + 1
        )
    )
)

EXIT /b

This solution assumes, that none of your filenames contain a bang ! or that you call your function from a disabled delayed expansion context.

Another solution
When you need a bullet proof variant, then you could replace the endlocal/set "$$%%C=%%D" with the macroReturn technic.

When you can live with a temporary file you could also use the set /p technic.

:collectFiles
dir /b !FOLDERPATH! > temp.$$$
FOR /F %%C in ('type temp.$$$ ^| find /v /c ""') DO (
    echo count %%C
    < temp.$$$ (
        for /L %%n in (0 1 %%C) DO (
            set /p $$%%n=
        )
    )
)
exit /b

Answer 3

This method create the array in a disabled delayed expansion environment in a very fast way:

@echo off
setlocal DisableDelayedExpansion

for /f "tokens=1* delims=:" %%g in ('dir %FOLDERPATH% /b ^| findstr /N "^"') do (
   set "$$%%g=%%h"
   set "ind=%%g"
)

REM View the results
echo %ind%
set $$

REM View the results using DelayedExpansion
setlocal EnableDelayedExpansion
for /L %%i in (1,1,%ind%) do echo %%i- !$$%%i!

pause
EXIT /b

You may also transfer the entire array to the environment of the caller program in a very simple way:

for /F "delims=" %%a in ('set $$') do (
   endlocal
   set "%%a"
)

In this method the endlocal command is executed several times, but it just works the first time when it is matched with the initial setlocal of the function. If this method could release other previous environments, then just add a simple test:

set _FLAG_=1
for /F "delims=" %%a in ('set $$') do (
   if defined _FLAG_ endlocal
   set "%%a"
)

EDIT: Complete example code added

I wrote the code below after the complete example code was posted by the OP; I used the same commands and variable names from the original code. This solution use the method I originally posted here to create the array in a disabled delayed expansion environment (that generate the indices of the elements via findstr /N "^" command), and then extract the elements of the array in random order using a very efficient method that I already used at this answer. I also inserted a couple modifications that increase the efficiency, like avoid call commands and change the append redirection >> (that is executed one time for each output line) by a standard redirection > (that is executed just once). The resulting program should run much faster than the original OP's code.

@echo off
chcp 1254>nul
setlocal DisableDelayedExpansion
set /p COUNT=Select Desired Number of Random Episodes per Album:

(for /d %%f in (H:\itunes\Podcasts\*) do (
   for /f "tokens=1* delims=:" %%g in ('dir "%%f" /b /A-D 2^>NUL ^| findstr /N "^"') do (
      set "$$%%g=%%h"
      set "ind=%%g"
      set "RN%%g=%%g"
   )
   setlocal EnableDelayedExpansion
   if %COUNT% LEQ !ind! (set /A DCOUNT=ind-COUNT+1) ELSE set DCOUNT=1
   for /l %%g in (!ind!, -1, !DCOUNT!) do (
      set /A "ran=(!random!*%%g)/32768+1"
      set /A "num=RN!ran!, RN!ran!=RN%%g"
      for %%N in (!num!) do echo !$$%%N!
   )
   endlocal
)) > "%USERPROFILE%\Desktop\RandomEpisodes.m3u8"

pause
Exit /b

2ND EDIT

After read the explanation of user dbenham about a problem with the environment release in our original methods, I introduced the same modification suggested by him in my code in order to fix the problem. It is expected that both codes now run faster than the original OP's code...

@echo off
chcp 1254>nul
setlocal DisableDelayedExpansion
(
for /F "delims==" %%a in ('set') do set "%%a="
set "ComSpec=%ComSpec%"
set "USERPROFILE=%USERPROFILE%"
)
set /p COUNT=Select Desired Number of Random Episodes per Album:
(for /d %%f in (H:\itunes\Podcasts\*) do call :Sub "%%f"
) > "%USERPROFILE%\Desktop\RandomEpisodes.m3u8"
pause
Exit /b

:Sub
setlocal DisableDelayedExpansion
   cd %1
   for /f "tokens=1* delims=:" %%g in ('dir /b /A-D 2^>NUL ^| findstr /N "^"') do (
      set "$%%g=%%h"
      set /A "ind=%%g, N%%g=%%g"
   )
   setlocal EnableDelayedExpansion
   if %COUNT% LEQ %ind% (set /A DCOUNT=ind-COUNT+1) ELSE set DCOUNT=1
   for /l %%g in (%ind%, -1, %DCOUNT%) do (
      set /A "ran=!random!%%%%g+1"
      set /A "num=N!ran!, N!ran!=N%%g"
      for %%N in (!num!) do echo %CD%\!$%%N!
   )
exit /B

3RD EDIT

The method to generate unique random numbers used by dbenham and me is a general-purpose method that efficiently manage most situations; however, it seems that such method is not best suited for this particular problem. The new code below is an attempt to write a solution for this problem that run in the fastest possible way.

Mod: A small bug have been fixed.

@echo off
chcp 1254>nul
setlocal DisableDelayedExpansion
set "findstr=C:\Windows\System32\findstr.exe"
for /F "delims=" %%a in ('where findstr 2^>NUL') do set "findstr=%%a"
(
for /F "delims==" %%a in ('set') do set "%%a="
set "ComSpec=%ComSpec%"
set "USERPROFILE=%USERPROFILE%"
set "findstr=%findstr%"
)
set /p COUNT=Select Desired Number of Random Episodes per Album:
(for /d %%f in (H:\itunes\Podcasts\*) do call :Sub "%%f"
) > "%USERPROFILE%\Desktop\RandomEpisodes.m3u8"
pause
Exit /b

:Sub
setlocal DisableDelayedExpansion
   cd %1
   for /f "tokens=1* delims=:" %%g in ('dir /b /A-D 2^>NUL ^| "%findstr%" /N "^"') do (
      set "$%%g=%%h"
      set "ind=%%g"
   )
   setlocal EnableDelayedExpansion
   if %COUNT% LEQ %ind% (set "DCOUNT=%COUNT%") ELSE set "DCOUNT=%ind%"
   for /l %%g in (1, 1, %DCOUNT%) do (
      set /A "num=!random!%%%%g+1"
      if not defined $!num! call :nextNum
      for %%N in (!num!) do echo %CD%\!$%%N!& set "$%%N="
   )
exit /B

:nextNum
   set /A num=num%%ind+1
   if not defined $%num% goto nextNum
exit /B

Answer 4

10 Minutes! It's a modified version of the 13 Minute EnabledDelayedExpansion version shown in the original post. I'm waiting for a slight coding fix from dbenham to see if his is an even faster solution.

Here's the critical piece of coding:

for /f "tokens=* delims=" %%g in ('dir "!buffer!" /b') do (
  setlocal DisableDelayedExpansion
  for /f "tokens=1* delims=!" %%m in ("%%g") do if not "%%m"=="%%g" (
    set "item=%%g"
    call set BangFile=%%item:^&=¬%%
    call set BangFile=%%Bangfile:!=^^^^!%%
    call echo %%BangFile%%>"G:\BangFilename.txt"
)
  endlocal

Yes it's ugly. I don't like writing to temporary files, but could find no other way. And again there's probably only a few dozen filenames containing bangs (!) in the entire 120K collection, so very few read-writes. I had to use the above code twice: once for the directory names and once for the filenames.

The full code is here:

@echo off
chcp 1254>nul
setlocal EnableDelayedExpansion
IF EXIST "%USERPROFILE%\Desktop\RandomEpisodes.m3u8" del "%USERPROFILE%\Desktop\RandomEpisodes.m3u8"
set /p COUNT=Select Desired Number of Random Episodes per Album:
call:timestart
for /d %%f in (G:\itunes\Podcasts\* H:\itunes\Podcasts\*) do (

setlocal DisableDelayedExpansion
  if exist g:\BangDirectory.txt del g:\BangDirectory.txt
  for /f "tokens=1* delims=!" %%d in ("%%f") do if not "%%d"=="%%f" (
    set "BangDir=%%f"
    call set BangDir=%%BangDir:^&=¬%%
    call set BangDir=%%BangDir:!=^^^^!%%
    call echo %%BangDir%%>g:\BangDirectory.txt
  )
endlocal

setlocal
  set "buffer=%%f"
  set directory=%%~nf
  call:timecalc

  call:Set$$Variables

  if !COUNT! LEQ !ind! ( set DCOUNT=!COUNT! ) ELSE set DCOUNT=!ind!
  for /l %%g in (1, 1, !DCOUNT!) do (
    call:GenerateUniqueRandomNumber

    for %%N in (!num!) do echo !$$%%N!>>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8"
    )
endlocal
  )
pause
Exit /b

:Set$$Variables
set /a cnt = 0
  if exist g:\BangDirectory.txt for /f "usebackq delims=" %%t in ("G:\BangDirectory.txt") do (
  set "buffer=%%t"
  set "buffer=!buffer:¬=&!"
  )
for /f "tokens=* delims=" %%g in ('dir "!buffer!" /b') do (
  setlocal DisableDelayedExpansion
  if exist g:\BangFilename.txt del g:\BangFilename.txt
  for /f "tokens=1* delims=!" %%m in ("%%g") do if not "%%m"=="%%g" (
    set "item=%%g"
    call set BangFile=%%item:^&=¬%%
    call set BangFile=%%Bangfile:!=^^^^!%%
    call echo %%BangFile%%>"G:\BangFilename.txt"
)
  endlocal

  if exist g:\BangFilename.txt for /f "usebackq tokens=* delims=" %%p in ("G:\BangFilename.txt") do (
  set Filename=%%p
  set "Filename=!Filename:¬=&!"
  set $$!cnt!=!buffer!\!Filename!
  )
  if not exist g:\BangFilename.txt set "$$!cnt!=!buffer!\%%g"
  set /a cnt+=1
)
set "ind=!cnt!"
EXIT /b

:GenerateUniqueRandomNumber
:nextone
set /a "num = (((!random! & 1) * 1073741824) + (!random! * 32768) + !random!) %% !ind!"
for %%N in (!num!) do (
  if !RN%%N!==1 (
  goto:nextone
  )
  set "RN%%N=1"
)
EXIT /b

:timestart
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
   set /A "start=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100")
exit /b

:timecalc
REM Get end time:
for /F "tokens=1-4 delims=:.," %%a in ("%time%") do (
   set /A "end=(((%%a*60)+1%%b %% 100)*60+1%%c %% 100)*100+1%%d %% 100"
)
REM Get elapsed time:
set /A elapsed=end-start
REM Show elapsed time:
set /A hh=elapsed/(60*60*100), rest=elapsed%%(60*60*100), mm=rest/(60*100), rest%%=60*100, ss=rest/100, cc=rest%%100
if %mm% lss 10 set mm=0%mm%
if %ss% lss 10 set ss=0%ss%
set "TimeElapsed=    Time elapsed (mm:ss) %mm%:%ss%"
title %timeelapsed%    WIP: "%directory%"
exit /b

Note on testing. I tested all the versions with virus protection off and with no other running programs. I selected 5 as the Desired Number of Random Episodes. I use an Evo N410c running Win XP. I tried to be as even as possible for each version of the script, but I noticed the 13 minute run can sometimes be as fast as 10 minutes (my guess is XP is creating & keeping some indices on-the-fly which affect runtime).

The 120K item library is contained on an external harddrive connected by USB. The library has about 300 directories with hundreds to thousands of items. It has another 1400 directories with between a few and a few dozen items. This is a live library, but I included an additional testing directory with filenames having every combination and permutation of !, & and %.

Philosophic note. Several times I've had to do 'stuff' with this 120K item library and in the end it has always been the case that avoidance of DisableDelayedExpansion is the best rule. I typically do everything I can with EnabledDelayedExpansion and then take care of any exceptions (bangs) as exceptions.

Any recommendations to reduce the ugliness of this solution (but not its speed) are very welcome.

Postscript--------------------

I incorporated Aacini's and dbenham's random number routine into my 10 minute version. Their coding looked to be much more elegant than my original coding. I deleted my GenerateUniqueRandomNumber subroutine and incorporated the following:

   if !ind! lss !Count! (set end=1) else set /a "end=ind-Count+1"
   for /l %%N in (!ind! -1 !end!) do (
     set /a "ran=!random!%%%%N+1"
     set /a "num=RN!ran!, RN!ran!=RN%%N
     for %%N in (!num!) do echo !$$%%N!>>"%USERPROFILE%\Desktop\RandomEpisodes.m3u8"
   )

This change added 2 minutes to processing time (increased run-time from 10 minutes to 12 minutes). Sometimes elegant just ain't as fast as plain ugly. I'm stickin' with my original.

Answer 5

...
setlocal DisableDelayedExpansion
endlocal&set "item=%%g"
...

should work.

Answer 6

@ECHO OFF
SETLOCAL DISABLEDELAYEDEXPANSION
FOR /f "tokens=1*delims=:" %%a IN (
 'dir /b /ad c:\106x^|findstr /n /r "."') DO (
 SET "thing%%a=%%b"
)
SET thing
GOTO :EOF

This should establish your array. c:\106x is just a directory where I have some strange directory-names.

---

My directory c:\106x:

!dir!
%a
%silly%
-
1 & 2
a silly dirname with & and % and ! an things
exe
flags
nasm32working
nasmxpts
no test file(s)
some test file(s)
stl
wadug
with spaces
with'apostrophes'test

Result of running above code

thing1=!dir!
thing10=nasmxpts
thing11=no test file(s)
thing12=some test file(s)
thing13=stl
thing14=wadug
thing15=with spaces
thing16=with'apostrophes'test
thing2=%a
thing3=%silly%
thing4=-
thing5=1 & 2
thing6=a silly dirname with & and % and ! an things
thing7=exe
thing8=flags
thing9=nasm32working

works for me! - and only executes the setlocal once.