Counting the files in multiple directories

Question

Beware, long build up and exposition to question and problem. If you want to see the full question first, see the header below.

I have a batch file that I'm writing that is for automatically moving specific files around. Basically, I have a set up where I have two networks, one connected to the internet and one isolated on an internal network. In order to move files between these two networks, users put files in their own folder on a server so that someone can pull those files onto an external drive and physically move them to a machine on the other network and put those files on a similar folder no the other network.

Here's a crappy picture of the folder layout and the process of moving

Here's the batch file (relevant parts) that move the files. This script not only moves the files between folders, but it also generates a log of all the files that were waiting to be moved.

@echo off

rem Copies the contents of the external drive to the "AlreadyMoved" folders
dir /b /s /a:d "\\SERVER\Path\Users\*AlreadyMoved" | for /f "delims=\; tokens=3,4,5*" %%a in ('findstr AlreadyMoved') do @xcopy /i /s /y "E:\%%b" "\\SERVER\Path\%%a\%%b\%%c\"    

set folder="E:\"
cd /d %folder%

IF EXIST "E:\" (
goto :DRIVECLEAN
) ELSE (
echo ERROR: "%folder%" DRIVE NOT CONNECTED!
echo(
echo(
echo PLEASE CONNECT %folder% DRIVE AND TRY AGAIN.
exit /b)

:DRIVECLEAN
set folder="E:\"
rem echo This will wipe the %folder% drive clean!
cd /d %folder%
for /F "delims=" %%i in ('dir /b') do (rmdir "%%i" /s/q || del "%%i" /s/q)

rem gets the current date and time in 24 hr format for the text file name.
set hh=%time:~-11,2%
set /a hh=%hh%+100
set hh=%hh:~1%
set dateseed=%date:~10,4%_%date:~4,2%_%date:~7,2%_%hh%%time:~3,2%

set output="%userprofile%\Desktop\Folder\%dateseed%.txt"

rem creates contents of the log file
echo File Header >> %output%
echo( >> %output%
echo Scanned By: %username% >> %output%
echo Date Scanned: %mydate% >> %output%
echo Time Scanned: %mytime% >> %output%
echo( >> %output%

rem echoes the File Name, Size, and Last Modified date in the log file
for /f "delims=" %%F in (
    'dir /b /s \\SERVER\Path\Users ^| findstr ToBeMoved\'
) do for /f "delims=\; tokens=3,4,5*" %%a in ("%%F") do echo %%b: %%d -- Size: %%~zF bytes -- Last Modified: %%~tF >> %output%

rem copies the contents of the "ToBeMoved" folders to the External Drive
dir /b /s /a:d "\\SERVER\Path\Users\*ToBeMoved" | for /f "delims=\; tokens=3,4,5*" %%a in ('findstr ToBeMoved') do @xcopy /i /s /y "\\SERVER\Path\%%a\%%b\%%c" "E:\%%b\"

pause

All this results in a log file that looks something like this:

File Header 

Scanned By: username
Date Scanned: 12-08-2016 
Time Scanned: 01:03 PM 

Last, First1: FileName1.ext -- 10000 bytes -- Last Modified: 12/08/16 2:50 PM
Last, First1: FileName2.ext -- 10000 bytes -- Last Modified: 12/08/16 2:55 PM
Last, First2: FileName3.ext -- 10000 bytes -- Last Modified: 12/08/16 2:50 PM
Last, First2: FileName4.ext -- 10000 bytes -- Last Modified: 12/08/16 2:55 PM

QUESTION STARTS HERE!

So after that extremely long winded build up, here's the question.

I want to get the text file to look like this:

File Header

Moved By: username
Date Moved: 12-08-2016 
Time Moved: 01:03 PM 

Last, First1
FileName1.ext -- 9 KB -- Last Modified: 12/08/16 2:50 PM
FileName2.ext -- 9 KB -- Last Modified: 12/08/16 2:55 PM
Total Moved: 2 Files

Last, First2
FileName3.ext -- 9 KB -- Last Modified: 12/08/16 2:50 PM
FileName4.ext -- 9 KB -- Last Modified: 12/08/16 2:55 PM
FileName4.ext -- 9 KB -- Last Modified: 12/08/16 2:56 PM
Total Moved: 3 Files

You'll notice there are a few differences with this one and the previous text file but the key difference that I'm trying to figure out is how to count the number of files in a particular user's "ToBeMoved" folder.

I've noticed there's no obvious counting function in batch but I've come across a few methods for counting files in a directory but I can't seem to figure out how to implement those into this script and it actually not blow up.

The obvious solution in my mind would be to implement the count inside the same loop that outputs the contents of the log file. Otherwise, I don't know how I would keep all the user's information together. That said, the loop I have has already taken me forever to cobble together (I'm no expert in batch) and I haven't the faintest clue how to add a complex addition to these loops.

Because see there's an added bonus that (this is not entirely relevant to the scope of this question but I wouldn't object if anyone offered assistance to these issues as well. I'm still researching them and if I can't figure it out, I'll probably have to make separate questions for them.) I need to be able to do some conversions of the size value and add line breaks.

Can anyone think of how I could achieve this? Or at least point me in the right direction?

EDIT: This is in response to elzooilogico's answer. I implemented the changes he suggested and the code executed without any errors but the text file isn't formatted quite right. I put some test files into some folders to see what should happen.

I should have gotten something like this:

File Header

Moved By: username
Date Moved: 12-09-2016
Time Moved: 10:30 AM

Last,First1
FileName1.ext -- Size: 1 KB -- Last Modified: 12/09/2016 10:25 AM
FileName2.ext -- Size: 1 KB -- Last Modified: 12/09/2016 09:28 AM 
FileName3.ext -- Size: 1 KB -- Last Modified: 12/09/2016 09:29 AM 
FileName4.ext -- Size: 1 KB -- Last Modified: 12/09/2016 09:31 AM
Total Moved: 4 Files

Last,First2
FileName5.ext -- Size: 1 KB -- Last Modified: 12/09/2016 09:31 AM 
FileName6.ext -- Size: 1 KB -- Last Modified: 12/09/2016 09:37 AM 
FileName7.ext -- Size: 1 KB -- Last Modified: 12/09/2016 09:42 AM
Total Moved: 3 Files

Instead it looked like this:

File Header

Moved By: username
Date Moved: 12-09-2016
Time Moved: 10:30 AM

Last,First1
Last,First1
Last,First1
Last,First1
Last,First2
Last,First2
Last,First2
FileName1.ext -- Size: 103 bytes -- Last Modified: 12/09/2016 10:25 AM
FileName2.ext -- Size: 463 bytes -- Last Modified: 12/09/2016 09:28 AM 
FileName3.ext -- Size: 463 bytes -- Last Modified: 12/09/2016 09:29 AM 
FileName4.ext -- Size: 531 bytes -- Last Modified: 12/09/2016 09:31 AM 
FileName5.ext -- Size: 531 bytes -- Last Modified: 12/09/2016 09:31 AM 
FileName6.ext -- Size: 527 bytes -- Last Modified: 12/09/2016 09:37 AM 
FileName7.ext -- Size: 527 bytes -- Last Modified: 12/09/2016 09:42 AM

Now it is possible that I somehow implemented it incorrectly because the loop he wrote is definitely above my skill level. But I copied it directly into my code and corrected the generic bits where necessary.

FINAL EDIT: I must have put it in wrong the first time because now it works perfectly!

score 1 · Accepted Answer · edited May 23 '17 at 12:09

Not tested

If your batch logic works ok, try changing from the rem creates contents of the log file line to

rem creates contents of the log file
>>"%output%" (
  echo(File Header
  echo(
  echo(Scanned By: %username%
  echo(Date Scanned: %mydate%
  echo(Time Scanned: %mytime%
  echo(
)

setlocal EnableDelayedExpansion 
set "footer="
set "current="
set/a counter=0

rem echoes the File Name, Size, and Last Modified date in the log file
>>"%output%" (
  for /f "delims=" %%F in (
    'dir /b /s \\SERVER\Path\Users ^| findstr ToBeMoved\'
  ) do (
    for /f "delims=\; tokens=3,4,5*" %%a in ("%%F") do (
      if "!current!" neq "%%b" (
         set "current=%%b"
         if defined footer echo(Total Moved: !counter! Files & echo(
         set/a counter=0, footer=1
         echo(
         echo(%%b
      )
      set/a counter+=1, size=%%~zF, kbSize=size/1024
      echo %%d -- Size: !kbSize! Kb -- Last Modified: %%~tF
    )
  )
  if defined footer echo(Total Moved: !counter! Files
)  
EndLocal

rem copies the contents of the "ToBeMoved" folders to the External Drive
dir /b /s /a:d "\\SERVER\Path\Users\*ToBeMoved" | for /f "delims=\; tokens=3,4,5*" %%a in ('findstr ToBeMoved') do @xcopy /i /s /y "\\SERVER\Path\%%a\%%b\%%c" "E:\%%b\"

pause

Note that

>>"%output%" (
   for ... do (
     echo(...
     echo(...
   )
)

Opens the file only once, whereas

   for ... do (
     echo(...>>"%output%"
     echo(...>>"%output%"
   )

Opens and closes the file any time >>"%output%" redirection is used, so first one speeds up the block.

EDIT: brief explanation

BTW

>>"%output%" (
   for ... do (
     echo(...
     echo(...
   )
)

is the same as

(
   for ... do (
     echo(...
     echo(...
   )
)>>"%output%"

EXPLANATION

First: the header is echoed to the output file in one opening-closing cycle as explained above.

Remember: when the command processor finds a block (anything between parentheses), parses it completely and expand variables to the value they have when the block is evaluated. If you update a variable value within a block, you need to enable delayed expansion for the variable to reflect changes made. There's a nice explanation on How does the Windows Command Interpreter (CMD.EXE) parse scripts?

So, we enter delayed expansion mode with the setlocal EnableDelayedExpansion for we are going to use variables updated within the block, this means we must change variable expansion syntax from %var% to !var! to make delayed expansion works.

setlocal EnableDelayedExpansion 
set "footer="
set "current="
set/a counter=0

In this moment fotter and current are undefined. That's the purpose of set "fotter=" and set "current=" and we initialize counter to 0.

Now we are prepared to enter the main for loop,

for /f "delims=" %%F in (...) do (
  for /f "delims=\; tokens=3,4,5*" %%a in ("%%F") do (
    if "!current!" neq "%%b" (
      set "current=%%b"
      if defined footer echo(Total Moved: !counter! Files & echo(
      set/a counter=0, footer=1
      echo(
      echo(%%b
    )
    set/a counter+=1, size=%%~zF, kbSize=size/1024
    echo %%d -- Size: !kbSize! Kb -- Last Modified: %%~tF
  )
)
if defined footer echo(Total Moved: !counter! Files

The if test the content of current var with the content of the second token given by the second for loop. This block will be executed only when current changes, and remember that was undefined, so first time is executed too. Every time it changes its value, means that token has changed (so, in this case is a folder change) and

we update the value of current
then if defined footer prints the footer (the content of the counter). Again first time footer is undefined, so no footer is printed.
Then the counter is reset, and footer is defined, so next time it will be printed.
Then the name of the folder is printed.

After the if block lines will be executed every for iteration

first line, updates the counter and calculate the Kb size taking the filesize.
second one, flush the info of the current action to the output file

The last if defined footer echo(Total Moved: !counter! Files after the for block ensures that the last footer is printed, as the if block won't be executed the last for iteration, leaving last foder contents without footer.

Hope it helps!

NEW EDIT: Think I need more coffee in the evenings...

BTW, if your comparisions are not case sentitive you may use if /I "!current!" neq "%%b" instead of if "!current!" neq "%%b" (case sensitive)

Okay this is brilliant and definitely above my skill level so hopefully I haven't simply poorly implemented it. I replaced the original block of code that generates the username, filename, size, and date modified with the bit you added. However, it didn't quite get what I was trying for though it did execute without error. I'm going to make an edit for what it output. Also, I loved the bit you changed with the output only being written once. I didn't know you could do that and I'm definitely going to use that method from here on. — Neal, Dec 09 '16 at 16:34
not in front of a computer, but if your previous script run ok, this should do the job. I suggest placing an *echo on* before the changed block of code, *rem* the redirection line and the closing matching parenthesis, and put a pause in the for, so you can see what's going on on every *for* iteration. — elzooilogico, Dec 09 '16 at 16:47
I just tried putting it in again. I must have messed up something because now it works perfectly! Thank you! You are basically batch Jesus right now. — Neal, Dec 09 '16 at 16:55
I think first time you forget _setlocal enabledelayedecpansion_ When command processor process blocks (anything between parentheses) it expands block variables only once _before executing the block_. So if a var is updated and its new value needed within the block, we need _delayedexpansion_ . In this mode vars are expanded inline. BTW you need to change %var% to !var!. — elzooilogico, Dec 09 '16 at 17:17

Counting the files in multiple directories

QUESTION STARTS HERE!

1 Answers1