Iterating a large amout of files with Batch

Question

I wrote a short batch script that iterates through the files of a directory and its subdirectories. In total there are more than a million files. My batch is working as intented if I use it for smaller numbers of files and directories. But if I try to use it for all of them, it just seems to never stop working. My impression is, that the script needs to "check" every file before I get an output. So my question is: Is there a way to get this done faster or at least to test, if the batch is working at all?

Here is my example code:

FOR /F "delims=*" %%i IN ('dir /s /b *.txt') do echo "test"

Thank you in advance!

Answer 1

EDITED to include information discussed in comments

The original answer to this question was

for /r "c:\startingPoint" %%a in (*.txt) do echo %%~fa

which works as intended by the OP: it will recursively process files as they are located in disk, with no wait or pause or at least with no unnecessary pause (of course the first file needs to be found).

What is the difference between the anwswer and the original code

FOR /F "delims=*" %%i IN ('dir /s /b *.txt') do echo "test"

in the question?

In general, for /f is used to iterate over a set of lines instead of a set of files, executing the code in the body of the for command for each of the lines. The in clause of the command defines from "where" to retrieve the set of lines.

This "where" can be a file on disk to be read or a command or set of commands to execute and whose output will be processed. In both cases, all the data is fully retrieved before start processing it. Until all the data is in a memory buffer, the code in the body of the for command is not executed.

And this is where a difference appears.

When a file in disk is read, for /f gets the size of the file and allocates a memory buffer big enough to acomodate the full file in memory, reads the file into the buffer and starts to process the buffer (and of course, you can not use for /f to process a file bigger than free memory)

But when for /f processes a command, it allocates a starting buffer, appends data into it from the stdout stream of the executed command and, when the buffer is full, a new larger buffer is allocated, data from the old buffer is copied to the new buffer and old buffer is discarded. New data is retrieved in the adecuated point of new buffer. And this process is repeated each time the buffer gets full. And this scenario is exacerbated by the fact that the buffer is increased in small amounts.

So, when the data generated by the command is very large, a lot of memory allocation, copy, free is done. And this needs time. For large data, a lot of time.

Summarizing, if for /f is used to process the output of a command and the data to process is large, the time needed to to it will increase exponentially.

How to avoid it? The problem (in this cases) is to retrieve the data from the command, not to process it. So, when the volume of data is really big, instead of the usual for /f %%a in (' command ' ) .... syntax, it is better to execute the command redirecting the output to a temporary file and then use for /f to process the file. The generation of data will need the same amout of time, but the difference in data processing delay can go from hours to seconds or minutes.

Answer 2

Performance: Iterating over all files in a directory and all its subdirectory is not fast. I dont know it for sure but I think that the batch has to check each directory directly on your hard drive. Accessing the hard drive is allways slow. If you want to speed it up, you could use one batch that splits the directories in smaller batches and passes this batches to other scripts who do the real work.

Progress: I dont know the exact answer in terms of syntax but if you use the echo command to display the current File you are iterating you can see if the batch is running.

Answer 3

try like this :

@echo off
FOR /F "delims=*" %%i IN ('dir /s /b *.txt') do (
cls    
echo treating : [%%i]
)
echo Done.....

Answer 4

With no clear idea of what you are actually trying to do and one nominal line of code which does nothing of consequence, it's very difficult to make anything other than a general comment.

If you were to change your batchette to

FOR /F "delims=*" %%i IN ('dir /s /b %1*.txt') do echo "test"

and invoke it with

for %%a in (a b c d...x y z 0 1..9) do start yourlittlabatch %%a

(I'll assume you'd have the sense to realise that d..x means all of the characters d to x - I'll not list them for you - and that you'd also need to include in the list any non-alphameric initial characters in use)

then you'd get 36 processes in parallel, each dealing with a portion of the target structure. This should be quicker if you've got a multiprocessor machine - and obviously, I'm also assuming some regular distribution of your filenames' initial character.

The more information you provide, the fewer assumptions we have to make...

Answer 5

There is a bug in For /f when it has to process many files, and is made worse with long filenames (the total data being parsed as filenames is the overriding factor). It sits and can wait for over an hour simply parsing the list.

The solution is to use the dir command into a file and then use the file in the for /f command.

dir /s /b /a-d *.txt >file.tmp

FOR /F "delims=*" %%i IN (file.tmp) do echo "test"