Is calling FOR with "tokens=*" or "delims=" functionally equivalent?

Question

Consider the following two loops:

for /f "tokens=*" %%a in ('dir /b %TEMP%') do (
   echo %%a
)

and

for /f "delims=" %%a in ('dir /b %TEMP%') do (
   echo %%a
)

If my desired result is to provide the variable %%a with the contents of the entire line being evaluated, are the options tokens=* and delims= functionally equivalent?

Are there any situations where I might get a different output with one or the other given a particular input?

I would like to know if these two options should always be used in combination to guarantee coverage of all potential cases or if specifying both is redundant.

In my experience yes. Sometimes I just put both in there at the same time. I normally just use `DELIMS=` — Squashman, Jun 24 '18 at 01:34
Switch to PowerShell and you won't have to worry about the esoteric quirks in `cmd.exe` syntax. — Bill_Stewart, Jun 24 '18 at 02:08
Well if we're posting ridiculous ideas that OP didn't ask for, you might as well learn Python or C. — SomethingDark, Jun 24 '18 at 03:28
`"delims="` means "the entire line". `"tokens=*"` means "all the tokens", but the spaces _before the first token_ are _not_ included in this case, so the result is different when the line have leading spaces. — Aacini, Jun 24 '18 at 05:18
@Bill_Stewart: If you post a non-requested PowerShell comment in a Batch file question (with some criticism against Batch), then I have the right to post a counter-criticism on PowerShell. I invite you to read _the comments_ at end of [this article](http://www.itprotoday.com/powershell/break-your-batch-habit-and-move-powershell) written by Batch-file users about their experiences on PowerShell, or the interesting article [What I Hate About PowerShell](https://helgeklein.com/blog/2014/11/hate-powershell/) written by an experienced programmer. — Aacini, Jun 24 '18 at 12:39
You may read my point of view about _non-requested_ PS solutions on Batch file questions [here](https://stackoverflow.com/questions/42603119/arithmetic-operations-with-hhmmss-times-in-batch-file?noredirect=1&lq=1#comment72493443_42603273); for example: _"Yes: PowerShell is more (put any good adjetive you want here) that Batch files, but this don't means it is the right choice in all cases, particularly when the problem to solve is simple, like this one"_. — Aacini, Jun 24 '18 at 12:39
My comment was not intended to antagonize. I was simply pointing out that when someone asks what kind of kitchen knife is best for cutting tree branches, you might consider that it may be useful to inform them that there are better tools available. — Bill_Stewart, Jun 24 '18 at 14:55
@Bill_Stewart: I don't see how a simple question about a specific detail on a `cmd.exe` command could be mistaken for a request of _"what kind of kitchen knife is best for cutting tree branches"_... **`(?)`** PS fanatics insist that PS is better than Batch _in all cases_ (even the most simplest ones) and ignore the huge amount of examples in contrary... — Aacini, Jun 25 '18 at 20:48

aschipfl · Accepted Answer · 2018-06-25T19:42:39.397

Yes, tokens=* and delims= are different:

delims= returns the whole line unedited;
tokens=* returns the line with any leading delimiters (SPACE and TAB as per default) removed;
tokens=* delims= behaves exactly the same way as delims=;

Note that empty lines are always skipped. Also lines that begin with ; are ignored as that character is the default eol.

If tokens=* is specified and a line contains delimiters only, the for /F loop iterates and the meta variable returns an empty string. As soon as any token number is provided (like tokens=3, tokens=1,3, tokens=2-3, tokens=2*, etc.), delimiter-only lines are skipped.
However, a line that contains one or more delimiters plus the eol character plus an arbitrary string are ignored even when tokens=* is provided.

For evidence I did some tests, using the following text file sample.txt (note that the 2^nd line is empty, the 4^th line contains four SPACEs; click on the edit button below this answer and view the raw text):

text_without_delimiters

text with delimiters

  text with leading and trailing delimiters  
; comment text
  ; comment text with leading delimiters
text plus ; comment text

And here is what I did on the console together with the respective return strings:

>>> for /F %I in (sample.txt) do @echo "%I"
"text_without_delimiters"
"text"
"text"
"text"

>>> for /F "tokens=*" %I in (sample.txt) do @echo "%I"
"text_without_delimiters"
"text with delimiters"
""
"text with leading and trailing delimiters  "
"text plus ; comment text"

>>> for /F "delims=" %I in (sample.txt) do @echo "%I"
"text_without_delimiters"
"text with delimiters"
"    "
"  text with leading and trailing delimiters  "
"  ; comment text with leading delimiters"
"text plus ; comment text"

>>> for /F "tokens=* delims=" %I in (sample.txt) do @echo "%I"
"text_without_delimiters"
"text with delimiters"
"    "
"  text with leading and trailing delimiters  "
"  ; comment text with leading delimiters"
"text plus ; comment text"

>>> for /F "tokens=3" %I in (sample.txt) do @echo "%I"
"delimiters"
"leading"
";"

>>> for /F "tokens=1,3" %I in (sample.txt) do @echo "%I"    "%J"
"text_without_delimiters"    ""
"text"    "delimiters"
"text"    "leading"
"text"    ";"

>>> for /F "tokens=2-3" %I in (sample.txt) do @echo "%I"    "%J"
"with"    "delimiters"
"with"    "leading"
"plus"    ";"

>>> for /F "tokens=2*" %I in (sample.txt) do @echo "%I"    "%J"
"with"    "delimiters"
"with"    "leading and trailing delimiters  "
"plus"    "; comment text"

So the only really strange and unexpected output is the line "" with the tokens=* option alone.

Is calling FOR with "tokens=*" or "delims=" functionally equivalent?

1 Answers1