delim command only reading parts of a txt file

Question

I'm a newbie to programming.

I have a txt fileA containing the list of files within a directory.

Scenario: I'm running the delim command to remove new line special character codes and replace with commas from the list of jpg file names (cicra 10000) in the txt fileA and output the result to output fileB.

Problem: Only a random amount of file names reach the output file! Some or many are missing.

Here's the code...

========================================

dir /o /b > fileA.txt
set "txt="
for /f "delims=" %%a in (fileA.txt) do (
  set "txt=!txt!%%a,"
)
set "txt=!txt:~0,-1!"
>fileB.txt echo !txt!

=========================================

Any help appreciated :) Thank you

There is no `delim` command, it's the [`for`](http://ss64.com/nt/for.html)`/F` command, using the `delims` option. Anyway, note that strings are limited to a length of about 8190 characters/bytes; and strings beginning with `;` are ignored by `for /F` as this character is the default `eol` option... — aschipfl, Nov 14 '17 at 10:33
Thank you aschipl1 - that explains the 8kb output file then! — Bert Read, Nov 14 '17 at 12:14
Push me on the right direction please, What's the outline of what I need to do instead? — Bert Read, Nov 14 '17 at 12:16

Mofi · Answer 1 · 2018-07-19T10:35:14.597

Read the Microsoft support article Command prompt (Cmd. exe) command-line string limitation which explains why not all file names are appended to the environment variable txt on such a large list of file names.

One workaround solution is writing the file names into a text file and use tools to replace all carriage return + line-feed pairs by a comma.

Here are two solutions using this workaround from answers on
How can you find and replace text in a file using the Windows command-line environment?

@echo off
rem Replace using very fast Xchang32.exe written by Clay Ruth.
dir *.jpg *.jpeg /A-D /B /ON >ImageList1.txt
"%~dp0Xchang32.exe" ImageList1.txt "^x0D^x0A" ",," >nul

rem Replace using a bit slower jrepl.bat written by Dave Benham.
dir *.jpg *.jpeg /A-D /B /ON >ImageList2.txt
call "%~dp0jrepl.bat" "\r\n" "," /M /F ImageList2.txt /O -

Xchang32.exe written by Clay Ruth or jrepl.bat written by Dave Benham must be in same directory as the batch file.

Note 1:
The contents of the image list files end on both solutions with a comma because last line output by DIR also ends with carriage return + line-feed.

I tried also following command line:

call "%~dp0jrepl.bat" "\r\n(?=.)" "," /M /F ImageList2.txt /O -

The regular expression search string \r\n(?=.) with replace string , is working in text editors supporting regular expressions in PRE syntax which would not replace the last carriage return + line-feed in file because the positive lookahead for any character except newline character is not true on last CR+LF, but that is not working with jrepl.bat using Microsoft's JScript (at least on Windows XP).

Note 2:
On not using *.jpg *.jpeg on DIR command line it is recommended to create the image list file not in the same directory as DIR is searching for files of any name because in this case the list file would be also included in the list.

Important note: Clay Ruth, author of Xchang32.exe and owner of domain clayruth.com died years ago. For that reason the ZIP file with this tool is not available anymore in world wide web although it was permitted explicitly to distribute this package free of charge according to read me file in the ZIP file.

Thanks Mofi I've tried this command structure and this seems to work but does take a long time to process (10 minutes!). It outputs one line at a time, so doesn't stress out the 8kb max. break>output.txt for /f "delims=" %%a in (JPGFileList.txt) do ( set txt=%%a, set txt=!txt:~0,-1! > output.txt ) This will do for now until I have learnt a bit more about coding. :) Thanks again. — Bert Read, Nov 15 '17 at 10:45

score 0 · Answer 2 · answered Nov 15 '17 at 18:36

The problem is that you are collecting all data in a single string, so you quickly reach the limit of about 8190 characters/bytes for strings in the Windows command prompt cmd.

One way to work around that is to use set/P, which is intended to prompt the user for the value of a variable, using a display message; this message is output without any trailing line-breaks:

> "fileA.txt" dir /O /B
> "fileB.txt" < nul (
    set "FLAG=#"
    for /F "usebackq delims= eol=|" %%a in ("fileA.txt") do @(
        if defined FLAG (
            set /P ="%%a"
            set "FLAG="
        ) else (
            set /P =",%%a"
        )
    )
)

Actually you do not need the interim file fileA.txt:

> "fileB.txt" < nul (
    set "FLAG=#"
    for /F "delims= eol=|" %%a in ('dir /O /B') do @(
        if defined FLAG (
            set /P ="%%a"
            set "FLAG="
        ) else (
            set /P =",%%a"
        )
    )
)

However, there are the following restrictions:

a single item must not contain more than 1021 characters/bytes (this is a limitation of set /P);
leading white-spaces (and also some trailing ones) become lost;
no string must begin with an = sign (a syntax error arises in case);

delim command only reading parts of a txt file

2 Answers2