Batch script. Handling strings with CR line endings

Question

I decided to reformat my question due to comments of users (thanks to all) and new knowledge I have got. I'll not open a new post because the main problem remains the same:

How to handle strings produced by some program (e.g. CURL) line-by-line dynamically if those strings don't have LF terminator?

CURL produces each line with interval approx 1 sec. I need to choose only some fields from CURL output for further processing (Current speed, Downloaded bytes, Downloded %), so I need to process each new line that appears. But I can't because each line is ended with CR.

The FOR loop doesn't show anything in CMD window until download is complete, and other users explained to me why:

FOR /F "delims=" %%x in ('curl ... http://some_url 2^>^&1') do echo %%x

So, I am forced to abandon the FOR:

curl ... http://some_url | string_handler.bat

And using JREPL for replacing CR to CRLF on-the-fly:

curl ... http://some_url  2>&1 | jrepl "\r([^\n])" "\r\n$1" /xseq

But this solution produces the hole output, not line-by-line, after CURL is finished because if string doesn't have LF terminator there is nothing to PIPE (thanks @Stephan). Maybe there is the solution to bypass the PIPE behaviour?

For solving problem I've made the simple script - simulation of CURL output. This is countdown timer and it produces every 1 sec a line with CR terminator except first and last lines that have CRLF:

:: bears.bat

@echo off
setlocal enabledelayedexpansion

::Define LF variable containing a linefeed (0x0A)
(set LF=^
%=empty line%
)

::Get a CR character (0x0D)
for /F %%a in ('copy /Z "%~F0" NUL') do set "CR=%%a"

:: First argument is CountDown high level, default 4 sec
if "%1"=="" (set /a high=4) else (set /a high=%1)

echo Hello bears
for /l %%i in (%high%,-1,0) do (
    if %%i gtr 0 (
        <nul set /p="Counter: %%i!CR!" 
    ) else (
        <nul set /p="Counter: %%i!CR!!LF!"
    )
    :: Pause 1 sec
    if %%i gtr 0 ping 127.0.0.1 -n 2 >nul
)
exit /b

Answer 1

If you want the Mac 'CR' ending lines replaced with Windows 'CRLF`, which is what I think you are asking for, then the following method should achieve that:

@(For /F Delims^=^ EOL^= %%G In ('%SystemRoot%\System32\curl.exe
 "http://some_url" 2^>^&1') Do @(Set "LineWithCR=%%G"
 SetLocal EnabledelayedExpansion & Echo !LineWithCR!& EndLocal)) 1>"curl.log"

Quick Explanation:

A For /F command terminates a line at the first occurrence of a 'LF', and if that is preceded by a 'CR', that 'CR' is stripped, but any other 'CR' characters remain. If you expand a For variable containing a 'CR', in this case %%G, that 'CR' will also remain. However if you expand an environment variable, %LineWithCR%, all 'CR' characters will be stripped. However, in order to expand an environment variable containing a 'CR', you would need to use delayed expansion, i.e. !LineWithCR!.

Answer 2

As already said, it can't be solved with a FOR /F loop, because it always wait for the complete output before starting to loop.

But it can be solved with an async pattern and two threads.
You can test it, replace the call slowOutput.bat to call bears.bat

Important: The consumer just reads all available content into the line variable.
If the producer is fast enough, this can lead into fetching two lines (with CR).
With your bears.bat example:
Content of line = Counter: 4<CR>Counter: 3<CR> If your real world curl command produces this problem, you need to split the line at the <CR> character.

@echo off
REM *** Trampoline jump for function calls of the form ex. "C:\:function:\..\MyBatchFile.bat"
FOR /F "tokens=3 delims=:" %%L in ("%~0") DO goto :%%L

REM Create an empty file, this has to exist before the consumer starts
break > async.tmp

REM Start the "producer" thread
start "" /b "cmd /c "%~d0\:producer:\..\%~pnx0"

REM The consumer thread runs parallel to the producer thread
call :consumer
exit /b

:producer
(
  call slowOutput.bat 
  (echo ende)
) > async.tmp
exit /b

:consumer
echo This is the consumer thread
setlocal EnableDelayedExpansion
< async.tmp call :_consumer
exit /b

:_consumer
set "line="
set /p line=

if not defined line goto :_consumer
if "!line!" EQU "ende" exit /b

echo(!line!
goto :_consumer