Pipe in for loop breaks double quoted variables

Question

Situation: Using batchscript to retrieve certain values from a JSON.
I've got the following batchscript:

@ECHO off

ECHO Enter npo.nl program-url :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('curl.exe -s http://e.omroep.nl/metadata/aflevering/%%A ^| jq.exe -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO (
        FOR /F "delims=" %%C IN ('ECHO %%B ^| jq.exe -r .start') DO SET ss=%%C
        FOR /F "delims=" %%C IN ('ECHO %%B ^| jq.exe -r .eind') DO SET to=%%C
        FOR /F "delims=" %%C IN ('ECHO %%B ^| jq.exe -r .tijdsduur') DO SET t=%%C
        ECHO Start:    !ss!
        ECHO End:      !to!
        ECHO Duration: !t!
    )
)
ENDLOCAL
PAUSE

What does it do?
Upon having entered a npo.nl program-url, the first for-loop strips the url down to the prid:POMS_VPRO_850040. In the second for-loop curl.exe retrieves the JSON...:

parseMetadata({"STATUS":"OK","VERSION":"1.11.12","prid":"VPWON_1232766","titel":"Schuim & As","aflevering_titel":"","info":"Schuim & As met Jelle Brandt Corstius","ratio":"16:9","medium":"tv","gidsdatum":"2015-05-03","tijdsduur":"00:05:27","start":"00:23:13","eind":"00:28:40","url":"","webcast":1,"images":[{"size":"640x480","ratio":"4:3","url":"http:\/\/images.poms.omroep.nl\/image\/sx480\/c640x480\/606030.jpg"},{"size":"720x405","ratio":"16:9","url":"http:\/\/images.poms.omroep.nl\/image\/sx405\/c720x405\/606030.jpg"}],"omroepen":[{"naam":"VPRO"}],"pubopties":["adaptive","h264_bb","h264_sb","h264_std"],"tt888":"ja","serie":{"srid":"VPWON_1232748","serie_titel":"Buitenhof","serie_url":null},"sitestat":{"baseurl":"http:\/\/b.scorecardresearch.com\/p?c1=2&c2=17827132&ns_site=po-totaal","programurl":"uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","programurlpost":"category=uitzendinggemist&thema=informatief&po_source=video","baseurl_subtitle":"http:\/\/nl.sitestat.com\/klo\/po\/s","subtitleurl":"uitzendinggemist.publiekeomroep.ondemand.tv.player.tt888.buitenhof","subtitleurlpost":"category=uitzendinggemist&po_source=video&po_sitetype=webonly"},"reclame":"http:\/\/pubads.g.doubleclick.net\/gampad\/ads?_cookie_&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&sz=_sz_&correlator=_correlator_&iu=\/9233\/_site_\/buitenhof&url=_url_&cust_params=genre%3Dinformatief%2Cnieuws%2Factualiteiten%26dur%3D3284%26prid%3DVPWON_1232766%26srid%3DVPWON_1232748%26player%3D_player_","streamSense":{"episode":"buitenhof","program":"buitenhof","station":"nederland_1","sitestatname":"uitzendinggemist.publiekeomroep.ondemand.tv.buitenhof.20150503","sko":"TRUE","sko_dt":"20150503","sko_pr":"buitenhof","sko_stid":"1","sko_ty":"tv.seg","sko_prid":"vpwon1232766","sko_t":"1210","sko_cl":"3284"}})
//epc

...and sends it through a pipe to jq.exe which removes the non-JSON-data parseMetadata( and ) //epc and leaves the single line intact. This is for 2 reasons: 1) with non-JSON-data present we can't process anything, and 2) for-loops process only 1 line at a time.
The subsequent jq.exe's retrieve values for the specified objects without double quotes.
As long as curl.exe and jq.exe are in the same directory as the batchscript, or in the %path%-variable, this is working all fine:

Start:     00:23:13
End:       00:28:40
Duration:  00:05:27

Now I want to call curl.exe and jq.exe from another map. One with spaces in it:

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq.exe"

@ECHO off

ECHO Enter npo.nl program-url :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('%curl% -s http://e.omroep.nl/metadata/aflevering/%%A ^| %jq% -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO (
        FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -r .start') DO SET ss=%%C
        FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -r .eind') DO SET to=%%C
        FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -r .tijdsduur') DO SET t=%%C
        ECHO Start:    !ss!
        ECHO End:      !to!
        ECHO Duration: !t!
    )
)
ENDLOCAL
PAUSE

For the 2nd for-loop this causes problems:

'C:\map' is not recognized as an internal or external command,
operable program or batch file.

While 'ECHO %%X ^| %jq%' does work, it seems '%curl% ^| %jq%' doesn't. So for some reason things go wrong as soon as 2 variables in a pipe are parsed.

Well, no more pipe then:

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq.exe"

@ECHO off

ECHO Enter npo.nl program-url :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('%curl% -s http://e.omroep.nl/metadata/aflevering/%%A') DO (
        FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO (
            FOR /F "delims=" %%D IN ('ECHO %%C ^| %jq% -r .start') DO SET ss=%%D
            FOR /F "delims=" %%D IN ('ECHO %%C ^| %jq% -r .eind') DO SET to=%%D
            FOR /F "delims=" %%D IN ('ECHO %%C ^| %jq% -r .tijdsduur') DO SET t=%%D
            ECHO Start:    !ss!
            ECHO End:      !to!
            ECHO Duration: !t!
        )
    )
)
ENDLOCAL
PAUSE

Now curl.exe and jq.exe each in a for-loop. At first this seems to work fine. The 3 values are echoed, but then things go wrong:

parse error: Invalid numeric literal at line 1, column 5

parse error: Invalid numeric literal at line 1, column 5
parse error: Invalid numeric literal at line 1, column 5
parse error: Invalid numeric literal at line 1, column 5
Start:    00:23:13
End:      00:28:40
Duration: 00:05:27

Like I said before; for-loops parse and process only 1 line at a time. The non-JSON-data //epc on the 2nd line causes the for-loop to start over, which goes horribly wrong as you can see. That's the reason for the pipe between curl and jq in the code above. To output 1 single line to process. Sadly that didn't work either...sigh.

Of course using temporary files is a last resort when curl and jq are still in a map with spaces in it, but I prefer to use variables, so I'm trying to solve the pipe-issue. I've tried 'usebackq' in the for-loop using backticks around the command instead of single-quotes for instance, but to no avail.
So far I haven't found a solution. Does anyone have an explanation for this behaviour and how to solve it?

Answer 1

Thanks to Dave Benham's answer on a related issue I've found the solution!
It appeared to be a specific FOR /F bug in WinXP, and guess what, here I'm still on WinXP. To fix the main offender, the curl-pipe-jq-for-loop, I had to put ^" in front of and after the entire command. Thus the entire batchscript, which I've also further improved:

@ECHO off
CLS

:: NPO JSON-extractor geschreven door Reino Wijnsma, 2015 (reino@degeelebosch.nl)

SET batchname=NPO JSON-extractor
SET version=1.1
TITLE %batchname% %version%

SET curl="C:\map with spaces\curl.exe"
SET jq="C:\map with spaces\jq-1.5rc1.exe"

:Check
IF EXIST %curl% (
    IF EXIST %jq% (
        GOTO Input
    ) ELSE (
        ECHO 'jq.exe' niet gevonden.
        ECHO.
        PAUSE
        GOTO :eof
    )
    GOTO Input
) ELSE (
    ECHO 'curl.exe' niet gevonden.
    ECHO.
    PAUSE
    GOTO :eof
)

:Input
ECHO Voer npo.nl programmalink in :
SET url=
SET /P url=
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 bijvoorbeeld
IF "%url%"=="" GOTO :eof

SETLOCAL ENABLEDELAYEDEXPANSION
FOR %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('^"%curl% -s http://e.omroep.nl/metadata/aflevering/%%~nxA ^| %jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"^"') DO (
        ECHO.
        ECHO JSON:
        FOR /F "delims=" %%C IN ('ECHO %%B ^| %jq% .') DO ECHO %%C
        ECHO.
        FOR /F "tokens=1-3" %%C IN ('ECHO %%B ^| %jq% -r "[.tijdsduur,.start,.eind] | @tsv"') DO (
            ECHO Tijdsduur: %%C
            IF NOT "%%D"=="" (
                SET ss=%%D
                SET to=%%E
                SET /A "_ss=((1!ss:~0,2!-100)*3600)+((1!ss:~3,2!-100)*60)+(1!ss:~6,2!-100)"
                SET /A "_to=((1!to:~0,2!-100)*3600)+((1!to:~3,2!-100)*60)+(1!to:~6,2!-100)"
                ECHO Start:     %%D (!_ss!s^)
                ECHO Einde:     %%E (!_to!s^)
            )
        )
    )
)
ECHO.
ENDLOCAL
GOTO Input

Important notes for future reference:

jq-syntax:           jq -R -r -s  '.[1+index("("):   rindex(")")]'
cmd-shell:           jq -R -r -s  ".[1+index(\"(\"): rindex(\")\")]"
for-loop:           'jq -R -r -s  ".[1+index(\"(\"): rindex(\"^)\")]"'
for-loop (path): '^"%jq% -R -r -s ".[1+index(\"(\"): rindex(\"^)\")]"^"'

- For the cmd-shell you have to escape the double quotes with a line-feed \.
- While 2 of the closing parenthesis are part of jq's syntax, the 1 between double quotes is not. So when in a for-loop, to prevent this for-loop from closing, you'd have to escape this one with a ^.
- When jq's executable path is put in a variable with double quotes, to circumvent a WinXP bug, you'd then also have to put the entire command between ^", because the parenthesis are now considered special characters! This workaround is compatible with Vista and beyond. (See also DosTips.com)

Answer 2

Without being able to test it, I suggest to try your batch code as written below:

@ECHO off
SET "curl=C:\map with spaces\curl.exe"
SET "jq=C:\map with spaces\jq.exe"
ECHO Enter npo.nl program-url :
SET "url="
SET /P "url="
:: http://www.npo.nl/buitenhof/03-05-2015/VPWON_1232766/POMS_VPRO_850040 for example
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /F "tokens=6 delims=/" %%A IN ("%url%") DO (
    FOR /F "delims=" %%B IN ('"%curl%" -s http://e.omroep.nl/metadata/aflevering/%%A') DO (
        FOR /F "delims=" %%C IN ('ECHO %%B ^| "%jq%" -R -r -s ".[1+index(\"^(\"): rindex(\"^)\")]"') DO (
            FOR /F "delims=" %%D IN ('ECHO %%C ^| "%jq%" -r .start') DO SET "ss=%%D"
            FOR /F "delims=" %%D IN ('ECHO %%C ^| "%jq%" -r .eind') DO SET "to=%%D"
            FOR /F "delims=" %%D IN ('ECHO %%C ^| "%jq%" -r .tijdsduur') DO SET "t=%%D"
            ECHO Start:    !ss!
            ECHO End:      !to!
            ECHO Duration: !t!
        )
    )
)
ENDLOCAL
PAUSE

The difference to your code is the position of the double quotes on assigning a string to an environment variable.

The command set is usually used with the parameter variable=value whereby this entire string is the parameter.

The command

SET "curl=C:\map with spaces\curl.exe"

puts the entire parameter of command set into double quotes. As a result the environment variable curl is defined with string C:\map with spaces\curl.exe without double quotes assigned to it.

As an extra bonus of using double quotes on entire parameter string of command set, usually not visible trailing spaces and tabs at end of command line are now ignored.

But using

SET curl="C:\map with spaces\curl.exe"

results in a completely different behavior. The entire parameter variable=value of command set is now not enclosed in double quotes. As a result of the different position of first double quote in the line the command set creates now the environment variable curl with assigning the string "C:\map with spaces\curl.exe" to it with the double quotes and with including perhaps also existing spaces and tabs at end of the command line.

The batch code below copied into a batch file and executed demonstrates the differences:

@echo off
set "Var1=String with spaces and "double quotes""   
set Var2="String with spaces and "double quotes""
set Var3="String with spaces and "double quotes" and 3 trailing spaces"   
echo Var1=#%Var1%#
echo Var2=#%Var2%#
echo Var3=#%Var3%#
pause

Character # on output of the 3 variables is used to display where the assigned strings really start and end.

The output is:

Var1=#String with spaces and "double quotes"#
Var2=#"String with spaces and "double quotes""#
Var3=#"String with spaces and "double quotes" and 3 trailing spaces"   #

There are 3 trailing spaces at end of line defining Var1, but they are ignored as the double quotes enclose the entire parameter string of command set.

There are no trailing spaces on line defining Var2, but all 4 double quotes are now part of the assigned string and not just the two double quotes around double quotes part of the string.

And last the line defining Var3 has also 3 trailing spaces which are also assigned to the variable which is very often not wanted on command lines referencing the value of the environment variable.

So best is to use always set "variable=value" even if whether variable name nor value contains a space. This notation is simply a protection against invisible trailing spaces or tabs being also assigned to the environment variable.

As curl.exe with path containing spaces is now assigned to the environment variable curl without the double quotes, it is necessary to use the double quotes around the complete string containing %curl% which is here simply "%curl%". Same is true for %jq% as being not used inside a longer string and therefore always just "%jq%" can be used.

One last hint: Debugging of batch files is often quite simple. Changing first line from @ECHO off to @ECHO ON or removing/commenting first line results in execution of batch file with showing also what is really executed by command line interpreter. The error in code can be often quickly found now by looking on the command lines processed.