5

background:
I once answered this question that was about flushing two input strings from a Java process to a batch script. Since I found a workaround solution I am still very interested to solve the remaining mystery and find out why the obvious solution is not working.

problem description
See this very simple batch script:

@ECHO OFF
SET /P input1=1st Input: 
SET /P input2=2nd Input: 
ECHO 1st Input: %input1% and 2nd Input: %input2%

If you run this batch script with Java using ProcessBuilder and flush two input strings into it you will notice that only the first input string will be consumed while the second will be ignored. I found out that SET /P command consumes input from pipes when

  • CRLF token is found
  • by timeout
  • by full buffer(1024 Bytes)

My accepted workaround was based on the last two options by using a Thread.sleep(100) statement between the inputs or using a 1024 Byte Buffer for each input.

It always works for single input or in this case the first input because closing the stream has the effect that the batch script reads one input and empty returns all following SET /P statements.

the question
Why is the first option by using the CRLF token "input\r\n" not working?

research
I already tried to workaround the String.getBytes() method by creating a Byte Buffer myself using \x0d and \x0a as last bytes for CRLF token but it has no effect.

And I tried all other OutputStream wrappers like PrintWriter to check if there is a problem with the flush() implementation without any success.

I created a C++ program that basically does the same as the java programm by using CreateProcess and stangely it works like a charm.

testing code
Not working Java code:

ProcessBuilder builder = new ProcessBuilder("test.bat");
Process process = builder.start();

OutputStream out = process.getOutputStream();
out.write("foo\r\n".getBytes());
out.flush();
out.write("bar\r\n".getBytes());
out.flush();
out.close();

BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = in.readLine()) != null)
    System.out.println(line);
in.close();

Full working C++ code:

DWORD dwWritten;
char cmdline[] = "test.bat";
CHAR Input1[] = "foo\r\n";
CHAR Input2[] = "bar\r\n";
HANDLE hStdInRd = NULL;
HANDLE hStdInWr = NULL;
SECURITY_ATTRIBUTES saAttr; 
PROCESS_INFORMATION piProcInfo; 
STARTUPINFO siStartInfo;

// Create Pipe 
saAttr.nLength = sizeof(SECURITY_ATTRIBUTES); 
saAttr.bInheritHandle = TRUE; 
saAttr.lpSecurityDescriptor = NULL;
CreatePipe(&hStdInRd, &hStdInWr, &saAttr, 0); 
SetHandleInformation(hStdInWr, HANDLE_FLAG_INHERIT, 0);

// Create Process
ZeroMemory( &piProcInfo, sizeof(PROCESS_INFORMATION));  
ZeroMemory( &siStartInfo, sizeof(STARTUPINFO));
siStartInfo.cb = sizeof(STARTUPINFO); 
siStartInfo.hStdError = GetStdHandle(STD_ERROR_HANDLE);
siStartInfo.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
siStartInfo.hStdInput = hStdInRd;
siStartInfo.dwFlags |= STARTF_USESTDHANDLES;    
CreateProcess(NULL, cmdline, NULL, NULL, TRUE, 0, NULL, NULL, &siStartInfo, &piProcInfo);
CloseHandle(piProcInfo.hProcess);
CloseHandle(piProcInfo.hThread); 

// Write to Pipe
WriteFile(hStdInWr, Input1, (DWORD)strlen(Input1), &dwWritten, NULL);
FlushFileBuffers(hStdInWr);
WriteFile(hStdInWr, Input2, (DWORD)strlen(Input2), &dwWritten, NULL);
FlushFileBuffers(hStdInWr);
CloseHandle(hStdInWr);

the question again
The problem does not make any sense to me and is bugging me a lot. Why does sending the CRLF token from Java does not have any effect on batch file inputs while it does when sending from C++ program?

Community
  • 1
  • 1
ArcticLord
  • 3,999
  • 3
  • 27
  • 47
  • 1
    I suggest you to try this trick: _each line_ that you send to the Batch file must be exactly 1023 bytes long, that is, instead of send `"foo\r\n"` you should send `"foo\r\nspaces\r\n"` so the number of chars. from the first "f" to the last "\n" be exactly 1023. Do the same for the second line. See [this thread](http://www.dostips.com/forum/viewtopic.php?f=3&t=6134) for details. – Aacini Oct 31 '16 at 13:33
  • 1
    It is not correct to assume you can defer reading from process.getInputStream() until you have sent all of your data to process.getOutputStream(). Those streams operate concurrently. You should move either the reading or the writing to a separate thread (or call `processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT)`). – VGR Oct 31 '16 at 14:06
  • @Aacini As I wrote in my question I know this trick and it works. But my question is why this 1024 Byte workaround is necessary. – ArcticLord Nov 01 '16 at 08:50
  • @VGR You are right. Using different threads for input and output pipes is the correct approach. But when I do I don't get other results. This is not connected to my problem. I could even remove the whole process output reading part and there is still only one input send to the batch file. It is also a batch file only problem. With other processes this works fine. And redirecting output doesn't help either. – ArcticLord Nov 01 '16 at 08:50
  • Fair enough. I just wanted to rule that out as a possible cause. – VGR Nov 01 '16 at 14:06
  • @Aacini Why? It isn't necessary. – user207421 Nov 02 '16 at 12:24
  • @OP It isn't a 'Java CRLF token'. It is a widely used escape sequence for CR and then LF, and you are using it in C++, not Java. – user207421 Nov 02 '16 at 12:31
  • @EJP I know its a widely used escape sequence. I'm using it in my C++ and Java code but from Java it is not working. – ArcticLord Nov 02 '16 at 12:35
  • If you carefully read the thread at the given link you'll realize that this is a problem with Batch files that is perfectly identified, and that a method that allows to deal with it had been developed. The 27 replies posted there and the multipe programs used to identify and try to solve this problem is a prove that it had received enough attention already. If your question is "Why this happen?", then the easy answer is because there is a bug in cmd.exe, the same way than a lot of other "bugs" or unexpected behaviours that happen in the execution of Batch files. – Aacini Nov 02 '16 at 14:14
  • If your question is "Why this don't happen if the starting program is written in C++?", then a more precise answer depend on the differences that Java have in the Windows API functions used to implement the pipeline when it is compared vs. the ones used in C++. We have not access to the source code of Java language, so this point can only be researched via guesses and multiple tests, perhaps with the aid of a debugger that allows to inspect the execution of Java code... Good luck with the answer! – Aacini Nov 02 '16 at 14:14
  • The console uses CR and ignores LF. Although LF is treated a a delimiter by cmd. PS MS languages use CR, but notepad uses LF and CR is a zero width character. –  Nov 04 '16 at 04:16

1 Answers1

3

About "put /p" and the pipe and child processes on Windows O.S.

Just for a test I've a little expanded your test batch to get four input instead of two Now have a look at this nice test

>type test.txt | test.bat
1st Input:2nd Input:3rd Input:4th Input:1st Input: one   and 2nd Input:  and 3rd
Input:  and 4rd Input:
"--"

>test.bat < test.txt
1st Input:2nd Input:3rd Input:4th Input:1st Input: one   and 2nd Input: two and
3rd Input: three and 4rd Input: four
"--"

The interesting thing in here is that the first example works exactly as the java code (only the first "set /P" receive a value, while the second one works as expected More interesting in if you put a line somewhere in the batch file like this one: wmic Process >> TestProcesses.txt by inspecting the TestProcesses.txt, in my enviromnet, I can see that whith the first method (pipe) is present cmd.exe C:\Windows\system32\cmd.exe /S /D /c" test.bat" that isn't present when we use the second one (redirection)

I we run out new test batch (including wmic diagnostics) from java; when we inspect the TestProcesses we should see two different processes:

java.exe              java  -cp .\build\classes javaappcrlf.JavaAppCRLF
cmd.exe               C:\Windows\system32\cmd.exe /c C:\projects\JavaAppCRLF\test.bat  

as in the first method (pipe) we have a separate process for the batch where "put /p" doesnt works

From the chapter Pipes and CMD.exe of the article Pipes and CMD.exe

This has several side effects: Any newline (CR/LF) characters in the batch_command will be turned into & operators. see StackOverflow If the batch_command includes any caret escape characters ^ they will need to be doubled up so that the escape survives into the new CMD shell.

also the linked article on stack overflow is interesting

About C++ Testing

I make a little change to the c++ program described in Creating a Child Process with Redirected Input and Output just to read a file of four lines ad passing its content to a child process that execute our batch through a pipe and the results are the same of your Java program

Alternative refactoring/workaround

from the findings mentioned above, it comes that a java program that read and write to (temporary) files ( ...I know is not the same thing ) should works; I successfully tested a working solution by changing the builder this way

    ProcessBuilder builder = new ProcessBuilder(
            "cmd",
            "/c",
            "(C:\\projects\\JavaAppCRLF\\test4.bat < C:\\projects\\JavaAppCRLF\\tmp-test4.in)",
            ">",
            "C:\\projects\\JavaAppCRLF\\tmp-test4.out"
    );

Post Scriptum: an interesting note about other shell (i.e: bash on ”os x" or linux )

AFAIK not all the other platforms suffers this "issue" in the same way; i.e. on a bash (os x terminal) I made the following test with a script that acts just as our previous testing under Windows:

cd ~/projects/so-test/java-crlf-token/JavaAppCRLF  
$ cat test.sh
#!/bin/bash - 
# SET /P input1=1st Input: 
echo -n "1st Input:"; 
read input1;
#SET /P input2=2nd Input: 
echo -n "2nd Input:"; 
read input2;
#ECHO 1st Input: %input1% and 2nd Input: %input2%
echo -n "1st Input: ${input1} and 2nd Input: ${input2}"

then the only one changed to the java program is to reference the script:

ProcessBuilder builder = new ProcessBuilder("/Users/userx/projects/so-test/java-crlf-token/JavaAppCRLF/test.sh");

let's see what happens:

$ cat test.txt
abc
cde

# :pipe
$ cat test.txt | test.sh
$ cat test.txt | ./test.sh
1st Input:2nd Input:1st Input: abc and 2nd Input: cde    

# :redirection
$ ./test.sh < test.txt
1st Input:2nd Input:1st Input: abc and 2nd Input: cde

# :java 
$ java -cp build/classes/ javaappcrlf.JavaAppCRLF
1st Input:2nd Input:1st Input: foo
and 2nd Input: bar
Community
  • 1
  • 1
Franco Rondini
  • 10,841
  • 8
  • 51
  • 77
  • 1
    Thank you very much for your comprehensive research. All in all this misbehaviour seams to be a bug of the win cmd as @Aacini already stated. Will wait one or two days before reward the bounty. – ArcticLord Nov 03 '16 at 10:41
  • @ArticLord I added a note because I successfully tested a workaround solution by changing the builder (may be it's interesting for someone) , hope this is not considered out-of-scope for the question :-) – Franco Rondini Nov 03 '16 at 18:35
  • I am confused by this answer. In my first comment above I gave the link of a thread where much more extensive tests on this behavior were posted already. I also commented that this problem appear "because there is a bug in cmd.exe". However, the OP replied: _"my question is: why this 1024 Byte workaround is necessary?"_. IMHO this "answer" does _not_ answer such a question. If this point is not clear yet, then lets remember **the question again**: _"Why does sending the CRLF token from Java does not have any effect on batch file inputs...?"_. This is _not_ an answer... – Aacini Nov 04 '16 at 23:58