Problem with calling rsync+ssh from java on windows

Question

I'm having trouble calling rsync from java on windows vista with cygwin installed. Its strange as pasting the exact same command into a command shell works fine.

My test java call looks like this.

String[] envVars = {"PATH=c:/cygwin/bin;%PATH%"};
File workingDir = new File("c:/cygwin/bin/");
Process p = Runtime.getRuntime().exec("c:/cygwin/bin/rsync.exe -verbose -r -t -v --progress -e ssh /cygdrive/c/Users/dokeeffe/workspace/jrsync/ www.servername.net:/home/dokeeffe/rsync/",envVars,workingDir);

Then I start 2 stream reader threads to capture and log the inputStream and errorStream of the Process p.

Here is the output....

DEBUG: com.mddc.client.rsync.StreamGobbler - opening connection using: ssh www.servername.net rsync --server -vvtre.iLs . /home/dokeeffe/rsync/
DEBUG: com.mddc.client.rsync.StreamGobbler - rsync: pipe: Operation not permitted (1)
DEBUG: com.mddc.client.rsync.StreamGobbler - rsync error: error in IPC code (code 14) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/pipe.c(57) [sender=3.0.4]

The rsync code where the error happens is pipe.c(57) which is this.

if (fd_pair(to_child_pipe) < 0 || fd_pair(from_child_pipe) < 0) {
    rsyserr(FERROR, errno, "pipe");
    exit_cleanup(RERR_IPC);
}

So, for some reason fd_pair(to_child_pipe) is < 0 or fd_pair(from_child_pipe) < 0.

If anyone has any suggestions it would be great as I'm stuck now. Thanks,

Answer 1

My experience with a similar issue:

My constants:

1. the build system was ANT, running in Hudson

2. the build server is Windows Server

3. the remote test server is Linux

4. not allowed to use rsync daemon only rsync over ssh

What we did

1. Installed DeltaCopy to the windows server

2. Followed these instructions to setup DeltaCopy so it will work with SSH

   • http://www.aboutmyip.com/AboutMyXApp/DisplayFAQ.do?fid=1

   • http://troy.jdmz.net/rsync/index.html

3. I built the following ANT macro, and ran it in hudson. At first this hung on me, with no response.

   Adding the -vvv rather than -v made a different. This caused more info to be added to the hudson log.

   <macrodef name="rsync">
   <attribute name="username"/>
   <attribute name="remoteServer"/>
   <attribute name="remoteDir"/>
   <attribute name="sourceDir"/> <!-- relative to basedir start and end with / (e.g. "/html/resources/" -->
   <sequential>
       <echo message="Starting rsync @{sourceDir} to @{remoteDir} on server @{remoteServer} as user @{username} at ${release.start}"/>
   
       <pathconvert property="cygwinbasepath" dirsep="/" pathsep="" description="Replace ':' '\' with '/' and assign value to property">
           <path path="${basedir}" description="Original version" />
           <!--Pathconvert will try to add the root directory to the "path", replace with empty string --> 
           <filtermapper> 
               <replacestring from=":" to="" />  
           </filtermapper>
       </pathconvert>
       <echo message="${basedir} converted to ${cygwinbasepath}"/>
   
       <property name="rsync.cygwinSourcePath" value="/cygdrive/${cygwinbasepath}@{sourceDir}" />
       <!-- ${rsync.user}@${rsync.server}:${rsync.dir} -->
       <property name="rsync.remoteDestinationPath" value="@{remoteServer}:@{remoteDir}" />
   
       <echo message="trying to run:" />
       <!-- &quot; is escaped quote -->
       <echo>cmd /C rsync -avz -e &quot;ssh -v -l ${deployment.rsync.username} -i /cygdrive/C/cygwin/home/service-tomcat/.ssh/id_rsa&quot; --recursive --chmod=a=rw,Da+x ${rsync.cygwinSourcePath} ${rsync.remoteDestinationPath}</echo>
       <echo>
       </echo>
       <!-- 10 minute timeout -->
       <exec executable="C:\cygwin\bin\mintty.exe"
           timeout="300000"
       >
           <arg value="rsync"/>
             <arg value="-avvvz"/>
             <arg value="-e" />
             <arg value="ssh -v -l ${deployment.rsync.username} -i /cygdrive/C/cygwin/home/vj-service-tomcat/.ssh/id_rsa" />
             <arg value="--recursive"/>
             <arg value="--chmod=a=rw,Da+x"/> 
             <arg value="${rsync.cygwinSourcePath}"/>
             <arg value="${rsync.remoteDestinationPath}"/>
       </exec>
   
       <echo>
       </echo>
   
   </sequential>
   

   The key to getting this to work was to execute C:\cygwin\bin\mintty.exe rather than cmd.

4. I then manually ran the rsync command on the windows server its known_hosts file is updated with the remote host. This had to be done as the hudson user. We used the runas command in windows to launch a cygwin console as the hudson user.

Answer 2

I recently implemented something similar and thought I would share my experience, as it took quite some time to debug the problem. This question is one of the only instances of this problem on google.

My application was being launched as a service. When running the whoami command, the result was "nt authority\system". Having setup my SSH keys under /home/myusername/.ssh, the rsync command was failing as no configuration was set under /home/SYSTEM/.ssh.

Copying my /home/myusername/.ssh directory to /home/SYSTEM/.ssh and using the following String array worked for my purposes. Replace PORT with your ssh port.

new String[]{"rsync", "-z", "-I", "-e", "ssh -p PORT", server + ":" + serverPath, clientPath}

See the link here for details about launching cmd.exe under system privileges for testing purposes if this is your issue.

Answer 3

Do you have ssh private/public keystore somewhere that you have forgotten about?

Answer 4

I've never tried reading/writing using a different thread, only the same thread that executed the command. Could this be causing your problem?

You can also try passing -v to ssh to get debug output. That has helped me in the past to resolve ssh connection problems. You would pass -e "ssh -v".

Answer 5

Does cygwin's binary distribution of rsync recognize the ssh command? The command you are trying to execute probably works fine when you put into the shell because something like bash knows how to interpret the ssh command to establish a connection, i.e. use the ssh program to create a connection for the rsync program. From your debugging information I think rsync is expecting a connection but instead receives a string "ssh ..." which it fails to pipe.

To get around this you have to let bash do the work for you. Look at this page in the section on threads to tend the child. You would need to fork an instance of a command shell and write your command to the shell subprocess. So something like this:

Process p = Runtime.getRuntime().exec("c:/cygwin/bin/bash.exe");
OutputStreamWriter osw = new OutputStreamWriter(p.getOutputStream());
BufferedWriter bw = new BufferedWriter( osw, 100);
try{
    bw.write(command);
    bw.close();
}catch(IOExeception e){
    System.out.println("Problem writing command.");
}

You would probably want to read the processes output too because it will stop executing if you do not read it. The link provided covers the topic well.

Answer 6

Thanks everyone esp Daniel. The link you sent me is great. In the end I used ProcessBuilder instead of Runtime.exec. Here is the code if its any help to anyone.....

            ProcessBuilder pb = new ProcessBuilder(new String[]{"c:/cygwin/bin/rsync.exe"
                                                                ,"-verbose",
                                                                "-r",
                                                                "-t",
                                                                "-v",
                                                                "--progress",
                                                                "-e",
                                                                "ssh",
                                                                "/cygdrive/c/Users/dokeeffe/workspace/jrsync/",
                                                                "www.servername.net:/home/dokeeffe/rsync/"});
        Map<String, String> env = pb.environment();
        env.put("PATH","/cygwin/bin;%PATH%");
        pb.directory(new File("c:/cygwin/bin/"));
        Process p = pb.start();
        StreamGobbler outputGobbler = new StreamGobbler(p.getInputStream(),"OUTPUT");
        StreamGobbler errorGobbler = new StreamGobbler(p.getErrorStream(),"ERROR");
        outputGobbler.start();
        errorGobbler.start();
        int val = p.waitFor();
        if(val!=0){
            throw new Exception("Rsync return code="+val);
        }