2

I am using Windows MATLAB to run SSH commands, but every once in a while the SSH command hangs indefinitely, and then so does my MATLAB script (sometimes when running overnight). How can I have a command time-out after a certain amount of waiting time? For example, suppose I don't want to wait more than 3 seconds for a SSH command to finish execution before breaking it and moving on:

% placeholder for a command that sometimes hangs
[status,result] = system('ssh some-user@0.1.2.3 sleep 10')
% placeholder for something I still want to run if the above command hangs
[status,result] = system('ssh some-user@0.1.2.3 ls')

I'd like to make a function sys_with_timeout that can be used like this:

timeoutDuration = 3;
[status,result] = sys_with_timeout('ssh some-user@0.1.2.3 sleep 10', timeoutDuration)
[status,result] = sys_with_timeout('ssh some-user@0.1.2.3 ls', timeoutDuration)

I've tried the timeout function from FEX but it doesn't seem to work for a system/SSH command.

Cris Luengo
  • 55,762
  • 10
  • 62
  • 120
MountainDrew
  • 443
  • 4
  • 20

2 Answers2

2

I don't think the system command is very flexible, and I don't see a way to do what you want using it. But we can use the Java functionality built into MATLAB for this, according to this answer by André Caron.

This is how you'd wait for the process to finish with a timeout:

runtime = java.lang.Runtime.getRuntime();
process = runtime.exec('sleep 20');
process.waitFor(10, java.util.concurrent.TimeUnit.SECONDS);
if process.isAlive()
    disp("Done waiting for this...")
    process.destroyForcibly();
end
process.exitValue()

In the example, we run the sleep 20 shell command, then waitFor() waits until the program finishes, but for a maximum of 10 seconds. We poll to see if the process is still running, and kill it if it is. exitValue() returns the status, if you need it.

Running sleep 5 I see:

ans =
     0

Running sleep 20 I see:

Done waiting for this...
ans =
   137
Cris Luengo
  • 55,762
  • 10
  • 62
  • 120
1

Building on @Cris Luengo's answer, here is the sys_with_timeout() function. I didn't use the process.waitFor() function because I'd rather wait in a while loop and display output as it comes in. The while loop breaks once the command finishes or it times out, whichever comes first.

function [status,cmdout] = sys_with_timeout(command,timeoutSeconds,streamOutput,errorOnTimeout)
    arguments
        command char
        timeoutSeconds {mustBeNonnegative} = Inf
        streamOutput logical = true    % display output as it comes in
        errorOnTimeout logical = false % if false, display warning only
    end
    % launch command as java process (does not wait for output)
    process = java.lang.Runtime.getRuntime().exec(command);
    % start the timeout timer!
    timeoutTimer = tic();
    % Output reader (from https://www.mathworks.com/matlabcentral/answers/257278)
    outputReader = java.io.BufferedReader(java.io.InputStreamReader(process.getInputStream()));
    errorReader = java.io.BufferedReader(java.io.InputStreamReader(process.getErrorStream()));

    % initialize output char array
    cmdout = '';
    while true
        % If any lines are ready to read, append them to output
        % and display them if streamOutput is true
        if outputReader.ready() || errorReader.ready()
            if outputReader.ready()
                nextLine = char(outputReader.readLine());
            elseif errorReader.ready()
                nextLine = char(errorReader.readLine());
            end
            cmdout = [cmdout,nextLine,newline()];
            if streamOutput == true
                disp(nextLine);
            end
            continue
        else
            % if there are no lines ready in the reader, and the
            % process is no longer running, then we are done
            if ~process.isAlive()
                break
            end
            % Check for timeout.  If timeout is reached, destroy
            % the process and break
            if toc(timeoutTimer) > timeoutSeconds
                timeoutMessage = ['sys_with_timeout(''',command, ''',', num2str(timeoutSeconds), ')',...
                                  ' failed after timeout of ',num2str(toc(timeoutTimer)),' seconds'];
                if errorOnTimeout == true
                    error(timeoutMessage)
                else
                    warning(timeoutMessage)
                end
                process.destroyForcibly();
                break
            end
        end
    end
    if ~isempty(cmdout)
        cmdout(end) = []; % remove trailing newline of command output
    end
    status = process.exitValue(); % return
end

Replacing ssh some-user@0.1.2.3 with wsl (requires WSL of course) for simplicity, here is an example of a function that times out:

>> [status,cmdout] = sys_with_timeout('wsl echo start! && sleep 2 && echo finished!',1)
start!
Warning: sys_with_timeout('wsl echo start! && sleep 2 && echo finished!',1) failed after timeout of 1.0002 seconds 
> In sys_with_timeout (line 41) 
status =
     1
cmdout =
    'start!'

and here is an example of a function that doesn't time out:

>> [status,cmdout] = sys_with_timeout('wsl echo start! && sleep 2 && echo finished!',3)
start!
finished!
status =
     0
cmdout =
    'start!
     finished!'
MountainDrew
  • 443
  • 4
  • 20