Non-blocking call to external program without losing return code

Question

When starting the external program (let's call it "EX" for brevity) from within Matlab I can either do it like this

[status, result] = system('EX.exe');

which blocks until EX returns, or like this

[status, result] = system('start EX.exe');

which returns to Matlab immediately but can't fetch EX's return code.

It would be nice to still have EX's return code available in Matlab once it is done. This would be the easiest way for the calling Matlab script to notice any problems EX might run into. On the other hand, I would like Matlab to do other stuff while EX is running, e.g. displaying information on progress. So, the call needs to be non-blocking.

I tried to work around this obvious conflict by starting EX as described in the first example above. To be able to run some other code (displaying progress information) while EX is busy, I put this code into a function and called that by using a timer with a small StartDelay.

Unfortunately this does not provide real multithreading (something that Matlab seems to be incapable of without Parallel Computing Toolbox), i.e. if the code in the timer callback runs longer than EX, execution again blocks until the timer callback returns. Even worse: Since I don't know how long EX will run, the timer callback must run endlessly until it is either stopped or passed some flag that tells it to stop. I tried with global variables or even storing this flag in application data like this:

setappdata(0, 'running', 1);

tim = timer(...
    'StartDelay', 2, ...
    'TimerFcn',   'while getappdata(0, ''running''), fprintf(''timer running ...\n''); pause(1); end' ...
);

fprintf('Starting timer.\n');
start(tim);

fprintf('Calling external program ...\n');
[s,r] = system('EX.exe');
fprintf('External program returned %d.\n', s); % <-- This is never reached.

setappdata(0, 'running', 0);

fprintf('Stopping timer.\n');
stop(tim);
delete(tim);

The code following the call to system() never seems to be executed and the timer callback runs forever ... Is there any other way I can get this to work or is it really either system's return value OR non-blocking call?

I'm not yet sure how, but maybe you can use Java/.NET capabilities in MATLAB to perform asynchronous operations? — Amro, Jul 15 '11 at 16:33

score 10 · Accepted Answer · answered Jul 15 '11 at 17:15

10

I'm not sure MATLAB has native/built-in support for spawning a process. You can use Java from MATLAB to implement it, though.

For instance,

>> runtime = java.lang.Runtime.getRuntime();
>> process = runtime.exec('program arg1 arg2');
>> ...
>>
>>   % when you want to collect results.
>> process.waitFor(); % block until program returns.
>> process.exitValue(); % fetch process return code.
>>
>>   % or, if you need to abandon the work,
>> process.destroy(); % violently kill process

answered Jul 15 '11 at 17:15

André Caron

44,541
12
67
125

Not sure if I can do that. I'll have to make it a stand-alone application later (using the Matlab Compiler) and additional dependencies were not part of the deal, i.e. I cannot count on Java being installed on the target machines. As far as I can tell from Mathworks' docs, the compiler does not include a Java runtime. So, the program would require one to be already installed. Please correct me if I'm wrong. – Atze Kaputtnik Jul 16 '11 at 12:42
@Atze: Indeed, it's not clear whether the generated program includes a Java Runtime, or any of the toolboxes on the local machine. You probably need to contact the MathWorks support staff to get more precise info. – André Caron Jul 16 '11 at 15:15
I set up a test machine and ran a compiled test application based on your code. The test machine has no Java runtime installed. Everything works fine. I'll go with this solution since it is much cleaner than the textfile workaround. – Atze Kaputtnik Jul 17 '11 at 11:09
2

@Atze: I guess MATLAB ships its own Java runtime. AFAIK, it needs Java to run the interpreter and many core functions are written in MATLAB code. Glad to know it works, I'll be needing to do the same in the near future! – André Caron Jul 17 '11 at 14:12
MATLAB does ship its own jre, u can find it in MATLABROOT/sys/java/jre and MCRROOT/sys/java/jre :-) – omarzouk Sep 11 '13 at 15:49
Good solution! MATLAB's ability to basically call Java like a scripting language using the embedded JVM is awesome. For the sake of completeness it would be nice to see a ProcessBuilder version, as well. – rob Sep 26 '13 at 21:48
I was just looking for a way to spawn multiple helper processes (written in C# or C++) to run in parallel, and collect the output back from MATLAB, without using multithreading in MATLAB. This looks perfect. Now, off to read the process documentation and figure out how to fetch the output (the blocking version using MATLAB `system` is very easy). It might be worth adding stdout support to your answer, just so it is a complete non-blocking replacement for `system()`. – Ben Voigt Oct 13 '14 at 14:12
@BenVoigt: feel free to edit the answer :-) I won't be able to test it though -- I haven't used MATLAB in quite a while now! – André Caron Oct 15 '14 at 21:31
@AndréCaron: I discovered that it is actually quite difficult. The code you gave doesn't actually allow the process to end if it is writing a non-trivial amount to its stdout or stderr. You have to keep the pipes drained. So it looks like the options are either a mess of threads to copy data around or MEX. :( – Ben Voigt Oct 15 '14 at 21:34
@BenVoigt: Somehow, that's unsurprising. On Windows, anonymous pipes don't support asynchronous I/O and unfortunately, Java uses the common-denominator strategy, which probably means you have to go for a 1-thread-per-process approach. – André Caron Oct 15 '14 at 23:26

score 2 · Answer 2 · edited May 23 '17 at 11:58

2

If you have programmatic access to the other program, then the easiest way to get around this by far is to simply have the other program save results or progress notes to a textfile, and have matlab read that.

If you can't to do it that way, you may want to look into the parallel computing toolbox. You can use one thread to start the process and another thread to monitor it. I had a few questions myself on how to use the toolbox; try out their examples and post future questions here. Good luck!

edited May 23 '17 at 11:58

Community

1
1

answered Jul 15 '11 at 16:42

eykanal

26,437
19
82
113

Directly generating files to communicate the return value is not an option because, you guessed it, I cannot alter EX. I do not have the Parallel Computing Toolbox available either. But I could plant an additional program between Matlab and EX. This wrapper will then call EX and save EX's return value to a file. Not as nice and clean as getting the return value directly from the call in Matlab, but this seems to be a passable workaround. – Atze Kaputtnik Jul 16 '11 at 12:32

score 0 · Answer 3 · edited May 31 '12 at 16:35

MATLAB can make system level calls. Instead of spending $ on the "Parallel Computing Toolbox", call matlab from itself.

my machine has 4 cores, each with 2 threads. When I work MATLAB hard, I only use 12.5% of the processor. So I've begun using more than one instance of MATLAB at a time.

ex. (on a windows machine with multiple cores, or hyper-threading)

system('matlab -r "for i=1:1000000000,disp(num2str(i));end;" &'); for i=1:1000000000,disp(num2str(i));end;

all in one line at the MATLAB command prompt, this opens a second instance, and both count to a billion.

Instead of the for i=1:1000000000,disp(num2str(i));end; argument, give MATLAB a function call, like disp('dsfljkhjhkalkhjdfsahjkldahjkdfshjk'). This doesn't allow you to get return variables easily, but a good workaround might be saving things in .mat files.

Hi Kyle, welcome to StackOverflow! While your information is interesting, the solution you propose at the end seems like a special case of the wrapper Atze discussed in his comment to eykanal's answer (using MATLAB itself as the wrapper). Is that what you meant? — Jonas Heidelberg, Sep 02 '11 at 22:38

Non-blocking call to external program without losing return code

3 Answers3

Linked