PHP running multiple scripts concurrently

Question

I have an array with object server like this:

Array
(
    [0](
                (
                    [id] => 1
                    [version] => 1
                    [server_addr] => 192.168.5.210
                    [server_name] => server1
                )
        )
    [1](
                (
                    [id] => 2
                    [server_addr] => 192.168.5.211
                    [server_name] => server2
                )
        )
)

By running the code below, I'm able to get the desired output

foreach ($model as $server) {
        $cpu_usage = shell_exec('sudo path/to/total_cpu_usage.sh '.$server->server_addr);
        $memory_usage = shell_exec('sudo path/to/total_memory_usage.sh '.$server->server_addr);
        $disk_space = shell_exec('sudo path/to/disk_space.sh '.$server->server_addr);
        $inode_space = shell_exec('sudo path/to/inode_space.sh '.$server->server_addr);
        $network = shell_exec('sudo path/to/network.sh '.$server->server_addr);
        exec('sudo path/to/process.sh '.$server->server_addr, $processString);
        $processArray = array();
        foreach ($processString as $i) {
          $row = explode(" ", preg_replace('/\s+/', ' ', $i));
          array_push($processArray,$row);
        }
        $datetime = shell_exec('sudo path/to/datetime.sh '.$server->server_addr);
        echo $cpu_usage;
        echo $mem_usage;
        echo $disk_space;
        ......
}

My scripts are similar like:

#!/bin/bash
if [ "$1" == "" ]
then
        echo "To start monitor, please provide the server ip:"
        read IP
else
        IP=$1
fi

ssh root@$IP "date"

But the whole process took like 10 sec for 5 servers compared to 1 server for less than 2 sec. Why is that? Is there anyway to lessen the time? My guess is that the exec command was waiting for the output to be assign to the variable before going to next loop? I tried to google a little bit but most of the answer are for without returning any output at all... I need the output though

Answer 1

You can run your scripts simultaneously with popen() and grab the output later with fread().

//execute
foreach ($model as $server) {
    $server->handles = [
        popen('sudo path/to/total_cpu_usage.sh '.$server->server_addr, 'r'),
        popen('sudo path/to/total_memory_usage.sh '.$server->server_addr, 'r'),
        popen('sudo path/to/disk_space.sh '.$server->server_addr, 'r'),
        popen('sudo path/to/inode_space.sh '.$server->server_addr, 'r'),
        popen('sudo path/to/network.sh '.$server->server_addr, 'r'),
    ];
}

//grab and store the output, then close the handles
foreach ($model as $server) {
    $server->cpu_usage = fread($server->handles[0], 4096);
    $server->mem_usage = fread($server->handles[1], 4096);
    $server->disk_space = fread($server->handles[2], 4096);
    $server->inode_space = fread($server->handles[3], 4096);
    $server->network = fread($server->handles[4], 4096);

    foreach($server->handles as $h) pclose($h);
}

//print everything
print_r($model);

I tested a similar code to execute 5 scripts that sleep for 2 seconds and the whole thing took only 2.12 seconds instead of 10.49 seconds with shell_exec().

Update 1: Big thanks to Markus AO for pointing out an optimization potential.

Update 2: Modified the code to remove the possibility of overwrite. The results are now inside $model.

This can also show which server refused the connection, in case that issue about sshd is affecting you.

Answer 2

All you need to do is add an > /dev/null & at the end on Linux, you wont get the output though, but it will run as a background ( async ) process.

shell_exec('sudo path/to/datetime.sh '.$server->server_addr.'  > /dev/null &');

see also this Background process script from my GitHub, ( it has windows compatible background processes )

https://github.com/ArtisticPhoenix/MISC/blob/master/BgProcess.php

Cheers!

Answer 3

I don't know how to make your logic faster but I can tell you how I use to track time of running when I have scripts. At the begin of the script put some var $start = date('c'); and at the end just simple echo ' start='.$start; echo ' end='.date(c);

Answer 4

Yes you're correct: your PHP script is waiting for each response before moving onward.

I presume you're hoping to run the requests to all servers simultaneously, instead of waiting for each server to respond. In that case, assuming you're running a thread-safe version of PHP, look into pthreads. One option is to use cURL multi-exec for making asynchronous requests. Then there's also pcntl_fork that may help you out. Also see this & this thread for possible thread/async approaches.

Aside that, do test and benchmark the shell scripts individually to see where the bottlenecks are, and whether you can speed them up. That may be easier than thread/async setups in PHP. If you have issues with network latency, then write an aggregator shell script that executes the other scripts and returns the results in one request, and only call that in your PHP script.