Gearman slow when client sends large payload

Question

I have an issue using Gearman that is slow to transfer tasks to workers when I send it large payloads via the Gearman PHP extension. In fact, we don't find the payload to be that big (it's 30MB). Everything (PHP, Gearman, node) runs locally for now, so network access is not the bottleneck.

The PHP script

Here is the PHP client :

ini_set('memory_limit', '1G');

$client= new GearmanClient();
$client->addServer('127.0.0.1', '4730');

$schema = file_get_contents('schema.json');
$data = file_get_contents('data.json');

$gearmanData = [
    'schema' => $schema,
    'data' => $data
];

echo "Encoding in JSON the payload\n";

$gearmanDataString = json_encode($gearmanData, JSON_FORCE_OBJECT);

echo "Sending job to Gearman\n";

// This line takes long to execute...
$result = $client->doNormal("validateJsonSchema", $gearmanDataString);

echo "Job finished\n";

var_dump($result);

This is my nodejs worker which will eventually do something, but is empty to demonstrate that worker code is not an issue :

var gearmanode = require('gearmanode');

var worker = gearmanode.worker({host: '127.0.0.1' port: 4730});

worker.addFunction('validateJsonSchema', function (job) {
    console.log('I will do something');

    job.workComplete('Toasty!');
});

I start my worker in background and then run my client, and it freezes while for 30 seconds or so while doing $client->doNormal (just after outputing Sending job to Gearman), and finishes by outputing string(7) "Toasty!" via PHP's var_dump. So it works, but its just long to process.

Also, if I reduce the size of the payload (data.json), it takes less time, so the payload size seems to matter.

I tried to code the same worker in PHP, with the same result:

$worker= new GearmanWorker();
$worker->addServer('127.0.0.1', '4730');
$worker->addFunction("validateJsonSchema", "validateJsonSchema");
while ($worker->work());

function validateJsonSchema($job)
{
  return 'ToastyPHP!';
}

UPDATE

Using the node.js client, doing almost the same thing than in PHP, it executes much faster (~3.5 seconds). Am I doing something wrong with the PHP version, or I am missing some configuration to make it quicker ?

My node.js client:

var gearmanode = require('gearmanode');
var fs = require('fs');

var start = Date.now(); 

var client = gearmanode.client();

schema = fs.readFileSync('schema.json', 'utf8');
data = fs.readFileSync('data.json', 'utf8');

var submitData = JSON.stringify({ "data": data, "schema": schema });

// Runs much faster than PHP
var job = client.submitJob('validateJsonSchema', submitData, {background: false});

job.on('complete', function() {
    console.log('RESULT >>> ' + job.response);
    client.close();

    var end = Date.now(); 

    console.log(end-start + ' milliseconds'); // Always shows around 3500 milliseconds
});

Any clue why is this happening ? Is Gearman made to handle this size of payload ? 30MB is not that big in my book.

Answer 1

Check whether this code works for you ,took really small time to complete the job.

worker.php:

echo "Starting\n";
$gmworker = new GearmanWorker();

# Add default server (localhost).
$gmworker->addServer('127.0.0.1', '4730');
$gmworker->addFunction("jsonValid", "jsonValid");



print "Waiting for job...\n";
while ($gmworker->work()) {
    if ($gmworker->returnCode() != GEARMAN_SUCCESS) {
        echo "return_code: " . $gmworker->returnCode() . "\n";
        break;
    }
}

function jsonValid($job)
{
    return 'ToastyPHP!';
}

Client.php

ini_set('memory_limit', '1G');

$client = new GearmanClient();
$client->addServer('127.0.0.1', '4730');
$client->setCompleteCallback("complete");
$time = time();

echo "<pre>Sending job..." . "\n";


$schema = file_get_contents('AllSets.json');
$data = file_get_contents('AllSets.json');


$gearmanData = Array(
    'schema' => $schema,
    'data' => $data
);

$gearmanDataString = json_encode($gearmanData, JSON_FORCE_OBJECT);

$client->addTask("jsonValid", $gearmanDataString, null, 'Json');
$client->runTasks();

echo "Job finished\n";

$endtime = time();
print "Completed in " . ($endtime - $time) . ' seconds' . "\n";

function complete($task)
{
    print "Unique : " . $task->unique() . "\n";
    print "Data : " . $task->data() . "\n";
}

I have used the addTask and runTasks methods instead of doNormal.For the json data to be send I used the AllSets.json file from http://mtgjson.com/ around 30Mb size(total load), the job finised in 1 Sec , and after trying a file of around 200Mb it took 4 sec.