Memcached Ignoring OPT timeouts

Question

Having an issue where calls to memcached->get( $key ) is taking over 30 seconds, and is throwing a fatal exception. Kind of a weird scenario, but basically, the environment we have, has 4 servers running memcached 0.4, and 3 running 1.08.

Here's the exception

Maximum execution time of 30 seconds exceeded in /docroot/www/library/lib/Core/Cache/Memcache.php on line 152

Line number changes, to different functions calling on the memcache connection, but they are always on either a replace or a get. I've checked the keys, and they are all alpha numeric.

line 152 of that file is

On the servers running 1.08, im trying to set it so theres a timeout of 3 seconds on a get (which is higher than what it really should be, but no matter)

So, i wrote a simple script to connect to some AWS memcache instances, set the timeouts to one microsecond and try and have it cache a big value.

Heres the script

$cache = new Memcached();
$cache->addServers( $servers );
$cache->setOption( \Memcached::OPT_RECV_TIMEOUT, 1 );
$cache->->setOption( \Memcached::OPT_SEND_TIMEOUT, 5000 );

$key = sha1( microtime( true ) . rand( 0, 9999999999 ) );
$value = sha1( $key );
for( $i = 0; $i < 10000; $i++ ) {
        $value .= sha1( $value );
}
$cache->set( $key, $value, 120 );
echo "Value Set\n";
sleep( 5 );
$start = microtime( true );
var_dump( [ 'key' => $key, 'value' => sizeof( $cache->get( $key ) ) ] );
var_dump( $cache->getResultMessage() );
$end = microtime( true );
echo "Elapsed: " . ( $end - $start ) . "\n";

AND, the output of time php test.php

Value Set
array(2) {
  ["key"]=>
  string(40) "c5c9da5ebef83a24da78cdd8bba61b0fa8473296"
  ["value"]=>
  int(1)
}
string(7) "SUCCESS"
Elapsed: 0.0075550079345703

real    0m14.326s
user    0m9.129s
sys 0m0.080s

Anyone have any idea how to solve either of my problems? Help is GREATLY appreciated

So the problems are that it takes over 30 seconds and throws an exception (what's the output for that exception?). What does this timing information have to do with it? You're only timing one part of the script, so clearly it's a different part of the script taking a lot of time.. — Brendan Long, Apr 22 '13 at 16:48
The point of the timing was to show its ignoring the timeouts I've set. (0.007 seconds > 1 microsecond), and its only on the get. Trying to show two issues in a single SO post i guess — Ascherer, Apr 22 '13 at 18:22
Recently found out that the servers were on different timezone settings. Have fixed this, and im waiting to see if the 1st issue happens again. — Ascherer, Apr 22 '13 at 18:50
I seriously doubt you could accurately time something that only takes a microsecond, especially since it involves IO (at the very least, a system call). — Brendan Long, Apr 22 '13 at 19:43
Regardless of that, its taking longer than a microsecond, around 7000 microseconds actually, which is what i wanted for this particular test, to show that the timeout isnt working — Ascherer, Apr 22 '13 at 19:57
Does it work if you set the timeout to a reasonable value, like 0.5 seconds? Also, your `var_dump` statements are probably throwing off your timing (even though `microtime()` probably isn't accurate enough for this anyway). — Brendan Long, Apr 22 '13 at 20:13
.5 seconds is way more time than its actually taking, im trying to catch it on the 30 seconds, but i cant get it to actually happen. Timing is really irrelevant in this example, as none of it is really working — Ascherer, Apr 22 '13 at 20:41
Most memcached issue like this I've seen like this are TCP issues, so if the problem remains that is where I'd look. If you don't need networking capability (i.e. local memcache instance which doesn't need to share anything with other machines), you could set it up to use unix sockets instead. — Wrikken, Apr 22 '13 at 21:36
yeah, thought that might be an issue, but i looked into it. We are on AWS's network, so its pretty fast, but we also spawned up a new cluster and its still happening on the new cluster too, when it doesnt interact with the old version — Ascherer, Apr 22 '13 at 21:55
what the hack is `for( $i = 0; $i < 10000; $i++ ) { $value .= sha1( $value ); }` ? — NullPoiиteя, Apr 23 '13 at 10:43
The point of the big for loop, was to create a big store for memcache, that would take longer to `get` — Ascherer, Apr 23 '13 at 14:22

score 2 · Accepted Answer · answered Apr 23 '13 at 17:47

Have you tried using persistent connections? It looks like you are using a server pool, so try something simple like this:

$poolId = sha1(serialize($servers));
$cache = Memcached( $poolId );
if( sizeof( $cache->getServerList() ) === 0 ) {
    $cache->addServers( $servers );
}

I had an issue with some odd behavior and random, hard to produce fatal errors and persistent connections had resolved it for me. Found my solution and code example here: Configuring the PHP Memcached client

Think this might be the issue, as the author mentions:

In a large server farm, you want to make sure you properly configure a connection pool. Otherwise, Memcached has a bad habit of happily leaking connections.

Memcached Ignoring OPT timeouts

1 Answers1