RedisClient fails strategy

Question

I'm using play framework with scala. I also use RedisScala driver (this one https://github.com/etaty/rediscala ) to communicate with Redis. If Redis doesn't contain data then my app is looking for data in MongoDB. When Redis fails or just not available for some reason then application waits a response too long. How to implement failover strategy in this case. I would like to stop requesting Redis if requests take too long time. And start working with Redis when it is back online. To clarify the question my code is like following now

private def getUserInfo(userName: String): Future[Option[UserInfo]] = {
    CacheRepository.getBaseUserInfo(userName) flatMap{
      case Some(userInfo) =>
        Logger.trace(s"AuthenticatedAction.getUserInfo($userName). User has been found in cache")
        Future.successful(Some(userInfo))
      case None =>
        getUserFromMongo(userName)
    }
  }

Answer 1

I think you need to distinguish between the following cases (in order of their likelihood of occurrence) :

1. No Data in cache (Redis) - I guess in this case, Redis will return very quickly and you have to get it from Mongo. In your code above you need to set the data in Redis after you get it from Mongo so that you have it in the cache for subsequent calls.

   You need to wrap your RedisClient in your application code aware of any disconnects/reconnects. Essentially have a two states - first, when Redis is working properly, second, when Redis is down/slow.

2. Redis is slow - this could happen because of one of the following.
   2.1. Network is slow: Again, you cannot do much about this but to return a message to your client. Going to Mongo is unlikely to resolve this if your network itself is slow.

   2.2. Operation is slow: This happens if you are trying to get a lot of data or you are running a range query on a sorted set, for example. In this case you need to revisit the Redis data structure you are using the the amount of data you are storing in Redis. However, looks like in your example, this is not going to be an issue. Single Redis get operations are generally low latency on a LAN.

3. Redis node is not reachable - I'm not sure how often this is going to happen unless your network is down. In such a case you also will have trouble connecting to MongoDB as well. I believe this can also happen when the node running Redis is down or its disk is full etc. So you should handle this in your design. Having said that the Rediscala client will automatically detect any disconnects and reconnect automatically. I personally have done this. Stopped and updgrade Redis version and restart Redis without touching my running client(JVM).

Finally, you can use a Future with a timeout (see - Scala Futures - built in timeout?) in your program above. If the Future is not completed by the timeout you can take your other action(s) (go to Mongo or return an error message to the user). Given that #1 and #2 are likely to happen much more frequently than #3, you timeout value should reflect these two cases. Given that #1 and #2 are fast on a LAN you can start with a timeout value of 100ms.

Answer 2

Soumya Simanta provided detailed answer and I just would like to post code I used for timeout. The code requires Play framework which is used in my project

private def get[B](key: String, valueExtractor:  Map[String, ByteString] => Option[B], logErrorMessage: String): Future[Option[B]] = {
    val timeoutFuture = Promise.timeout(None, Duration(Settings.redisTimeout))
    val mayBeHaveData = redisClient.hgetall(key) map { value =>
      valueExtractor(value)
    } recover{
      case e =>
        Logger.info(logErrorMessage + e)
        None
    }

    // if timeout occurred then None will be result of method
    Future.firstCompletedOf(List(mayBeHaveData, timeoutFuture))
  }