Correct Way of using Redis Connection Pool in Python

Question

How should two different modules foo.py and bar.py get a connection from a Redis connection pool? In other words, how should we structure the app?

I believe the goal is to have just a single connection pool for all modules to get a connection from.

Q1: In my example, does both modules get a connection from the same connection pool?

Q2: Is it Ok to create the RedisClient instance in RedisClient.py, then import the instance into the other 2 modules? Or is there a better way?

Q3: Is lazy loading of the conn instance variable actually useful?

RedisClient.py

import redis

class RedisClient(object):

    def __init__(self):
        self.pool = redis.ConnectionPool(host = HOST, port = PORT, password = PASSWORD)

    @property
    def conn(self):
        if not hasattr(self, '_conn'):
            self.getConnection()
        return self._conn

    def getConnection(self):
        self._conn = redis.Redis(connection_pool = self.pool)

redisClient = RedisClient()

foo.py

from RedisClient import redisClient

species = 'lion'
key = 'zoo:{0}'.format(species)
data = redisClient.conn.hmget(key, 'age', 'weight')
print(data)

bar.py

from RedisClient import redisClient

print(redisClient.conn.ping())

Or is this better?

RedisClient.py

import redis

class RedisClient(object):

    def __init__(self):
        self.pool = redis.ConnectionPool(host = HOST, port = PORT, password = PASSWORD)

    def getConnection(self):
        return redis.Redis(connection_pool = self.pool)

redisClient = RedisClient()

foo.py

from RedisClient import redisClient

species = 'lion'
key = 'zoo:{0}'.format(species)
data = redisClient.getConnection().hmget(key, 'age', 'weight')
print(data)

bar.py

from RedisClient import redisClient

print(redisClient.getConnection().ping())

somewhat similar to https://stackoverflow.com/questions/31663288/how-do-i-properly-use-connection-pools-in-redis — Sahil, Mar 21 '18 at 05:43

score 12 · Accepted Answer · edited May 05 '22 at 13:51

A1: Yes, they use the same connection pool.

A2: This isn't a good practice. As you cannot control the initialization of this instance. An alternative could be use singleton.

import redis


class Singleton(type):
    """
    An metaclass for singleton purpose. Every singleton class should inherit from this class by 'metaclass=Singleton'.
    """
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]


class RedisClient(metaclass=Singleton):

    def __init__(self):
        self.pool = redis.ConnectionPool(host = HOST, port = PORT, password = PASSWORD)

    @property
    def conn(self):
        if not hasattr(self, '_conn'):
            self.getConnection()
        return self._conn

    def getConnection(self):
        self._conn = redis.Redis(connection_pool = self.pool)

Then RedisClient will be a singleton class. Not matter how many times you call client = RedisClient(), you will get the same object.

So you can use it like:

from RedisClient import RedisClient

client = RedisClient()
species = 'lion'
key = 'zoo:{0}'.format(species)
data = client.conn.hmget(key, 'age', 'weight')
print(data)

And the first time you call client = RedisClient() will actually initialize this instance.

Or you may want to get different instance based on different arguments:

class Singleton(type):
    """
    An metaclass for singleton purpose. Every singleton class should inherit from this class by 'metaclass=Singleton'.
    """
    _instances = {}

    def __call__(cls, *args, **kwargs):
        key = (args, tuple(sorted(kwargs.items())))
        if cls not in cls._instances:
            cls._instances[cls] = {}
        if key not in cls._instances[cls]:
            cls._instances[cls][key] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls][key]

In Python 3, do you use `class RedisClient(metaclass=Singleton):` or `class RedisClient(BaseClass, metaclass=Singleton):`? I noticed that in your example code, you did not let `RedisClient` inherit `Singleton` class — Nyxynyx, Mar 21 '18 at 05:44
Both of them are correct. The latter one is inherited from `BaseClass`. — Sraw, Mar 21 '18 at 05:47
`Singleton` is a metaclass, you need `metaclass=Singleton`. To understand what is a metaclass, that is another big topic. — Sraw, Mar 21 '18 at 05:49
Can you please elaborate on what issue may be encountered when **you cannot control the initialization of this instance**? — Nyxynyx, Mar 22 '18 at 03:16
In your example, you are just creating a cheap instance, but how about if you need to initialize an expensive instance? You will be blocked for a while when you just `import RedisClient`. Even worse, you actually need to use another class in `RedisClient` module but not `redisClient`. So in this case, you never use `redisClient` but need to pay for its initialization. — Sraw, Mar 22 '18 at 03:34

Correct Way of using Redis Connection Pool in Python

1 Answers1

Linked