Creating a Pool of Connections as Threads for MongoDB Connection with Python

Question

Things I would like to Achieve: I want to store my mongodb connections as 3-4 threads on Memory by which I want to make a pool of connections. I don't want to create a connection everytime when my core functions work. If I have a pool of connections then I can take connections from the pool, use it and release it back to the pool, this is the typical usecase.

What I have tried: I thought of creating a Daemon process by which according to the number of workers corresponding threads will be created. But the thing is how can I keep the connections always alive, so that whenever I need it, I can use the connection and release it.

Links I have referred

I read that mongodb do have internal connection pooling mechanism and I can achieve it by setting maxPoolSize=200 from https://api.mongodb.com/python/current/faq.html#how-does-connection-pooling-work-in-pymongo. But here in my case still for each process one db connection will be opened and I can't afford that so would like to avoid that instead of that I would like to create and keep some connections alive, when my server is booted.

https://stackoverflow.com/a/14700365 for variable sharing between processes, using this I am able to talk between python scripts.

https://stackoverflow.com/a/14299004 for concurrent.futures library in Python, using this I am able to create pool.

https://realpython.com/intro-to-python-threading/ for various threading related libraries in Python

I am not able to combine above programs.

Question

Am I doing the right thing? Do we have any other ways to pool the connections and store it in RAM so that I can access it as and when my core scipt needs a db connection. I don't want to create the same via sockets as because connecting over socket may also become an overhead (I feel so but I am not sure.)

In the following script I tried to create threads with the help of thread pool and create some connections. I am able to do so, but I am not sure about how I can store this in memory so that each time I can access the connection.

import threading
import time
import logging
import configparser
from pymongo import MongoClient

logging.basicConfig(level=logging.DEBUG,
                format='(%(threadName)-9s) %(message)s',)

class ThreadPool(object):
    def __init__(self):
        super(ThreadPool, self).__init__()
        self.active = []
        self.lock = threading.Lock()
    def makeActive(self, name):
        with self.lock:
            self.active.append(name)
            logging.debug('Running: %s', self.active)
    def makeInactive(self, name):
        with self.lock:
            self.active.remove(name)
            logging.debug('Running: %s', self.active)

def f(s, pool):
    logging.debug('Waiting to join the pool')
    with s:
        name = threading.currentThread().getName()
        config = configparser.ConfigParser()
        config.read('.env')
        url = config['mongoDB']['url']
        port = config['mongoDB']['port']
        user = config['mongoDB']['user']
        password = config['mongoDB']['password']
        db = config['mongoDB']['db']

        connectionString = 'mongodb://' + user + ':' + password + '@' + url + ':' + port + '/' + db

        pool.makeActive(name)
        conn = MongoClient(connectionString)

        logging.debug(conn)
        #time.sleep(0.5)
        pool.makeInactive(name)

if __name__ == '__main__':
    pool = ThreadPool()
    s = threading.Semaphore(2)
    for i in range(10):
       t = threading.Thread(target=f, name='thread_'+str(i), args=(s, pool))
       t.daemon = True
       t.start()

Creating a Pool of Connections as Threads for MongoDB Connection with Python

0 Answers0