13

My Question is what is the best way to maintain the single database connection in the entire application? Using Singleton Pattern? How?

Conditions that are needed to be taken care of:

  1. In case of multiple requests, I should be using the same connection
  2. In case connection is closed, create a new connection
  3. If the connection has timed-out, on new request my code should create a new connection.

The driver to my Database is not supported by the Django ORM. And due to same driver related issues, I am using pyodbc to connect to the database. Right now I am having below class for creating and managing the DB connections:

class DBConnection(object):
    def __init__(self, driver, serve,
                 database, user, password):

        self.driver = driver
        self.server = server
        self.database = database
        self.user = user
        self.password = password

    def __enter__(self):
        self.dbconn = pyodbc.connect("DRIVER={};".format(self.driver) +\
                                     "SERVER={};".format(self.server) +\
                                     "DATABASE={};".format(self.database) +\
                                     "UID={};".format(self.user) +\
                                     "PWD={};".format(self.password) + \
                                     "CHARSET=UTF8",
                                     # "",
                                     ansi=True)

        return self.dbconn

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.dbconn.close()

But the issue with this approach is that it will create new database connection for each query. What will be the better way to do it following singleton pattern? The way I can think of will hold the reference to the connection if the connection is closed. Something like:

 def get_database_connection():
     conn = DBConnection.connection
     if not conn:
          conn = DBConnection.connection = DBConnection.create_connection()
     return conn

What will be the best way to achieve this? Any suggestion/ideas/examples?

PS: I was checking about using weakref which allows to create weak references to objects. I think it will be good idea to use weakref along with singleton pattern for storing the connection variable. This way I won't have to keep the connection alive when DB is not in use. What you guys say about this?

Moinuddin Quadri
  • 46,825
  • 13
  • 96
  • 126
  • There seem to be a few third-party backends for pyocdb, such as [django-pyocdb](https://pypi.python.org/pypi/django-pyodbc). Why not use one of those? – knbk Nov 10 '16 at 11:20
  • I am using IBM Netezza as a database which is not supported by django-pyodbc – Moinuddin Quadri Nov 10 '16 at 11:38
  • You could implement a minimal database backend that only supports creating connections and cursors. That way you can use Django's connection management to handle your connections. I guess that would be easier than implementing the connection handling yourself. – knbk Nov 10 '16 at 11:51
  • I tool a look at it for `django.db.backends.mysql`. There 6 files namely *base.py, client.py, compiler.py, creation.py, introspection.py* and *validation.py*. I think it will a lot of overhead to implement. May be i didn't totally understood the definition of *minimal database backend*. In my mind I was thinking of creating a Singleton class to maintain single connection (already did), but your idea looks better to me. Do you know library, or blog which will give me idea of how Django's backend works? – Moinuddin Quadri Nov 10 '16 at 21:42
  • 1
    Most of that has to do with the ORM. I think if you implement the `DatabaseWrapper` class in `base.py`, with dummy classes for the others, you can use connections and cursors to execute raw queries, and rely on Django's connection management. I'm afraid I don't know of any guide or something, and I only know a bit about it myself. – knbk Nov 10 '16 at 21:56
  • This is a few years old and will need to be updated, but perhaps you could fork this as a start? It's a Django Netezza backend with pyodbc: https://github.com/msabramo/django-netezza – FlipperPA Nov 11 '16 at 02:42
  • @knbk: It seems very nice suggestion to me. I needed it on the very urgent basis, so going ahead with the singleton class. But the solution you mentioned is more promising. Will be updating the code based on `DatabaseWrapper` next week. – Moinuddin Quadri Nov 11 '16 at 06:49
  • @FlipperPA: I already tried that. There are a lot of issues in that. Looks like it's development was stopped in its alpha phase itself – Moinuddin Quadri Nov 11 '16 at 06:49

2 Answers2

7

For now, I am going ahead with the singleton class approach. Anyone seeing the potential flaws in this, feel to mention them :)

DBConnector class for creating a connection

class DBConnector(object):

   def __init__(self, driver, server, database, user, password):

        self.driver = driver
        self.server = server
        self.database = database
        self.user = user
        self.password = password
        self.dbconn = None

    # creats new connection
    def create_connection(self):
        return pyodbc.connect("DRIVER={};".format(self.driver) + \
                              "SERVER={};".format(self.server) + \
                              "DATABASE={};".format(self.database) + \
                              "UID={};".format(self.user) + \
                              "PWD={};".format(self.password) + \
                              "CHARSET=UTF8",
                              ansi=True)

    # For explicitly opening database connection
    def __enter__(self):
        self.dbconn = self.create_connection()
        return self.dbconn

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.dbconn.close()

DBConnection class for managing the connections

class DBConnection(object):
    connection = None

    @classmethod
    def get_connection(cls, new=False):
        """Creates return new Singleton database connection"""
        if new or not cls.connection:
            cls.connection = DBConnector().create_connection()
        return cls.connection

    @classmethod
    def execute_query(cls, query):
        """execute query on singleton db connection"""
        connection = cls.get_connection()
        try:
            cursor = connection.cursor()
        except pyodbc.ProgrammingError:
            connection = cls.get_connection(new=True)  # Create new connection
            cursor = connection.cursor()
        cursor.execute(query)
        result = cursor.fetchall()
        cursor.close()
        return result
Moinuddin Quadri
  • 46,825
  • 13
  • 96
  • 126
  • 1
    You should make sure that a connection is not shared between threads, or you might run into some surprising and hard-to-debug behaviour. You can save the connection on a `threading.local()` object so that each thread has it's own singleton connection. – knbk Nov 11 '16 at 11:12
  • 1
    @knbk: I am just sharing the connection object, but creating a new local `cursor` for each query in `execute_query` function. So, since query/result relationship are maintained by the cursor, I do not see any harm even if `execute_query` is called asynchronously. That is again my assumption. Or I am wrong? – Moinuddin Quadri Nov 11 '16 at 11:55
  • 2
    From the [pyodbc docs](https://mkleehammer.github.io/pyodbc/api.html): "**threadsafety** The integer 1, indicating that threads may share the module but not connections. Note that connections and cursors may be used by different threads, just not at the same time." -- So you can't share the connection between threads that try to access the database concurrently. – knbk Nov 11 '16 at 12:42
3
class DBConnector(object):
    def __new__(cls):
        if not hasattr(cls, 'instance'):
            cls.instance = super(DBConnector, cls).__new__(cls)
        return cls.instance

    def __init__(self):
        #your db connection code in constructor

con = DBConnector()
con1 = DBConnector()
con is con1 # output is True

Hope, above code will helpful.

Sout parl
  • 89
  • 9
Mahi Kumar
  • 91
  • 1
  • 1
  • 2
    Note, this didn't get me anywhere without passing `*args, **kwargs` to `__new__`. – xtian May 23 '20 at 19:40
  • @xtian because this is only showing how to create a singleton in python , if you need to send parameters you have to use args – LumbusterTick Aug 11 '23 at 09:23