0

My former co-worker has created some "manager" classes that are interfaces to key-value tables in our database. We have, for example, a ConfigurationManager, that maps to a Configuration table in the database that holds configuration values with keys (i.e.: "mailserver") and values (i.e.: "127.0.0.1").

This table might hold a few hundred of these key-value items, and we'd like to look one of them up occasionally from our code.

To prevent that every time we need a value from the configuration table, we need to do a query, he has implemented a method to "cache" all of the values in a list in the class, so it holds on to these lists in a class variable.

If the _loaded variable is True, he loads the list from the database, if not, he returns it from the class variable, like so:

class ConfigurationManager(object):
    """
    Provides configuration option values
    """

    _all_values = []
    _loaded = False

    @staticmethod
    def value(name):
        if not ConfigurationManager._loaded:
            ConfigurationManager.load_database()

        config = [x for x in ConfigurationManager._all_values if x.conf_name.lower() == name.lower()]
        if len(config) > 0:
            option = config[0]
            return option.conf_value
        else: 
            logger.debug('Configuration value {0} is empty or does not exist.'.format(name))
            return None

    @staticmethod
    def load_database():
        """
        Load configuration options from database
        """
        from flask import current_app
        ConfigurationManager._all_values = Configuration.query.all()
        for confval in ConfigurationManager._all_values:
            current_app.db_session.expunge(confval)
        ConfigurationManager._loaded = True

Now, I must add that he is a C# developer originally, and this might be standard practice in C#. However, I have never seen an example like this in Python.

The actual problem is, that this seems to work when I test this in a unit test, but the code is run by a Huey Task worker, using a Flask instance in production. What we notice is that after some time the values can't be found any more, although they are present in the database. It seems like our class-variable-cache is emptied.

What I suspect is that either:

  • Python garbage-collects these classes and thereby erases the class-variables at some point. However, this doesn't explain why the ConfigurationManager doesn't just refresh the list from the database.
  • It has something to do with Flask and threads, where some objects are used that are "loaded" and later on another instance is used that is empty. However, the same thing goes here, the _loaded bit should indicate that the data needs to be loaded and it should, in theory, work.

If anyone can shed a light on if this method of "cacheing" works, and what could cause the variables to go missing in action, I would be most grateful.

Erik Oosterwaal
  • 4,272
  • 3
  • 44
  • 64
  • which consumer worker type are you using for huey? – dm03514 Jan 24 '19 at 16:02
  • What if you replace `Configuration.query.all()` in `load_database()` by something like `[v for v in Configuration.query.all()]`? I suspect the class to keep a reference to the query... – olinox14 Jan 24 '19 at 16:11
  • If all you need is to cache the return of `value()` go simple and use `functools.lru_cache` – yorodm Jan 24 '19 at 16:18

0 Answers0