14

I am making a Google AppEngine application and am doubting were I should store (sensitive) configuration data like credentials.

Should I make a single bigtable entity for configuration, or is there another advised way to store it.

Peter Smit
  • 27,696
  • 33
  • 111
  • 170
  • @systempuntoout For example my Amazon AWS credentials for accessing S3 – Peter Smit Sep 23 '10 at 10:53
  • If you don't like your plain credentials on your source code, the only way to go is using a configuration model on datastore otherwise you could store your settings on some .ini file accessing it with [ConfigParser](http://docs.python.org/library/configparser.html). – systempuntoout Sep 23 '10 at 12:38
  • See: https://stackoverflow.com/questions/22669528/securely-storing-environment-variables-in-gae-with-app-yaml – James Ward Apr 01 '19 at 17:26

3 Answers3

18

If you're okay with embedding them in your source, you can do that, but if you need it to be dynamically configurable, the datastore is the way to go. You can avoid fetching the settings on every request by caching them in local memory. Here's a helper class for that:

class Configuration(db.Model):
  _INSTANCE = None

  @classmethod
  def get_instance(cls):
    if not cls._INSTANCE:
      cls._INSTANCE = cls.get_or_insert('config')
    return cls._INSTANCE

Simply subclass this with whatever configuration values you need (or modify the class itself). Because loaded code persists between requests, you'll only have to do a single fetch per app instance - though if you want to be able to update the configuration dynamically, you may want to build in a timeout.

If you want to cache stuff for a limited time, your best option is simply storing the timestamp when you fetched it:

class Configuration(db.Model):
  CACHE_TIME = datetime.timedelta(minutes=5)

  _INSTANCE = None
  _INSTANCE_AGE = None

  @classmethod
  def get_instance(cls):
    now = datetime.datetime.now()
    if not cls._INSTANCE or cls._INSTANCE_AGE + cls.CACHE_TIME < now:
      cls._INSTANCE = cls.get_or_insert('config')
      cls._INSTANCE_AGE = now
    return cls._INSTANCE
Nick Johnson
  • 100,655
  • 16
  • 128
  • 198
  • Very interesting, I've never seen this technique before. Typically I would use memcache in order to minimize the number of fetches from the datastore but this looks like to be even faster as long as the instance is loaded. Any memory limit which would prevent from using this too much ? – Franck Sep 23 '10 at 14:39
  • Yes, there are memory limits - though they're subject to change. You probably shouldn't use instance memory for generalized caching, but it's ideal for configuration data. – Nick Johnson Sep 23 '10 at 14:43
  • 1
    Ok thanks I will keep this in mind. Do you have any pointer on how to build the timeout you were talking about ? – Franck Sep 23 '10 at 14:48
  • @Nick It would be cool to see a quick&dirty timeout here. First time I've seen this approach. – systempuntoout Sep 23 '10 at 15:07
  • 2
    @Franck @systempuntoout Updated the answer with a demonstration. – Nick Johnson Sep 23 '10 at 15:59
10

Store them in a module. You can go simple, like having a config.py module with, say:

AMAZON_KEY = 'XXXX'

Then use:

import config
service = my_amazon_service(config.AMAZON_KEY)

Or have a slightly more sophisticated config object that allows you to have sensible defaults for your app, namespaced config keys etc.

moraes
  • 13,213
  • 7
  • 45
  • 59
4

If it's sensitive data, you should not store it in source code as it will be checked into source control. The wrong people (inside or outside your organization) may find it there. Also, your development environment probably uses different config values from your production environment. If these values are stored in code, you will have to run different code in development and production, which is messy and bad practice.

In my projects, I put config data in the datastore using this class:

from google.appengine.ext import ndb

class Settings(ndb.Model):
  name = ndb.StringProperty()
  value = ndb.StringProperty()

  @staticmethod
  def get(name):
    NOT_SET_VALUE = "NOT SET"
    retval = Settings.query(Settings.name == name).get()
    if not retval:
      retval = Settings()
      retval.name = name
      retval.value = NOT_SET_VALUE
      retval.put()
    if retval.value == NOT_SET_VALUE:
      raise Exception(('Setting %s not found in the database. A placeholder ' +
        'record has been created. Go to the Developers Console for your app ' +
        'in App Engine, look up the Settings record with name=%s and enter ' +
        'its value in that record\'s value field.') % (name, name))
    return retval.value

Your application would do this to get a value:

AMAZON_KEY = Settings.get('AMAZON_KEY')

If there is a value for that key in the datastore, you will get it. If there isn't, a placeholder record will be created and an exception will be thrown. The exception will remind you to go to the Developers Console and update the placeholder record.

I find this takes the guessing out of setting config values. If you are unsure of what config values to set, just run the code and it will tell you!

Martin Omander
  • 3,223
  • 28
  • 23