How to share dictionary between gunicorn processes?

Question

I have a server code with a global dictionary variable. It works fine with a single-process gunicorn but when I make it multi-process I get errors because the values do not reach this dictionary.

I believe the root cause is that the dictionary can not be shared between processes.

Is there any workaround for this kind of a problem?

Answer 1

I'm sorry to say that if you're trying to access to a shared python object between different instances from Gunicorn workers (gevent for example) you'll need to implement it using a shared service such loading/saving into Redis Database.

Since each Gunicorn worker process has it's own process id, you cannot access shared memory from other process.

Here you'll find more info: https://github.com/benoitc/gunicorn/issues/1026

Regards.

Answer 2

Looking for how to share a variable between gunicorn workers without using Redis or Session, here is a good alternative with the awesome python dotenv:
The principle is to read and write shared variables from a file that could be done with open() but dotenv is perfect in this situation.

1. pip install python-dotenv

2. In app directory, create .env file:
   ├── .env
   └── app.py
   .env:

    dic1="{\"key1\":\"value1\"}"
    dic2="{\"key2\":\"value2\"}"

   app.py: # flask app

from flask import Flask
import os
from dotenv import load_dotenv

app = Flask( __name__ )

# define the path explicitly if not in same folder
#env_path = os.path.dirname(os.path.realpath(__file__)) +'/../.env'



def getdotenv(env):
    try:
        #global env_path
        #load_dotenv(dotenv_path=env_path,override=True)
        load_dotenv(override=True)
        val = os.getenv(env)
        return val
    except :
        return None

def setdotenv(key, value):  # string
    global env_path
    if key :
        if not value:
            value = '\'\''
        cmd = 'dotenv -f '+env_path+' set '+key+' '+value  # set env variable
        os.system(cmd)


@app.route('/get')
def index():
    serialized_dic1 = getdotenv('dic1')   # retreive value of variable dic1
    return serialized_dic1

@app.route('/set')
def update():
    setdotenv('dic2','{\"key1\":\"value1\"}')   # set variable dic2='{\"key1\":\"value1\"}'

Answer 3

I do the following, maybe not an exact solution but still works. I use Redis on pyredis to keep the values need to be shared btw processes. And it is pretty easy after you installed Redis.