3

I have a webapp using Flask, Gunicorn and Nginx. In accordance with the documentation, I'm using a worker count larger than the number of CPUs.

However, when I have a scheduled task that writes to a file, it's being rewritten once by each worker, instead of being written only once overall.

the gunicorn configuration

command = /home/brun0/infraestruturas/venv/bin/gunicorn -w 3 app:app

if i use 1 instead of 3 workers it only prints once (like i want) but i dont know if its good to do this with only 1 worker

flask app

app = Flask(__name__)

@app.route('/')
def index():
    return render_template('index.html')

def trains():
    try:
        data = list(getRequest())
        trainList = getTrain(data)
        delayedSet = getDelayedCSV(trainList)
        delayedCounter(delayedSet)
    except Exception as e:
        print(e)



scheduler = BackgroundScheduler()
scheduler.add_job(func=trains, trigger='interval', minutes=10 ,max_instances=100, misfire_grace_time=None)
scheduler.start()

if __name__ == '__main__':
    app.run()
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
Bruno Teixeira
  • 128
  • 1
  • 12
  • Please post the code from your script so we can better answer the question. Gunicorn should assign a single worker to a single request so need to see the code. – Nate Dellinger Aug 23 '19 at 12:31
  • the code is realy big ... more than one file. you want me to show you the function that does the request? – Bruno Teixeira Aug 23 '19 at 13:21
  • That would probably be helpful, along with the command you are using to run gunicorn or your custom gunicorn config if you are using one. – Nate Dellinger Aug 23 '19 at 14:35
  • Ok @NateDellinger its better now? i am using supervisor too – Bruno Teixeira Aug 23 '19 at 15:09
  • @BrunoTeixeira,...as a rule, what we want here is a [mcve] -- *the shortest possible code* that can be used to reproduce the same problem (and test whether a fix succeeds) without needing any changes before it's ready to run, complete with enough logging (but preferably only just enough!) to be able to tell whether the fix succeeded. – Charles Duffy Aug 23 '19 at 15:10
  • that said, each request should only go to one worker; if you have the same request being processed more than once, that's... very unusual. – Charles Duffy Aug 23 '19 at 15:10
  • ...so, a good place to start is to build the shortest possible flask+gunicorn "hello world" program you can, and then figure out what needs to be added to it to make it demonstrate the issue. – Charles Duffy Aug 23 '19 at 15:12
  • thanks for the words @CharlesDuffy . i think the issue is already demonstrated by adding the code and the gunicorn command. the only thing that is doing three times is when writing to a file and when i run that same piece of code, running localy it only writes in the file once so the only problem i see is the 3 workers in gunicorn – Bruno Teixeira Aug 23 '19 at 15:15
  • We gotta see the @route which is utilizing `getRequest`. With flask you can pretty easily create a single file which reproduces your problem with the app initiation and route setup in one file. It could be something strange is going on with your yield. – Nate Dellinger Aug 23 '19 at 15:19
  • @NateDellinger there you go dude. can you confirm? – Bruno Teixeira Aug 23 '19 at 15:25
  • 4
    Oh haha. You are using a scheduler. The function run is not related to any request. Gunicorn forks multiple workers which starts up multiple of your background schedulers. Try adding --preload when launching gunicorn, which should load the app before launching workers and only create one background scheduler. `/home/brun0/infraestruturas/venv/bin/gunicorn -w 3 --preload app:app` – Nate Dellinger Aug 23 '19 at 15:32

0 Answers0