I need to implement multiprocessing within a module that is called by a Flask service. I know that I need to use the protection if __name__ == "__main__" when doing multiprocessing in Python. But in that case I am not sure where the __name__ == "__main__" statement should be placed.
The first piece of my code is the Flask service:
#core_api.py
import flask
import requests
import Process
app = flask.Flask(__name__)
print("starting app")
@app.route('/query', method=['POST'])
def query():
pl = requests.get_json()
result = Process.process_query(pl)
return result
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8888)
The Process.py module uses multiprocessing in a recursive way:
#Process.py
import multiprocessing
import multiprocessing.pool
from Node import nodeProcess, aggregate, processPoint
class NoDaemonProcess(multiprocessing.Process):
def _get_daemon(self):
return False
def _set_daemon(self, value):
pass
daemon = property(_get_daemon, _set_daemon)
class MyPool(multiprocessing.pool.Pool):
Process = NoDaemonProcess
def process_query(pl):
if pl['nodes']:
pool = MyPool(multiprocessing.cpu_count())
nodeResults = [pool.apply(sub_process_query, node) for node in pl['nodes']]
pool.close()
res = aggregate(nodeResults)
else:
res = processPoint(pl)
return res
def sub_process_query(pl):
return process_query(pl)
This code is not running properly, and I can see it running a multitude of child process (printing endlessly "starting app"). I assume the if __name__ == '__main__' statement is needed somewhere, or a workaround, but I am not sure exactly how.
