Launch detached process from flask

Question

Lauch subprocess from flask and restart flask will kill the subprocess also.

someone know how can detach a process from the main process ? so if i kill the main process the subprocess will remain active ?

Es:

In the main flask app when i open a route, for example /store_changes the app run the code below:

subprocess.Popen("python hdb_store_changes.py --country {0} --id_first_level {1} --threads {2} --qty {3} --username {4}".format(country.upper(),id_first_level,threads,fixedqty,username), shell=True,stdout=subprocess.PIPE)

this code will execute another program that will takes long time to process... if i restart the flask app this process die also, i need to find a way to detach from the main process...

Thanks

Celery. Here's an introduction: http://blog.miguelgrinberg.com/post/using-celery-with-flask — wgwz, May 24 '16 at 01:50
if you want it to continue running even if `flask` is dead then who reads from its stdout? If the answer is nobody then drop `stdout=PIPE`—do not use `stdout=PIPE` unless you read from the pipe while the process is running otherwise it may hang when the corresponding OS pipe buffer fills up. [To hide the output, use `stdout=DEVNULL`](http://stackoverflow.com/q/11269575/4279). Unrelated: drop `shell=True` and pass the arguments as a list instead. — jfs, May 24 '16 at 08:49

score 1 · Accepted Answer · answered May 24 '16 at 10:54

1

The solution i adopted is to use celery as mentioned by wgwz

but the solution of J.F. Sebastian is correct also.

answered May 24 '16 at 10:54

ilmetu

448
11
27

score 0 · Answer 2 · answered Mar 22 '20 at 04:16

...Not very simple:

You can set this on top of you "hdb_store_changes.py" module

try:
    pid = os.fork()
    if pid > 0:
        # Exit first parent
        _logger.info("Exit first parent PID %d" % pid)
        sys.exit(0)
except OSError as e:
    msg = "fork_child #1 failed: %d (%s)" % (e.errno, e.strerror)
    _logger.info(msg)
    sys.exit(1)

# Decouple from parent environment
os.setsid()
os.umask(0)

# Do second fork_child
try:
    pid = os.fork()
    if pid > 0:
        # Exit from second parent; print eventual PID before exiting
        _logger.info("Daemon PID %d" % pid)
        sys.exit(0)
except OSError as e:
    _logger.info("fork_child #2 failed: %d (%s)" % (
        e.errno, e.strerror))
    sys.exit(1)
...
[...code ...]  
..
if __name__ == '__main__':
    main()

In your flask set a function like:

def fork_child(args, env, cwd=ROOT):
    cmd = shlex.split(args)
    process = subprocess.Popen(args=cmd, stdin=None, stdout=None, stderr=None,
                               cwd=cwd, close_fds=True, env=env)
    _logger.info('detached process pid %s' % process.pid)
    return process.pid

an then use it as

import sys
env = os.environ
env['PYTHONPATH'] = ','.join(sys.path)
fork_child(args='python3 hdb_store_changes.py', env=env, cwd=ROOT)

Have also to make sure all stuff in your code close correctly (atexit may help), catching errors is vital. You can also communitate with the process using sockets, pipes or a db

Also, after each attempt to implement do not forget to check for Zombi, this can spread a lot of zombies...

Have fun!

Launch detached process from flask

2 Answers2