ThreadPoolExecturor future never finishing

Question

Im writing a backend for a program that needs to be able to ping all devices on a network to see if they exist or not. This backend runs on flask sockets. I wrote a simple test script that does exactly this using threadExecutor and futures to speed up the process.

import platform
import subprocess
import netifaces as ni
import concurrent.futures
import socket

baseIP = ni.ifaddresses('enp4s0')[ni.AF_INET][0]['addr']
prefix = '.'.join(baseIP.split('.')[0:3])
aliveIPs = []

def ping(ip):
    command = ['ping', '-c 1', '-w 5', ip]
    print(ip)
    if(subprocess.call(command) == 0):
        return socket.gethostbyaddr(ip)
    else:
        return ''

ips = [prefix + '.' +str(i) for i in range(0,255)]

with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = []
    for ip in ips:
        futures.append(executor.submit(ping, ip=ip))
    for future in concurrent.futures.as_completed(futures):
        res = future.result()
        if(res != ''):
            aliveIPs.append(res)

print(aliveIPs)

This code works fine being ran with python3 on my pc and gives me a list of devices on the network.

But when I implement this code into my flask application, the future never finishes, causing it to hang on the first ip it tries to scan. The following code is from my flask application.

def ping(ip):
    print(ip)
    proc = subprocess.run(['ping', '-c 1', '-w 5', ip])
    if(proc == 0):
        print("alive")
        return socket.gethostbyaddr(ip)
    else:
        print("none")
        return ''


"""
gets a list of available hosts in the network for a connection
"""


def update_worker_list(executor):
    print("updating")
    network_device = get_network_info()['net_name']
    ip = ni.ifaddresses(network_device)[ni.AF_INET][0]['addr']
    ipPrefix = '.'.join(ip.split('.')[0:3])
    alive_ips = []

    ips = [ipPrefix + '.' + str(i) for i in range(0, 255)]
    futures = []
    for ip in ips:
        futures.append(executor.submit(ping, ip=ip))
    for future in concurrent.futures.as_completed(futures):
        res = future.result()
        if(res != ''):
            alive_ips.append(res)
    print(alive_ips)

The executor parameter here is a flask_executor from the flask-executor pip package. Its intitated like so:

app = Flask(__name__)
app.config['SECRET_KEY'] = DEV_SECRET
app.config['DEBUG'] = True
app.config['EXECUTOR_TYPE'] = 'thread'
app.config['EXECUTOR_MAX_WORKERS'] = 20
executor = Executor(app)

I originally tried it with the executor python provides, but this resulted in the same bug. The following is the output to STDOUT from the update_workers_list function:

updating
192.168.8.0
Do you want to ping broadcast? Then -b. If not, check your local firewall rules.
none

Any ideas as to why the future does not resolve is what I am looking for.

I see from the https://flask-executor.readthedocs.io/en/latest/#futures doc that the flask-executor is a subclass of concurrent.futures. My best guess is that maybe it doesn't play nice with the .as_completed. There was some talk in there about getting the results from the executor, but you may have to specify your own callback. — Ben, Sep 03 '20 at 14:34
@Ben I did originally try it with the inbuilt executor that python provides. The unfortunately gives the same hanging result. Providing my own callback to that does not change the situation. — duck, Sep 03 '20 at 15:05
So after a little more digging I found that the future only resolves after I stop the python program. The last function call is to `work_item = work_queue.get(block=True)`. This leads me to believe that there is some sort of deadlock going on that flask itself is causing. I have thus tried other methods that dont require as_completed by waiting for each thread to finish synchronously. This unfortunately did not help. It seems that Flask internally is causing some sort of deadlock. — duck, Sep 06 '20 at 11:39

score 0 · Accepted Answer · answered Sep 06 '20 at 11:52

So after digging through the source for flask-socketIO, I discovered that socketIO likes to hog threads. This means you have to monkey patch it yourself into using another either using gevent or eventlet. this post explains it quite well and lead me to the fix to my bug.

ThreadPoolExecturor future never finishing

1 Answers1