WinError6 The Handle is Invalid Python 3+ multiprocessing

Question

I am running a Python 3.7 Flask application which uses flask_socketio to setup a socketio server for browser clients, another python process to connect to a separate remote socketio server & exchange messages, and another python process to read input from a PIR sensor.

Both python processes communicate over multiprocessing.Queue - but, the socketio process always gets either [WinError6] - Invalid Handle or [WinError5] - Permission Denied. I have absolutely no idea what I'm doing wrong.

Here's the top-level (server) code; it does not appear to have issues:

from shotsocket import init as shotsocket_init
from shotsocket import util as matchmaking_util
import multiprocessing, os, config, uuid

match_queue = multiprocessing.Queue()
shot_queue = multiprocessing.Queue()

app = Flask(__name__, static_url_path='', static_folder='templates')
socketio = SocketIO(app)

_rooms = [] # I don't plan to keep this in memory, just doing it for debug / dev 

...

The above works fine and dandy. The 2nd to last line in the following block is the issue.

# THIS IS THE FUNC WHERE WE ARE TRYING TO USE 
# THE BROKEN QUEUE

@socketio.on('connect')
def listen():
    room_key = str(uuid.uuid4())
    join_room(room_key)
    _rooms.append((room_key, request.sid))
    possible_match = matchmaking_util.match_pending_clients(_rooms)
    if possible_match:
        shot_queue.put_nowait(possible_match)
        print('put it in there')

Here's how I start these processes:


if __name__ == '__main__':
    debug = os.environ.get('MOONSHOT_DEBUG', False)
    try:
        proc = multiprocessing.Process(target=start, args=(debug,match_queue))
        proc.start()
        shot_proc = multiprocessing.Process(target=shotsocket_init, args=(shot_queue,))
        shot_proc.start()
        socketio.run(app, host='0.0.0.0')
    except KeyboardInterrupt:
        socketio.stop()
        proc.join()
        shot_proc.join()

And here's the entirety of `shotsocket` (the code that cannot read the queue)

import socketio, multiprocessing # mp for the type

sio = socketio.Client(engineio_logger=True)
sio.connect('redacted woot', transports=['websocket'])


@sio.on('connect')
def connect():
    print("connected to shot server")

def init(queue: multiprocessing.Queue):
    while True:
        try:
            # WE NEVER GET PAST THIS LINE
            print(queue.get())
        except Exception as e:
            continue
        if not queue.empty():
            print('queue empty')
            shot = queue.get()
            print(shot)         
            match_id, opponents = shot
            sio.emit('start', {'id': match_id, 'opponents': [opponents[0], opponents[1]]})

I'm pulling my hair out. What the heck am I doing wrong?

Move all instantiations from your first code snippet below `if __name__ == '__main__':`, else they will be re-executed for every child process when the target-functions get imported by them. — Darkonaut, Feb 11 '19 at 22:23
@Darkonaut can you make this an answer and submit it? Include code if possible. I made the change and see important differences in execution but am still getting the same errors. — Michael Ward, Feb 12 '19 at 20:50
I cannot replicate sorry. Obviously for some reason the inheritence of the (FIFO) handles fails. I'm not on Windows nor am I using Flask so what I wrote already is pretty much everything I can say. Are you on Python 3.7.2? — Darkonaut, Feb 12 '19 at 21:01
Yeah I am - found a solution though. See my answer; any explanation of why this fixed the issue would be appreciated. (But obviously, if you can't repro, it's understandable you wouldn't know) — Michael Ward, Feb 12 '19 at 22:21
Staying with multithreading instead of going multiprocess is riding a different pony. The queues are differently implemented under the hood. Threads in the same process also naturally share memory. Using multithreading would be the first choice for I/O-bound tasks anyway. AFAIK Python 3.7.2 introduced a regression bug with multiprocessing and virtual environments, [see](https://stackoverflow.com/q/54480527/9059420). — Darkonaut, Feb 12 '19 at 22:32

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

0

Solution

I have no idea why this fixes the problem, but switching from multiprocessing.Queue to queue.Queue and multiprocessing.Process to threading.Thread did it.

edited Jun 20 '20 at 09:12

Community

1
1

answered Feb 12 '19 at 22:21

Michael Ward

496
5
13

WinError6 The Handle is Invalid Python 3+ multiprocessing

Here's the top-level (server) code; it does not appear to have issues:

The above works fine and dandy. The 2nd to last line in the following block is the issue.

Here's how I start these processes:

And here's the entirety of `shotsocket` (the code that cannot read the queue)

1 Answers1

Solution

Linked

WinError6 The Handle is Invalid Python 3+ multiprocessing

Here's the top-level (server) code; it does not appear to have issues:

The above works fine and dandy. The 2nd to last line in the following block is the issue.

Here's how I start these processes:

And here's the entirety of shotsocket (the code that cannot read the queue)

1 Answers1

Solution

Linked

And here's the entirety of `shotsocket` (the code that cannot read the queue)