Flask (running a REST server) conflicts with OpenCV reading a webcam on Ubuntu

Question

I'm making a large program that has to use a webcam via OpenCV and also has to act as a REST server. The functionality of the two is not related, i.e. I'm not asking how to send a message via REST, that is a different topic entirely.

The concern I'm having is if I start up a Flask app acting as a REST server, then the webcam can't connect. Here is small example that demonstrates the concern:

# test.py

import numpy as np
import cv2
from flask import Flask, jsonify, request

cap = cv2.VideoCapture(0)

cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)

flaskApp = Flask(__name__)

def main():

    flaskApp.run(debug=True)

    while True:
        # Capture frame-by-frame
        ret, frame = cap.read()

        # Display the resulting frame
        cv2.imshow('frame', frame)

        keyPress = cv2.waitKey(10)
        if keyPress == ord('q'):
            break
        # end if

    # end while

    # When everything done, release the capture
    cap.release()
    cv2.destroyAllWindows()
# end function

@flaskApp.route('/post_number', methods=['POST'])
def post_number():
    if not request.json or not 'data' in request.json:
        print('error')
    # end if

    if request.json is None:
        print('error, request.json is None')
    # end if

    if not 'data' in request.json:
        print('error, \'data\' is not in request.json')
    # end if

    data = request.json['data']
    print('data = ' + str(data))

    return jsonify({'data': data}), 201
# end function

if __name__ == '__main__':
    main()

If I comment out the flaskApp.run(debug=True) line then the webcam connects and shows streaming frames in the OpenCV window as expected. However with the flaskApp.run(debug=True) included as above, the OpenCV window never appears and I get this output:

$ python3 test.py 
 * Serving Flask app "cam_test" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
[ WARN:0] global /io/opencv/modules/videoio/src/cap_v4l.cpp (887) open VIDEOIO(V4L2:/dev/video0): can't open camera by index
 * Debugger is active!
 * Debugger PIN: 297-109-197

So the Flask app starts successfully (I can even successfully send POST messages via a test client) but the OpenCV windows never appears, and please note the [ WARN:0] global /io/opencv/modules/videoio/src/cap_v4l.cpp (887) open VIDEOIO(V4L2:/dev/video0): can't open camera by index line in the above output.

I'm using Ubuntu 18.04 if that matters. Video for linux (4VL) is the standard way (perhaps the only way?) to read from webcams on Ubuntu.

It seems there is a conflict between Flask and OpenCV / V4L for some resource, however I'm not sure what that resource is. Is there a way to configure Flask so it does not use the same resources as an OpenCV webcam, or some other way to resolve this so this example app can receive a webcam feed and act as a REST server at the same time?

--- Edit ---

@dtc just pointed out an oversight I made in the comments, that being that execution never gets past the flaskApp.run(debug=True) line, so even if the webcam connected the OpenCV window would never show. This begs the question as to how the small example should be set up. I can't start flaskApp as a separate multi-process because I need the the REST message received and the info for the image to both be accessed, and spawning a multiprocess would make the memory separate.

Answer 1

As pointed out by @dtc the main problem was I was overlooking execution stopping on the flaskApp.run() line. From this post

Start a flask application in separate thread

the easy solution is to use a separate thread.

Complete working example:

server:

# server_and_cam.py

import numpy as np
import cv2
from flask import Flask, jsonify, request
import threading

cap = cv2.VideoCapture(0)

cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)

flaskApp = Flask(__name__)

def main():

    threading.Thread(target=flaskApp.run).start()

    while True:
        # Capture frame-by-frame
        ret, frame = cap.read()

        # Display the resulting frame
        cv2.imshow('frame', frame)

        keyPress = cv2.waitKey(10)
        if keyPress == ord('q'):
            break
        # end if

    # end while

    # When everything done, release the capture
    cap.release()
    cv2.destroyAllWindows()
# end function

@flaskApp.route('/post_number', methods=['POST'])
def post_number():
    if request.json is None:
        print('error, request.json is None')
    # end if

    if not 'number' in request.json:
        print('error, \'number\' is not in request.json')
    # end if

    numReceived = request.json['number']
    print('numReceived = ' + str(numReceived))

    return jsonify({'number': numReceived}), 201
# end function

if __name__ == '__main__':
    main()

client:

# client.py

import requests
import json
import time

numToSend = 0

while True:
    time.sleep(1)

    # set the url
    url = 'http://localhost:5000/post_number'

    # configure headers
    headers = {'Content-type': 'application/json'}

    # build the data and put into json format
    data = {'number': str(numToSend)}
    data_json = json.dumps(data)

    # send message to server via POST
    response = requests.post(url, headers=headers, data=data_json)
    # print response code
    print('response = ' + str(response))

    numToSend += 1
# end while

Answer 2

It has to do with how Werkzeug (the debugger used by Flask) reloads the application when changes are made to the code.

When you first run your app, python executes all the code until it hits app.run(...). If the debug mode is enabled (debug=True), which implicitly enables the reloader (use_reloader=True) by default, it will restart your app in a new child process in order to be able to kill and restart it whenever needed. Having that said, your code is actually beeing executed twice. The first time by the parent process, where the debugger lives in and than again by the child process. The parent process however, is the first one that executes cv2.VideoCapture(0) and thus blocks the device file /dev/video0 since it never releases it again, until it terminates.

The solution here is to make sure, that you do not open the device file in the parent process where the reloader runs. This can achieved by many ways.

The most convenient way is to check the environment variable WERKZEUG_RUN_MAIN, which is set by Werkzeug when it starts the reloader. This way you can still use the debug mode and its reloading functionality while making sure that cv2.VideoCapture(0) isn't called by the same process Werkzeug lives in and thus does not block /dev/video0:

if os.environ.get('WERKZEUG_RUN_MAIN') or Flask.debug is False:
    cap = cv2.VideoCapture(0)

Note: The condition or Flask.debug is False is to make sure that your code will still run when you disable the debug mode and thus the reloader. However, WERKZEUG_RUN_MAIN is only set if the reloader itself is enabled! With that said, you could actually run the app in debug mode with the reloader beeing disabled: app.run(host='0.0.0.0', debug=True, use_reloader=False). In this case the code will never get executed since both conditions will evaluate to False. So keep this in mind.