0

I am working on an object detection API in Flask and I want to get a real-time video stream from the client which might be from an Android phone, an iPhone, or just from a python script running on windows/Linux. at first I tried following:

def processFrames():
    print('[DEBUG] call cv2.VideoCapture(0) from PID', os.getpid())
    camera = cv2.VideoCapture(0)
    while camera.isOpened():
        ret, frame = camera.read()
        if not ret:
            break
        else:
            frame = DetectObject(frame) 
            
            ret, buffer = cv2.imencode('.jpg', frame)
            frame = buffer.tobytes()
            yield (b'--frame\r\n'
                   b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@endpoints.route('/RealTime', methods=['GET','POST'])
def RealTime():
    return Response(processFrames(), mimetype='multipart/x-mixed-replace; boundary=frame')

But it kept on giving the error:

[ WARN:1] global /tmp/pip-req-build-13uokl4r/opencv/modules/videoio/src/cap_v4l.cpp (890) open VIDEOIO(V4L2:/dev/video0): can't open camera by index

Which I realized was because OpenCV was trying to open the Camera on the server. Then I came to this solution: https://stackoverflow.com/a/59998983/16396308

but I don't know how to receive the response from:

emit('response_back', stringData)

[EDIT]

Please Help, because when I used the solution from above, sent an Image using through postman and this is I have on the server:

frame = secure_filename(file.filename)
sbuf = StringIO()
sbuf.write(frame)
b = BytesIO(pybase64.b64decode(frame))
pimg = Image.open(b)

to receive the image as file(for now), but for one image I get the following error:

binascii.Error binascii.

Error: Incorrect padding

And for a different image I get the following Error:

PIL.UnidentifiedImageError

PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7f3d9bbcf5e0>

[EDIT 2]

I was able to stream video from a python script to a remote machine.

client.py

def RT_Rec():
    camera = cv2.VideoCapture(0)
    try:
        while True:
            ret, frame = camera.read()
            content_type = 'image/jpeg'
            headers = {'content-type': content_type}
            url = "https://[REMOTEMACHINEIP]/RealTime"
            _, img_encoded = cv2.imencode('.jpg', frame)       
            response = requests.post(url, data=img_encoded.tobytes(), headers= headers)
            if b"[SUCCESS]" in response.content or b"[ERROR]" in response.content :
                print(response.content.decode('utf-8'))
                break
            else:
                nparr = np.frombuffer(response.content,np.uint8)
                img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
                # print(type(img))
                cv2.imshow("camera", img)
            if cv2.waitKey(1) & 0xFF == ord("q"):
                break
    except Exception as e:
        print(e)
    finally:
        camera.release()
        cv2.destroyAllWindows()

SERVER.py

@endpoints.route("/RealTime",methods=["GET","POST"])
def realTime():
    if request.method == "POST":
        img = request.data
        nparr = np.fromstring(img,np.uint8)
        img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
        det,outImg = processFrame(img)
        if det:
            msg = "[SUCCESS] Processed"
            return msg
        _, img_encoded = cv2.imencode('.jpg', outImg)
        return img_encoded.tobytes()
    else:
        return "[ERROR] Invalid Request", 500

Now I want to know how can I send frames from the Android/iPhone App so that the server receives them without any error.?

M Zaid
  • 7
  • 9

1 Answers1

0

You can use SocketIO to get the user feed,

from flask import Flask, render_template, request
...
@socketio.on('connect', namespace='/web')
def connect_web():
    print('[INFO] Web client connected: {}'.format(request.sid))

@socketio.on('disconnect', namespace='/web')
def disconnect_web():
    print('[INFO] Web client disconnected: {}'.format(request.sid))


@socketio.on('connect', namespace='/cv')
def connect_cv():
    print('[INFO] CV client connected: {}'.format(request.sid))


@socketio.on('disconnect', namespace='/cv')
def disconnect_cv():
    print('[INFO] CV client disconnected: {}'.format(request.sid))

Here are some links which would come in handy :

Atharva Gundawar
  • 475
  • 3
  • 10
  • I have gone through these, these are for web clients, and I don't need to make it for a web client, rather the client would be an Android phone, an iPhone, or a python script. – M Zaid Jul 09 '21 at 06:41
  • Do you want something similar to this https://opencv.org/android/ ? – Atharva Gundawar Jul 10 '21 at 05:48
  • No, I want to be able to use real-time video(frames) from an android app, an iOS app, or even from python (cv2.VideoCapture()), convert those frames to bytes, and then send to the API, which then decodes the frames and process and return the frames as bytes, which are then decoded on the client-side. – M Zaid Jul 12 '21 at 04:05
  • sockets must be the way to go - or can't you point your phone at a .mobi site or embed a thin browser inside your app? – jtlz2 Jul 13 '21 at 07:38
  • yeah, the browser option is considered but, before going for that, I want to try to implement it natively. And, to implement sockets, I would have to completely change the system from start. – M Zaid Jul 14 '21 at 05:58