Using MTCNN with a webcam via OpenCV

Question

I wish to be able to use a webcam and utilize MTCNN as the primary facial detector. Just as one can use Haar Cascades, I want to use MTCNN to find faces on my webcam

This video is about breaking MTCNN, but nonetheless provides insight into my goal: https://www.youtube.com/watch?v=OY70OIS8bxs

Here is my code so far. It used to be so that the plot would show and I'd have to "X" it out but now it just doesn't work

from mtcnn.mtcnn import MTCNN 
import cv2 as cv
from matplotlib import pyplot
from matplotlib.patches import Rectangle

cap =  cv.VideoCapture(0)

detector = MTCNN()

#face = detector.detect_faces(img)


while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    if (ret):
        # Our operations on the frame come here
        gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)

        ax = pyplot.gca()
        face = detector.detect_faces(frame)
        face = pyplot.imread(frame)
        x, y, width, height = face[0]['box']
        rect = Rectangle((x, y), width, height, fill=False, color='red')
        ax.add_patch(rect)
        pyplot.imshow(frame)
        cv.imshow('frame',gray)
        pyplot.show()
     # Display the resulting frame
        #cv.imshow('frame',gray)
    if cv.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()
cv.destroyAllWindows()

I was hoping someone could help me...

Answer 1

Im experiencing a similiar issue at the moment. I have a program which crops the face of a video. Im using OpenCV to read in the frames and then perform the cropping on them. After that I want to save the cropped face video to a new video.

First I was also using Haar Cascade. Everything is working fine but has some general performance lacks --> It often doesnt recognize faces.

Now I wanted to use MTCNN. I changed the code in order to work with MTCNN. Everything is working fine --> It reads in the frames, performs the crop on it etc. HOWEVER, as soon as I go to saving the video the trouble happens. The code runs fine, however after opening the saved video I get an error that the video is corrupted.

I was sitting for 2h and was so confused, because the code is identical. All the outputs are same (i.e. format, size etc)

I now have to conclude that there is some error between MTCNN and Opencv. Even though this totally doesnt make sense to me why this should happen.

Update: If you run the following code: It runs fine and the video is saved again. However if you uncomment the 2 lines at the top --> it will corrupt the file and you will no longer get a working video back. Unfortunately, I could not figure out the reason for that yet.

import cv2

# from mtcnn.mtcnn import MTCNN
# cropper = MTCNN()

read_video = cv2.VideoCapture('video.mp4')
fps = read_video.get(cv2.CAP_PROP_FPS)

fourcc = cv2.VideoWriter_fourcc(*'mp4v')
write_video = cv2.VideoWriter('new3.mp4', fourcc, fps, (256,256))
images = []

success,image = read_video.read()
count = 0
while success:
    print(count)
    images.append(image)
    success, image = read_video.read()
    count += 1

for i in images:
    write_video.write(cv2.resize(i, (256, 256), interpolation=cv2.INTER_AREA))

Answer 2

This is my code to use MTCNN on a webcam and it works

import cv2
from mtcnn import MTCNN

ksize = (101, 101)
font = cv2.FONT_HERSHEY_SIMPLEX


def find_face_MTCNN(color, result_list):
    for result in result_list:
        x, y, w, h = result['box']
        roi = color[y:y+h, x:x+w]
        cv2.rectangle(color,
                      (x, y), (x+w, y+h),
                      (0, 155, 255),
                      5)
        detectedFace = cv2.GaussianBlur(roi, ksize, 0)
        color[y:y+h, x:x+w] = detectedFace
    return color


video_capture = cv2.VideoCapture(0, cv2.CAP_DSHOW)
detector = MTCNN()

while True:
    _, color = video_capture.read()
    faces = detector.detect_faces(color)
    detectFaceMTCNN = find_face_MTCNN(color, faces)
    cv2.imshow('Video', detectFaceMTCNN)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

video_capture.release()
cv2.destroyAllWindows()

About the problem below:

change

fourcc = cv2.VideoWriter_fourcc(*'mp4v')

to

fourcc = cv2.VideoWriter_fourcc(*'XVID')

It worked for me

Answer 3

Here is the simple code:

import cv2 
from mtcnn import MTCNN

cap = cv2.VideoCapture(0)
detector = MTCNN()


while True:

ret,frame = cap.read()

output = detector.detect_faces(frame)

for single_output in output:
    x,y,w,h = single_output['box']
    cv2.rectangle(frame,pt1=(x,y), pt2=(x+w,y+h),color=(255,0,0),thickness=2)
cv2.imshow('win',frame)

if cv2.waitKey(1) & 0xFF == ('x'):
    break

cv2.destroyAllWindows()

Dependencies, install these using the command in terminal then run the code above.

• pip install mtcnn

• pip install opencv-python

This works for me.