0

I would like to do image processing in Python. The problem is that I have a loop that is recording the image data from a camera into a numpy array, but in the loop I am trying to do a correlation of pixel data from the last image to the current image to determine whether or not I need to do further processing. This however is killing the execution speed of the loop which is showing lagged image output. 

def gen(camera):
    image = np.zeros([480, 640, 3])
    last_image = np.zeros([480, 640, 3])

    frame_number = 0

    while True:
        frame = camera.get_frame()
        #new code

        if(frame_number % 10 == 0):
            #save the current frame
            image = frame
            #this line takes forever on 480x640x3 sized nd.arrray
            corr = signal.correlate(image, last_image)

        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

        faces = faceCascade.detectMultiScale(
                    gray,
                    scaleFactor=1.1,
                    minNeighbors=5,
                    minSize=(30, 30),
                    flags=cv2.CASCADE_SCALE_IMAGE
                )

                # Draw a rectangle around the faces
        for (x, y, w, h) in faces:
                    cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)
        #cv2.imshow('Video', frame)
        ret, jpeg = cv2.imencode('.jpg', frame)
        #if cv2.waitKey(1) & 0xFF == ord('q'):
         #   break
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + jpeg.tobytes() + b'\r\n\r\n')
        if(frame_number % 10 == 0):
            #save current frame as last frame for next process 
            last_image = frame

        frame_number += 1

Is multi threading the way to solve this problem?

Jack Frye
  • 583
  • 1
  • 7
  • 28
  • Like this? https://stackoverflow.com/questions/14781895/running-python-multiprocess-for-image-processing – jwpfox Jan 06 '18 at 23:51
  • 2
    No, multithreading is *not* the solution. Notice that `.jpg` encoding *is* slow. – Antti Haapala -- Слава Україні Jan 06 '18 at 23:51
  • What do you call "forever" ? I would expect this correlation to run well below 100 ms (and with well optimized code, possibly below 1 ms). –  Jan 07 '18 at 10:53
  • in real time, 116 ms delay on the video input thread is too long. thats what im getting on my test – Jack Frye Jan 07 '18 at 15:32
  • I don't see where you are using the output of the correlation in the code you posted. How and why do you use it? Could you use something simpler like a MSE instead? Or the correlation of the subsampled images? Note also that correlation is computed through the DFT. You could save the DFT of `last_image` so you only compute it once (now you compute it twice for each image, since each image changes between being the left and right operand). – Cris Luengo Jan 07 '18 at 15:51
  • You are right I should not be doing correlation, I should be doing summation of point-by-point differences. This takes significantly less time to compute' – Jack Frye Jan 07 '18 at 16:15

1 Answers1

1

I am not sure if it is feasible, but you could make the operation faster if you correlate after converting to grayscale. This will make the correlation on a lot less data, and might actually be possible with decent FPS.

Another approach might be to correlate each channel independently, after a quick test, it might execute faster.

Alternatively, for a multithreaded approach you could correlate each of the RGB images on a different thread. How faster that would be, if faster at all, is left to find out with experiment. The link from jwpfox's comment might be a good start. However, last I was working with multiprocessing in Python, there was an issue of it copying the memory to each of the child processes. I do not know if this multiprocessing will do the same, but it is something to keep in mind for a future speed up.

dtasev
  • 540
  • 6
  • 12
  • im considering using a AWS gpu processing scheme. There will be some latency related to the ethernet upload/download but it should still operate more quickly. will keep you updated. – Jack Frye Jan 07 '18 at 04:13