3

I want to share my captured frame in OpenVC with my multiprocessing subprocess but video_capture.read() creates a new object and doesnt write in to my numpy array that iam going to share by wrapping it with multiprocessing.Array()

Here is the code:

ret, frame = video_capture.read()
shared_array = mp.Array(ctypes.c_uint16, frame.shape[0] * frame.shape[1], lock=False)
while True:
    b = np.frombuffer(shared_array)
    ret, b = video_capture.read()

But the buffer b gets overridden by the read() function. So i dont write in to my buffer/shared array.

In subprocess i do following:

img = np.frombuffer(shared_array)
cv2.imshow('Video', img)

The shared array only containing the first frame of the video. How can i correctly write into the shared array to be able to read every frame in subprocess? I dont want to use queues or pipes. Shared memory is the way to go coz more processes will consume the frames.

Frey
  • 85
  • 1
  • 9
  • 1
    Maybe if you do `ret, b[:] = video_capture.read()`? (also isn't it unnecessary to call `np.frombuffer` constantly on every loop iteration?) – jdehesa Mar 09 '18 at 10:30
  • Thank you! I tried it and i think its the right way to go. But it was saying: "could not broadcast input array from shape (720,1280,3) into shape (691200)". So i assigned the shape of the original opencv frame to the buffer (shared array) but then its saying "ValueError: cannot reshape array of size 691200 into shape (720,1280,3)". Do you have an idea how i can fix the size of the array? – Frey Mar 09 '18 at 10:49
  • the shared_array seems to be flat – Frey Mar 09 '18 at 10:52

1 Answers1

4

There are a couple of issues, one is the size and shape of the shared array and another is how you access it. To solve the first problem, first you need to make sure that the size of the created array corresponds to the size of a video frame. You have read a frame and use its width and height (although you can do it without reading any frame), but did not account for its number of channels.

ret, frame = video_capture.read()
shape = frame.shape
shared_array = mp.Array(ctypes.c_uint16, shape[0] * shape[1] * shape[2], lock=False)

You chose uint16 as data type, which is fine (again, you can use the video_capture.get(cv2.CAP_PROP_FORMAT) to get the exact data type of the frames, but you can choose whatever you want, since NumPy will convert the values to the array data type). However, when you create the NumPy array you must specify that is the data type you want to use, otherwise it will use float64 by default:

b = np.frombuffer(shared_array, dtype=np.uint16)

And then you have to reshape it to have the shape of a frame (note that this does not create a new NumPy array, it's just a view, so it still uses the shared buffer):

b = b.reshape(shape)

Finally, when you read the frame, you don't want to overwrite the b variable, but rather write to the array. Also, you should only do that when a frame was actually read:

while True:
    ret, frame = video_capture.read()
    if ret:
        b[:] = frame
jdehesa
  • 58,456
  • 7
  • 77
  • 121
  • 1
    Thank you. This makes sense :) I still have a little issue. When i display the image in my subprocess the frame is black. `b = np.frombuffer(self.shared_array, dtype=np.uint16)` `b = n.reshape(self.shape)` `cv2.imshow('Video', b)` `if cv2.waitKey(1) & 0xFF == ord('q'):` `break` `print(b)` The print(b) print the correct array. The output matrix is filled with numbers but the window stays black. – Frey Mar 09 '18 at 11:35
  • @Frey I'm not sure why that can be, I don't know much about OpenCV in Python, it seems like a different issue. – jdehesa Mar 09 '18 at 11:45
  • 1
    Thank you. Its works with np.uint8 but not with uint16. i think the captured frame has 8bit. I couldnt read the actual dtype from video_capture coz the enum is not known ("module 'cv2.cv2' has no attribute 'CV_CAP_PROP_FORMAT'") – Frey Mar 09 '18 at 11:52
  • @Frey Right, sorry, seems like it was `cv2.cv.CV_CAP_PROP_FORMAT` (see docs of [`VideoCapture::get`](https://docs.opencv.org/2.4.2/modules/highgui/doc/reading_and_writing_images_and_video.html#videocapture-get) and [`Mat::depth`](https://docs.opencv.org/2.4.13.6/modules/core/doc/basic_structures.html#int%20Mat::depth()%20const) for returned values, `cv2.cv.CV8U`, etc.). Glad you fixed the problem anyway. – jdehesa Mar 09 '18 at 12:00
  • (btw OpenCV HTTPS cert seems to be expired so browser shows a warning when accessing their website...) – jdehesa Mar 09 '18 at 12:01
  • Sorry to bother you. Then it is saying 'cv' is no attribut of cv2. I just imported cv as normal via `import cv2` – Frey Mar 09 '18 at 12:04
  • @Frey Ohh, sorry, you're on OpenCV 3, several [changes](https://docs.opencv.org/master/db/dfa/tutorial_transition_guide.html) apply, as pointed out [here](https://stackoverflow.com/q/33177376) or [here](https://stackoverflow.com/q/30013009). Correct way: `video_capture.get(cv2.CAP_PROP_FORMAT)`, returned value should be something like `cv2.CV_8UC3`, or one of the other format constants (see [pixel formats docs](https://docs.opencv.org/3.0-last-rst/modules/core/doc/intro.html#fixed-pixel-types-limited-use-of-templates)). This time I checked myself these exist in OpenCV 3.3.0! – jdehesa Mar 09 '18 at 12:33
  • Thank you! you saved my day btw :) – Frey Mar 09 '18 at 15:26
  • @Frey, sorry for bother but how can you fix this issues. In my case, video_capture.get(cv2.CAP_PROP_FORMAT) returns 16. So I still use np.uint16. I can get frame but it's black. – Jimmy Jul 05 '21 at 13:17