Save frames extracted from Deepstream pipeline to video using OpenCV

Question

I’m trying to save extracted frames from a Deepstream pipeline to video with OpenCV but all I end up with is a 9KB file.

This is my code (executed inside a probe function):

batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_frame = batch_meta.frame_meta_list
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
frame = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)
frame_copy = np.array(frame, copy=True, order='C')            
frame_copy = cv2.cvtColor(frame_copy, cv2.COLOR_RGBA2BGRA)

The above code is executed each time the probe function is invoked. Images are saved to a queue:

frame_buffer.put(frame_copy)

After the required number of frames has been pushed into the queue, I use below code to save the buffered frames to a video file:

codec = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('out.avi', codec, fps, (output_width, output_height))
out.write(frame_copy)                
total_frames = FRAME_RECORDING_THRESH 

while total_frames > 0:                
   frame = frame_buffer.get() 
   frame = cv2.resize(frame, (output_width, output_height), interpolation = cv2.INTER_LINEAR)  
   out.write(frame)
   total_frames -= 1
                
out.release()

Unfortunately the file produced is not a valid video file. Is there sth I am doing wrong in the above process? Any help would be greatly appreciated.

P.S. Just to test that the frames have been correctly stored inside the queue, if I attempt to save the frames as images inside the while loop:

cv2.imwrite(dest_folder + '/' + f'tmp{total_frames}.png', frame)

I get properly saved and valid png images.

P.S. 2 Frames have a resolution of output_width, output_height at the time they are buffered. Also, trying to do a cv2.resize before they are saved doesn't change anything.

if `frame` is different than `(output_width, output_height)` then you have to `resize()` it - `writer` doesn't resize frames to size `(output_width, output_height)` but it skips frames which have different size - and finally you can get file with only header infromation but without frames. — furas, Jun 03 '22 at 17:29

furas · Accepted Answer · 2022-06-04T10:14:14.053

2

I can't test it but common mistake is that people think than code

out = cv2.VideoWriter('out.avi', codec, fps, (output_width, output_height))

will automatically resize frames to size (output_width, output_height) but it is not true.

You have to manually resize frames.

If you don't do this then Writer will skip frames and you get broken file - without frames.

while total_frames > 0:                
   frame = frame_buffer.get() 

   frame = cv2(frame, (output_width, output_height))

   out.write(frame)
   total_frames -= 1

EDIT:

It seems problem can make image with transparent layer A - RGBA - because video don't use A.

It needs to remove it.

You can convert with cv2.COLOR_RGBA2BGR instead of cv2.COLOR_RGBA2BGRA

frame_copy = cv2.cvtColor(frame_copy, cv2.COLOR_RGBA2BGR)

Or you can remove last layer from numpy.array

frame_copy = frame_copy[ : , : , :3 ]

edited Jun 04 '22 at 10:14

answered Jun 03 '22 at 17:37

furas

134,197
12
106
148

The images, when buffered, have the same resolution to the one set in `cv2.VideoWriter`. Also, trying to resize them before saving makes no difference. – Giorgos Betsos Jun 03 '22 at 18:23
I can't test your code with your data so I have no idea what is the problem. I can only guess. Don't you get any error message when you run code? Did you tried to use `print()` to see if it runs `while`-loop? maybe `total_frames` is zero and it never run loop. – furas Jun 03 '22 at 21:04
Yes, the loop executes. I even tried to save each frame as image, then load the image with `c2.imread` then save with `out.write` and I got a video file! – Giorgos Betsos Jun 03 '22 at 21:23
I can't reproduce you problem - so I can't help. And without error mesage I don't see what can makes problem. – furas Jun 03 '22 at 21:27
do you imwrite/imread as `RGB` or `RGBA` ? Video rather can't use `A` (transparency) - and maybe this makes problem to write it directly. – furas Jun 03 '22 at 21:29
Hmm, the shape I get for the frame is 1920, 1080, 4. This is what I am using to write the video. Do you think this might be the problem? – Giorgos Betsos Jun 03 '22 at 21:42
I think I have to rewrite `frame_copy = np.array(frame, copy=True, order='C')` `frame_copy = cv2.cvtColor(frame_copy, cv2.COLOR_RGBA2BGRA)` some how in order to get rid of A? Is there a way to do this (some numpy operation maybe)? – Giorgos Betsos Jun 03 '22 at 21:50
there is `cv2.COLOR_RGBA2BGR` without `A` at the end so it should remove transparency. OR: every frame is `numpy.array` so you can get `frame[:,:,:]` to get only 3 layers - without layer `A` – furas Jun 03 '22 at 22:53
Using `cv2.COLOR_RGBA2BGR` solved the issue, thanks! If you edit your answer to add this then I will mark it as an answer. – Giorgos Betsos Jun 04 '22 at 07:39

Save frames extracted from Deepstream pipeline to video using OpenCV

1 Answers1