How to minimize the lag from the live stream using OpenCV/EmguCV WPF

Question

I am using EmguCV to get a live stream from a high-resolution webcam. (full HD) The issue is that there is a significant lag in the video. I compared with the Windows camera app and my application is much more delayed.

Here is the code snippet I am using to get the live stream and show on the canvas. I am wondering if I am missing anything important to minimize the lag. If anybody is experienced, please help me.

void init_camera()
{
    m_capture= new VideoCapture(0);
    m_capture.ImageGrabbed += ProcessFrame;
}

void ProcessFrame()
{
    if (m_capture_in_progress)
        return;
    m_capture_in_progress = true;
    if (m_capture != null && m_capture.Ptr != IntPtr.Zero)
    {
        m_capture.Retrieve(m_frame, 0);
        if (m_frame != null && m_frame.Bitmap != null)
        {
            if (IsRecording && m_video_writer != null && m_video_writer.IsOpened)
            {
                try
                {
                    m_video_writer.Write(m_frame);
                }
                catch (Exception ex)
                {
                    log(ex.Message + "\n" + ex.StackTrace);
                }
            }

            this.Dispatcher.Invoke(() =>
            {
                using (var stream = new MemoryStream())
                {
                    m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
                    using (var outstream = new MemoryStream(stream.ToArray()))
                    {

                        BitmapImage bitmap = new BitmapImage();
                        bitmap.BeginInit();
                        bitmap.StreamSource = new MemoryStream(stream.ToArray());
                        bitmap.CacheOption = BitmapCacheOption.OnLoad;
                        bitmap.EndInit();

                        ui_canvas.Background = new ImageBrush(bitmap);
                    }
                };
            });
        }
    }
    m_capture_in_progress = false;
}

have you tried [restricting VideoCapture's internal buffer](https://stackoverflow.com/questions/30032063/)? — zteffi, Jan 15 '20 at 10:49
I used a different approach and did not face any lag. Instead of ImageGrabbed handler, I had my own DispatcherTimer which ticked 25 times a second (25 FPS). On each tick I got the Frame and used its bitmap for updating the ImageSource of an image control. This did not require Dispatcher.Invoke too. — Insane, Jan 15 '20 at 11:13
I tried both methods but still see the lag... Basically I want the stream to have the same lag as the default windows camera app.. — Vladimir Dinic, Jan 15 '20 at 12:51
You could try rendering the image into an `Image` control instead of canvas. Also try modifying emgu capture parameters. And also you could try to clean `ProcessFrame` avoiding lots of `if` statements — Oscar Martinez, Jan 15 '20 at 14:21
For now, I am rendering to '''Image''' control. For the related capture parameter I use:Emgu.CV.CvEnum.CapProp.Buffersuze <- 3 Emgu.CV.CvEnum.CapProp.Fps <- 24 m_capture.SetCaptureProperty(Emgu.CV.CvEnum.CapProp.Fps, 24); — Vladimir Dinic, Jan 15 '20 at 14:25

Oscar Martinez · Answer 1 · 2020-01-15T15:17:36.010

OP is referring some kind of laggs while drawing a camera capture stream from EmguCv into canvas wpf control. OP also states that lagg disappears when streaming with third party software.

This is caused by low performance in OP's code.

After wataching the posted code sample I would suggest the following improvements:

Increasing FPS rate from EmguCV parameters

Maybe your device is supporting higher fps rates, try to increase this value.

Drawing the bitmap over another control like image

As far as I know, drawing a bitmap over a canvas is slower than drawing it over an imagecontrol. A good solution would be to overlap both controls so you can paint the frames over the imageand draw your shapes over the canvasas they are well overlapped.

Cleaning up the rendering method

Keep simple and try to avoid so much control structures when you are inside a critical program block. Specially in program blocks that are executed when an external devices fires an event.

Fixing possible program lock

As far as I can see, you are locking the ProcessFrame event with a bool variable called m_capture_in_progress which is set to true while you are drawing the frame and it is freed after you finish the drawing. This can cause that next incoming frames are not painted because the method is still blocked by the previous frame. This can cause a low performance issue as many frames get lost and not painted.

Quergo · Answer 2 · 2020-01-27T22:29:54.187

Use the Image control but set its Source Property to a WriteableBitmap instead of a BitmapImage. Then lock the bitmap, copy the pixel data from your EmguCV Mat to the backbuffer of the WriteableBitmap, call AddDirtyRect method and finally unlock the Bitmap. The final unlock call in combination with the AddDirtyRect will trigger a redraw of the UI image.

Adavantages:

It does not generate a new BitmapImage object each time you want to draw a new frame.
You copy the pixel data only once

Your copy, encode and decode data to often:

 // Copies and encodes pixel data to jpeg stream
 m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);

 // Copies jpeg stream
 using (var outstream = new MemoryStream(stream.ToArray()))
 {

     BitmapImage bitmap = new BitmapImage();
     bitmap.BeginInit();

     //Copies jpeg stream again
     bitmap.StreamSource = new MemoryStream(stream.ToArray());

     bitmap.CacheOption = BitmapCacheOption.OnLoad;

     // Triggers jpeg stream decoding
     bitmap.EndInit();
     ui_canvas.Background = new ImageBrush(bitmap);
  }

How to minimize the lag from the live stream using OpenCV/EmguCV WPF

2 Answers2