Run IVMRWindowlessControl9.GetCurrentImage() in background

Question

I have a webcam control using DirectShow.NET. I've created a custom control to display video and capture an image from a webcam. I'm using that custom control in another WPF window. I have a function public Bitmap CaptureImage() in the custom control to abstract away a little bit of the DirectShow programming and simply return a Bitmap. Since the images are relatively large (1920x1080), the GetCurrentImage() function of the IVMRWindowlessControl9 takes quite a while to process (2-3 seconds). I've stepped through my code and can confirm that this call is the only one that takes a long time to process.

Because of this, the GUI thread in my main WPF window hangs, causing it to be unresponsive for a few seconds, so if I want to display a progress spinner while the image is being captured, it will just remain frozen.

Here is the code for CaptureImage():

public Bitmap CaptureImage()
{
  if (!IsCapturing)
    return null;

  this.mediaControl.Stop();
  IntPtr currentImage = IntPtr.Zero;
  Bitmap bmp = null;

  try
  {
    int hr = this.windowlessControl.GetCurrentImage(out currentImage);
    DsError.ThrowExceptionForHR(hr);

    if (currentImage != IntPtr.Zero)
    {
      BitmapInfoHeader bih = new BitmapInfoHeader();
      Marshal.PtrToStructure(currentImage, bih);

      ...
      // Irrelevant code removed 
      ...

      bmp = new Bitmap(bih.Width, bih.Height, stride, pixelFormat, new IntPtr(currentImage.ToInt64() + Marshal.SizeOf(bih)));
      bmp.RotateFlip(RotateFlipType.RotateNoneFlipY);
    }
  }
  catch (Exception ex)
  {
    MessageBox.Show("Failed to capture image:" + ex.Message);
  }
  finally
  {
    Marshal.FreeCoTaskMem(currentImage);
  }

  return bmp;
}

In order to fix this, I've tried to run this as a background task as follows:

public async void CaptureImageAsync()
{
  try
  {
    await Task.Run(() =>
    {
      CaptureImage();
    });
  }
  catch(Exception ex)
  {
    MessageBox.Show(ex.Message);
  }
}

I've tried multiple ways to do this including using BackgroundWorkers, but it seems that any time I make this call asynchronously, it creates this error:

Unable to cast COM object of type 'DirectShowLib.VideoMixingRenderer9' to interface type 'DirectShowLib.IVMRWindowlessControl9'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{8F537D09-F85E-4414-B23B-502E54C79927}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).

The error always happens when on this line:

int hr = this.windowlessControl.GetCurrentImage(out currentImage);

Calling CaptureImage() synchronously yields normal results. The image is captured and everything works as intended. However, switching to using any kind of asynchronous functionality gives that error.

Answer 1

You are having two issues here. First of all the original slowliness of the API is behavior by design. MSDN mentions this as:

However, frequent calls to this method will degrade video playback performance.

Video memory might be pretty slow for read-backs - this is the 2-3 second processing problem, not the resolution of the image per se. Bad news is that it is highly likely that even polling snapshots from background thread is going to affect the visual stream.

This method was and is intended for taking sporadic snapshots, esp. those initiated by user interactively, not automated. Applications needing more intensive and automated, and those not affecting visual feed snapshots are supposed to intercept the feed before it is sent to video memory (there are options to do it, and most popular yet clumsy is use of Sample Grabber).

Second, you are likely to be hitting .NET threading issue described in this question, which triggers the mentioned exception. It is easy to use the same interface pointer in native code development by sneaky violating COM threading rules and passing the interface pointer between apartments. Since CLR is adding a middle layer between your code and COM object doing additional safety checks, you can no longer operate with the COM object/interfaces from background thread because COM threading rules are enforced.

I suppose you either have to keep suffering from long freezes associated with direct API calls, or add native code development that helps to bypass the freezes (especially, for example, a helper filter which captures frames before sending to video memory and in the same time implementing helper functionality for .NET caller to support background thread calls). Presumably you can also do all DirectShow related stuff in a pool of helper MTA threads which solve you the background thread caller problem, but in this case you will most likely need to move this code from your UI thread which is STA - I don't think it something people often, if at all, doing.