Test Environment
- OS: Windows 11 Home 22H2
- IR Camera: NexiGo HelloCam N930W Camera
- IDE: Visual Studio 2202
- programming language: C++ 20
- Application Kinds: Windows Desktop Application (C++/Winrt) [not universal Windows App][Xaml UI].
- Source Code: https://github.com/andresbeltranc/ir_camera_sample_win32
Context
The Windows Hello cameras use a IR camera and an IR emitter, and combine the 2 to sense depth in the frame. Its similar to Apple's FaceID or the tech in the Xbox Kinect. Accessing the depth sensing capabilities of the Windows Hello cameras could be incredibly valuable for researchers and app developers.
I was able to create an application which is able to access the IR camera embedded in Windows Hello enabled webcams. However I've found that the IR emitter has inconsistent behavior. I've posted my code in a github repository which I linked above.
Problem
After I was able to access each frame of the IR Camera, I realized that my camera was not turning on the Infrared Torch (Camera IR Emitter LED). But I know that the Infrared Torch works because when I use the Facial Recognition of Windows Hello the Infrared Torch does turn on.
I found that an IR Camera for Windows at a Driver level has the followings Modes for the infrared Torch:
| Tourch Mode | Description |
|---|---|
| KSCAMERA_EXTENDEDPROP_IRTORCHMODE_OFF | Off |
| KSCAMERA_EXTENDEDPROP_IRTORCHMODE_ALWAYS_ON | Always on |
| KSCAMERA_EXTENDEDPROP_IRTORCHMODE_ALTERNATING_FRAME_ILLUMINATION | On for every other frame |
Note: The behavior of the Infrared Torch depends a lot on the camera that is being used. I have been able to test several IR cameras and some cameras have the default value of On for every other frame but in others, such as the NexiGo HelloCam N930W Camera, the default value is Off, which requires it to be turned on in some way.
These values can be changed in theory by the Media Foundation API using InfraredTorchControl Class from the VideoDeviceController Class that can be accessed from the MediaCapture Class as follows:
auto infraredTorchControl = m_mediaCapture.VideoDeviceController().InfraredTorchControl();
if(infraredTorchControl.IsSupported()){
infraredTorchControl.CurrentMode(winrt::Windows::Media::Devices::InfraredTorchMode::On);
}
But the problem starts in this part of the code. When I tried to access the InfraredTorchControl the API says that it's not supported. I read through the VideoDeviceController Class documentation, and it states that some drivers may require the camera device to be in a preview state before it can determine which controls are supported. So I tried to solve this problem in several different ways:
Windows Desktop Application (C++/WinRT)[not universal Windows App][XAML UI]
First I developed a WinRT application without a UWP interface. Since the VideoDeviceController documentation said that the camera needs to be in a preview state I tried to start this preview state with the MediaCapture Class using the StartPreviewAsync(). But the problem here is that I need to set a source/sink(MediaCapture Class) to a CaptureElement Class so I tried to instantiate the CaptureElement Class and set the source to meet that requirement.
#include "winrt/Windows.UI.Xaml.Controls.h";
using namespace winrt::Windows::UI::Xaml::Controls;
CaptureElement captureElement;
captureElement.Source(m_mediaCapture);
m_mediaCapture.StartPreviewAsync();
m_mediaCapture.StopPreviewAsync();
but it doesn't work, it throws the following error:
WinRT originate error: 'Input format unsupported'.
WinRT originate error: 'No suitable transform was found to encode or decode the content.'.
The most likely reason for why this error is happening is that the CaptureElement does not support the format delivered by the IR camera, which is L8(An 8-bit-per-pixel, fixed-point pixel format in which the sole component is luminance.). If anyone knows how to fix this problem, please comment.
Blank App (C++/WinRT)[universal windows App][XAML UI]
I thought this error may be popping up because I was developing a WinRT application without a UWP interface, so I tried to implement a simple UWP app using the Blank App (C++/WinRT) template to check if I could use the CaptureElement directly with a XAML file.
MainPage.Xaml
<StackPanel Orientation="Horizontal" HorizontalAlignment="Center" VerticalAlignment="Center">
<CaptureElement x:Name="MyPreview" Stretch="UniformToFill" />
<Button x:Name="myButton" Click="ClickHandler">Click Me</Button>
</StackPanel>
MainPage.cpp
MyPreview.Source = m_mediaCapture;
Unfortunately, the same error appeared with this type of application, so I discarded the idea that it was a problem with my original application.
WinRT originate error : 'Input format unsupported'.
WinRT originate error: 'No suitable transform was found to encode or decode the content.'.
Custom Media Sink for Use with Media Foundation API
This solution is the one I understand the least, but it seems to be the most correct for an application without a user interface. Briefly, the idea is to create a Custom media Sink that can be accepted by the MediaCapture Class functions which are the following:
StartPreviewToCustomSinkAsync(MediaEncodingProfile, IMediaExtension)StartPreviewToCustomSinkAsync(MediaEncodingProfile, String, IPropertySet)
Which in theory allows a Custom Media sink to be passed as a parameter. But the problem is that I couldn't find any documentation where it specified how to create this Custom Media sink. I only found this tutorial: https://www.codeproject.com/Tips/772038/Custom-Media-Sink-for-Use-with-Media-Foundation-To. This tutorial was using C++/CX, so I tried to convert that code to C++/WinRT unsuccessfully.
Conclusions
First, I don't understand why I need to set a media sink to be able to start the preview state of the Camera. I only need to modify the values of the camera as is the InfraredTorch mode. I don’t need to show the output of the camera at that point.
The Second conclusion that comes to my mind is that Microsoft Media Foundation Platform API doesn’t support the preview state for infrared cameras even if the media sink requirement is met. Since the CaptureElement doesn’t support the image format (L8) apparently. but I'm not sure since in the InfraredTorchControl Documentation there is an example with the CaptureElement Class.
Third, it seems that the one way to solve this problem is with a custom media sink that requires a very deep knowledge and there is almost no documentation that explains how to put all the pieces together.
Lastly, I think that as a last option, we can consider the possibility of directly accessing the driver of the IR Camera with DeviceMFT.
