1

Info seems to be scarse, hoping someone can point me to a sdk, libary, code to get the infra frame from the hello camera in the surface pro.

Does opencv support this?

More info the camera is Intel AVStream Camera 2500 as listed in the device manager of the surface pro.

rukiman
  • 597
  • 10
  • 32
  • I already support grabbing infra red frames using the realsense intel sdk for the realsense SR300 camera. Does anyone know whether there is such a sdk for the Intel 2500 camera? – rukiman Feb 02 '20 at 22:49

1 Answers1

2

To my best knowledge Media Foundation API has no support for infrared cameras. Microsoft did not update the API to extend it to such inputs even though it is technically possible when it comes to undocumented.

You can read infrared frames through a newer API offered for UWP development: Process media frames with MediaFrameReader, the keyword there is this: MediaFrameSourceKind.Infrared. This API is built on top of Media Foundation and Sensor APIs and gets you infrared cameras even though underlying Media Foundation alone has no equivalent public interface.

Given that this is UWP API, you might have troubles fitting this all together with OpenCV if you need the latter. UWP/OpenCV bridging might be on help there: Create a helper Windows Runtime component for OpenCV interop.

Since OpenCV is supposedly interfacing directly to traditional Windows APIs, DirectShow and Media Foundation, it is highly unlikely that it is capable of capturing infrared stream out of the box, unless, of course, the driver itself represents it as normal video. "Proper" markup on Surface Pro as infrared, thus, hides sensor from the mentioned APIs and, respectively, OpenCV.

Roman R.
  • 68,205
  • 6
  • 94
  • 158
  • Is it possible to call the UWP API from a WPF .NET app? – rukiman Feb 02 '20 at 22:45
  • See https://stackoverflow.com/questions/33692898/is-it-possible-to-use-uwp-apis-in-a-wpf-app – Roman R. Feb 02 '20 at 23:48
  • I saw that before I posted the comment. I tried it but couldnt find the Windows.Devices.Perception namespace via the DLL they were referring to. – rukiman Feb 04 '20 at 23:23
  • Oh, I just noticed the namespace is different in your MediaFrameReader..I will give that a go again. – rukiman Feb 04 '20 at 23:24
  • I can confirm using the trick these APIs are available to .NET. Will now try to get the IR frame. – rukiman Feb 20 '20 at 02:28
  • I have managed to get the sample CameraFrames app to run but no luck getting the infra red frame. I have created a new question for this https://stackoverflow.com/questions/60333417/depth-and-infrared-frame-on-a-surface-pro – rukiman Feb 21 '20 at 06:48
  • The format was NV12 which the CameraFrames sample app doesn't support, I modified it and now I can see the infra red frames. I'm thinking windows.devices.perception seems like a more higher level API for what I want to do. Is this recommended API to get the infra frames? – rukiman Feb 24 '20 at 02:11
  • It is weird that infrared frame is NV12, as far as I remember it's grayscale, that is, with one color component in it. – Roman R. Feb 24 '20 at 08:07
  • Yes the image is in grayscale though when I converted from NV12 to ARGB8 to display it. Any idea why there is no depth frame ? I would have though it would have been generated. Because there is none, I guess my only choice is to look at the infrared and use something like the more whiter the closer the object it. But I wouldn't know the depth. – rukiman Feb 24 '20 at 11:28
  • As far as I remember, there is Infrared camera on Surface Pro ("Microsoft IR Camera Front"), but there is no depth camera. – Roman R. Feb 24 '20 at 12:11
  • Regarding NV12 - I checked my data, IR camera offers NV12 and IANV formats at low level. UWP API seems to be ignoring IANV and exposes NV12 only. So yes, you perhaps have NV12 as the only option, and then UV components there are probably expected to be zeroed. – Roman R. Feb 24 '20 at 12:16
  • Thanks for checking. Is it possible to generate or figure out changes in depth by looking at an IR frame? Otherwise my only hope of liveliness detection would be to check for a face in the IR frame as well as the RGB frame. – rukiman Feb 25 '20 at 10:43
  • There is clearly no depth data there. It takes to have different technology (depth camera = "time of flight" camera), so with RGB + IR you just have an extra channel of information to help you, for example, identify shapes and segment the image without 3D data. – Roman R. Feb 25 '20 at 11:01
  • OK i have obtained the infra and rgb frames. But I am a bit stubbed how to use the IR frame for liveliness. If I hold up a photo of a face, I can clearly see the face photo in the IR frame. Is Windows Helo possibly getting a depth frame somehow but it isn't exposed in the surface pro via the APIs? – rukiman Feb 27 '20 at 05:54