Get iplImage or Mat from directshow to opencv

Question

I had to change to directshow for my eyetracking software due to the difficulties to change resolution of the camera when using c++ and opencv.

Directshow is new to me and it is kind of hard to understand everything. But I found this nice example that works perfectly for capturing & viewing the web cam.

http://www.codeproject.com/Articles/12869/Real-time-video-image-processing-frame-grabber-usi

I am using the version that not requires directShow SDK. (But it is still directshow that is used in the example, right??)

#include <windows.h>
#include <dshow.h>

#pragma comment(lib,"Strmiids.lib")

#define DsHook(a,b,c) if (!c##_) { INT_PTR* p=b+*(INT_PTR**)a;   VirtualProtect(&c##_,4,PAGE_EXECUTE_READWRITE,&no);\
                                          *(INT_PTR*)&c##_=*p;   VirtualProtect(p,    4,PAGE_EXECUTE_READWRITE,&no);   *p=(INT_PTR)c; }


// Here you get image video data in buf / len. Process it before calling Receive_ because renderer dealocates it.
HRESULT ( __stdcall * Receive_ ) ( void* inst, IMediaSample *smp ) ; 
HRESULT   __stdcall   Receive    ( void* inst, IMediaSample *smp ) {     
    BYTE*     buf;    smp->GetPointer(&buf); DWORD len = smp->GetActualDataLength();
    HRESULT   ret  =  Receive_   ( inst, smp );   
    return    ret; 
}

int WINAPI WinMain(HINSTANCE inst,HINSTANCE prev,LPSTR cmd,int show){
    HRESULT hr = CoInitialize(0); MSG msg={0}; DWORD no;

    IGraphBuilder*  graph= 0;  hr = CoCreateInstance( CLSID_FilterGraph, 0, CLSCTX_INPROC,IID_IGraphBuilder, (void **)&graph );
    IMediaControl*  ctrl = 0;  hr = graph->QueryInterface( IID_IMediaControl, (void **)&ctrl );

    ICreateDevEnum* devs = 0;  hr = CoCreateInstance (CLSID_SystemDeviceEnum, 0, CLSCTX_INPROC, IID_ICreateDevEnum, (void **) &devs);
    IEnumMoniker*   cams = 0;  hr = devs?devs->CreateClassEnumerator (CLSID_VideoInputDeviceCategory, &cams, 0):0;  
    IMoniker*       mon  = 0;  hr = cams->Next (1,&mon,0);  // get first found capture device (webcam?)    
    IBaseFilter*    cam  = 0;  hr = mon->BindToObject(0,0,IID_IBaseFilter, (void**)&cam);
                               hr = graph->AddFilter(cam, L"Capture Source"); // add web cam to graph as source
    IEnumPins*      pins = 0;  hr = cam?cam->EnumPins(&pins):0;   // we need output pin to autogenerate rest of the graph
    IPin*           pin  = 0;  hr = pins?pins->Next(1,&pin, 0):0; // via graph->Render
                               hr = graph->Render(pin); // graph builder now builds whole filter chain including MJPG decompression on some webcams
    IEnumFilters*   fil  = 0;  hr = graph->EnumFilters(&fil); // from all newly added filters
    IBaseFilter*    rnd  = 0;  hr = fil->Next(1,&rnd,0); // we find last one (renderer)
                               hr = rnd->EnumPins(&pins);  // because data we are intersted in are pumped to renderers input pin 
                               hr = pins->Next(1,&pin, 0); // via Receive member of IMemInputPin interface
    IMemInputPin*   mem  = 0;  hr = pin->QueryInterface(IID_IMemInputPin,(void**)&mem);

    DsHook(mem,6,Receive); // so we redirect it to our own proc to grab image data

    hr = ctrl->Run();   

    while ( GetMessage(   &msg, 0, 0, 0 ) ) {  
        TranslateMessage( &msg );   
        DispatchMessage(  &msg ); 
    }
};

The method HRESULT Receive is called for every new frame from the cam. the the comments says that buf contains the data. But I have 3 problems/questions.

I cant include the opencv lib. I create a new project in visual studio, and add the same property sheets as I always include. the only difference from earlier projects is that I Now create a totaly empty project, earlier I created a win32 application. How to add opencv into the directshow project?
The example above. from buf. which is a pointer to the data. How do I get that into iplImage/Mat for the opencv calc?
Is there a way to not show the images from the webcam (I only need to perform some algorithms on the frames, I guess removing the window with the results might give me more power for the analyse algorithms?!)

Thanks!

The snippet you took as a sample is terrible: use of raw pointers, hook instead of normal API, no stop/termination code. You will quickly get nowhere with this. — Roman R., May 10 '13 at 10:00
Thanks for your comment. yeah I have read that this example is terrible. But the thing is that this is the only example that I found that I in some way understand and that is working. This COM stuff & directshow is totally new for me and I think it is really hard to understand. If you know any example with better code (that is a bit easier to understand) please give me a link. — Easyrider, May 10 '13 at 10:10
But I really need to learn, and to learn to build GOOD code. If there is anyone that have nice examplecode please let me know! I've been working with directshow for a couple of days only and need to learn. — Easyrider, May 10 '13 at 10:11

score 5 · Accepted Answer · edited May 23 '17 at 10:28

With DirectShow you typically create a pipeline, that is a graph and you add filters to it, like this:

Camera -> [possibly some extra stuff] -> Sample Grabber -> Null Renderer

Camera, Sample Grabber, Null Renderer are all standard components shipped with clean Windows. Sample Grabber can be set to call you back via ISampleGrabberCB::SampleCB and give you data for every video frame captured. Null Renderer is the termination of pipeline without displaying video on monitor (just video capture).

SampleCB is the keyword to bring you sample code you need. Having data received with this call, you can convert/wrap it into IPL/OpenCV class as suggested by @praks411.

Having it done as simple as this, you don't need DirectShow BaseClasses, and the code will be merely regular ATL/MFC code and project. Make sure to use CComPtr wrapper class to deal with COM interfaces to not lose references and leak objects. Some declarations might be missing in very latest Windows SDK, so you need to either use Windows SDK 6.x or just copy missing parts from there.

See also:

How to capture frames using Delphi/DSPack without displaying it on TVideoWindow? (Delphi code, but good description and figures)
DirectShow: Examples for Using SampleGrabber for Grabbing a Frame and Building a VU Meter
SetLifeCamStudioResolutionSample - A small DirectShow project showing how to set capture up, including resolution on camera, and Sample Grabber, and also missing SDK declarations; related Q is Can't make IAMStreamConfig.SetFormat() to work with LifeCam Studio
Building the Filter Graph on Sample Grabber and Null Renderer

Thank you very much! I looked into the "setlifecam...", and maybe it is you who wrote that piece of code? Some error occured when running the code, when setting the resolution, but I fixed that with first identieng available settings and then choose one of it. But it is one thing I dont understand. I understand what a null renderer is, (i think), but I cant find anything in the code that feels like its a null renderer. Can you tell me where it is? Or is it:pCaptureGraphBuilder->RenderStream(&PIN_CATEGORY_CAPTURE, NULL, pCurrentOutputPin, NULL, pBaseFilter) ? — Easyrider, May 10 '13 at 12:29
This piece of code (SetLife...) indeed does not use Null Renderer (I just gave you what was small enough to get familiar with, and readily available, and also close to your original Q of changing resolution). You will need to look up a Null Renderer snippet somewhere else, e.g. here [Building the Filter Graph](http://msdn.microsoft.com/en-us/library/windows/desktop/dd407288(v=vs.85).aspx#build_the_filter_graph) - also has a good description. — Roman R., May 10 '13 at 12:48

score 1 · Answer 2 · answered May 10 '13 at 08:06

I think you can include opencv in existing. I've done that for console application. You will need to include path to opencv headers and path to opencv lib in property page for you current project.

Go to project property: 1.To addheaders C/C++ -----> Additional Include Directories ---> Here add opencv include directories (You may want to include multiples directories)

To add libs Linker -----> Additional Library Directories ----> Here add opencv lib.

To create IplImage from buf. You can use following once you have the height and width of image.

IplImage *m_img_show;
CvSize cv_img_size = cvSize(m_mediaInfo.m_width, m_mediaInfo.m_height);
            m_img_show = cvCreateImageHeader(cv_img_size, IPL_DEPTH_8U,3);
            cvSetData(m_img_show, m_pBuffer, m_mediaInfo.m_width*3);

I think preview of image is quite helpful. It seems that your filter above take data from renderer. If you do want you may want to change your renderer and use it in windowless mode. Other option could be to use sample grabber filter.

Thank you, I will test this during the day. About the project properties. I use the same property files that I use for my other opencv project. they have all the include paths and so, but still I get the red underline when I include the opencv libs. like, "cv.h". — Easyrider, May 10 '13 at 09:29
I think you will have to recheck the path which is included. In my case I've include following path in c/c++ include C:\OpenCV-2.4.3.bak\include. So I use following in my main file #include — praks411, May 10 '13 at 09:45
I dont know really what is wrong, the "system build path" is different from this project and mo earlier ones. (right click include file -> open document -> error message shows paths.) Have to look deeper into this i guess.. Praks, thanks for your help! — Easyrider, May 10 '13 at 10:23

Get iplImage or Mat from directshow to opencv

2 Answers2

Linked