Direct Show Video capture performance

Question

I originally started out looking for an example of how I can use FFMPEG in c++ builder to create an application to record from usb capture device and playback video because of apparrent poor performance

I tried Mitov components, Datastead, FFMPEGVCL and winsoft camera which use directshow but their capture performance seemed poor.

I need to capture 1920x1080 at up to 60fps into a compressed format and play this back later at both normal speed and slow speed.

What I found was that DirectShow itself has a number of limitations which can be improved by adding things like FFMPEG, but ultimately PC hardware, in particular the HDD and processor limit capture capability.

1920x1080 60fps is basically the upper end for DirectShow so you need to have best performing hardware in order to achieve this sort of performance @Spektre kindly gave me examples of DirectShow using the API direct which were good for comparison with the purchased components.

Using this and comparing to the components I found that MITOV has a major issue with regards the larger video sizes and frame rates. Using this 1920x108 30fps and 60fps can be previewed but they have a massive delay between video feed and preview (5 or 6 seconds). The other components performed similar to the API direct method with only minor variations in performance. None were able to capture and record 1920x108 60fps with any sort of compression filter without large frame drops and very jerky preview.

C++ builder is my development use so I need a solution which works with it, hence it is very much related to my problem. — Barry Andrews, Apr 25 '19 at 23:24
@UmNyobe using VCL based components is **C++ Builder** and or **Delphi** specific so the tag is relevant. — Spektre, Apr 30 '19 at 07:04

Spektre · Answer 1 · 2019-05-02T08:17:49.270

I do not think your problem is in video capture component itself nor FFMPEG. The main problem is that Directshow and VFW API to obtain image from camera is relatively slow. The speed can be improved by setting proper format of the image like:

smaller resolution
different color encoding (RGB 24bpp is not a good idea)
using camera JPEG format (not all cameras support it)

Without JPEG output format I never got pass 15 fps even on small resolutions. Also my experience shows that DirectShow is slightly slower then VFW (at least for my cameras). However not all cameras provide a VFW driver :( anymore.

Also be sure to ensure USB bandwidth by using proper version of USB port and do not diminish its bandwidth with other devices on the same HUB !!!

This is what I use for camera capture (VFW I coded years ago) in Borland/Embarcadero BDS 2006 C++:

VideoCaptureVFW.h:

//---------------------------------------------------------------------------
//--- VFW Video Capture ver: 2.0 --------------------------------------------
//---------------------------------------------------------------------------
#ifndef _VideoCaptureVFW_h
#define _VideoCaptureVFW_h
//---------------------------------------------------------------------------
#include <vfw.h>
#include <jpeg.hpp>
#include <Clipbrd.hpp>
//---------------------------------------------------------------------------
const int _vfw_callbach_onframe=1;      // bit mask for each callback
//---------------------------------------------------------------------------
#ifndef _TDirectMemoryStream
#define _TDirectMemoryStream
class TDirectMemoryStream:TMemoryStream // just for accessing protected SetPointer
    {
public:
    void SetMemory(BYTE *ptr,DWORD siz) { SetPointer(ptr,siz); Position=0; };
    };
#endif
//---------------------------------------------------------------------------
#ifndef _avgfps
#define _avgfps
class avgfps
    {
public:
    int N,N2;
    int frame,frame0;
    double fps,t0,t1,dt,Ts;
    avgfps()
        {
        N=40; N2=N<<1;
        fps=0.0; t0=0.0; frame=0; frame0=0;
        LARGE_INTEGER i;
        QueryPerformanceFrequency(&i); Ts=1.0/double(i.QuadPart);
        }
    ~avgfps() {}
    void update()
        {
        double t;
        LARGE_INTEGER i;
        QueryPerformanceCounter(&i); t=double(i.QuadPart)*Ts; dt=t-t0;
        if (frame<=0)
            {
            t0=t; t1=t;
            dt=0.0;
            frame=0;
            frame0=0;
            }
        if (dt>1e-6) fps=double(frame0)/dt; else fps=0.0;
        frame++; frame0++;
        if (frame0==N ) t1=t;
        if (frame0==N2) { t0=t1; t1=t; frame0=N; }
        }
    };
#endif
//---------------------------------------------------------------------------
class VideoCaptureVFW
    {
private:
    HWND hcap,hown;             // video capture window
public:
    int ins_ix,ins_use;         // instance index and usage for callbacks class reference
    CAPDRIVERCAPS driver_cp;    // driver capabilities
    CAPTUREPARMS  capture;      // capture setup
    CAPSTATUS     state;
    BITMAPINFO    format;
    // on frame callback
    avgfps fps;                 // average fps
    TMemoryStream *mem;         // just for loading jpg from memory without copy
    Graphics::TBitmap *bmp;     // grabbed frame

     VideoCaptureVFW();
    ~VideoCaptureVFW();

    void ins_rst();
    void ins_inc();
    void ins_dec();

    void set_owner(HWND _hown);
    AnsiString get_video_drivers();
    void set_video_driver(int ix);

    void dlg_source() { if(driver_cp.fHasDlgVideoSource) capDlgVideoSource(hcap); }
    void dlg_format() { if(driver_cp.fHasDlgVideoFormat) capDlgVideoFormat(hcap); get_state(); get_format(); }
    void dlg_display(){ if(driver_cp.fHasDlgVideoDisplay)capDlgVideoDisplay(hcap);}
    void dlg_compress() { capDlgVideoCompression(hcap); }

    void get_capabil(){ capDriverGetCaps  (hcap,&driver_cp,sizeof(CAPDRIVERCAPS)); }
    void get_setup()  { capCaptureGetSetup(hcap,&capture,sizeof(CAPTUREPARMS)); }
    void set_setup()  { capCaptureSetSetup(hcap,&capture,sizeof(CAPTUREPARMS)); }
    void get_state()  { capGetStatus      (hcap,&state,sizeof(CAPSTATUS)); }
    void get_format() { capGetVideoFormat (hcap,&format,sizeof(BITMAPINFO)); }

    void preview_start(){ capPreview(hcap,TRUE ); }
    void preview_stop() { capPreview(hcap,FALSE); }
    void grab_start()   { set_callback_on_frame(); capGrabFrameNoStop(hcap);  }
    void grab_stop()    { res_callback_on_frame(); }
    void copy_to_clipboard() { capEditCopy(hcap); }

    void set_callback_on_frame();
    void res_callback_on_frame();
    LRESULT _on_frame(HWND hwnd,LPVIDEOHDR hdr);
    void (*on_frame)(VideoCaptureVFW &cap);
    };
//---------------------------------------------------------------------------
// on frame
const int _VideoCaptureVFW_ins=32;
void* VideoCaptureVFW_ins[_VideoCaptureVFW_ins]=
    {
    NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,
    NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,
    };
int VideoCaptureVFW_ins_get()
    {
    for (int i=0;i<_VideoCaptureVFW_ins;i++)
     if (VideoCaptureVFW_ins[i]==NULL) return i;
    return -1;
    }
LRESULT PASCAL VideoCaptureVFW00_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 0]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW01_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 1]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW02_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 2]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW03_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 3]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW04_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 4]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW05_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 5]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW06_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 6]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW07_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 7]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW08_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 8]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW09_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[ 9]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW10_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[10]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW11_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[11]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW12_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[12]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW13_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[13]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW14_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[14]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW15_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[15]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW16_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[16]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW17_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[17]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW18_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[18]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW19_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[19]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW20_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[20]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW21_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[21]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW22_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[22]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW23_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[23]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW24_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[24]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW25_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[25]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW26_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[26]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW27_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[27]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW28_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[28]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW29_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[29]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW30_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[30]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL VideoCaptureVFW31_on_frame(HWND hwnd,LPVIDEOHDR hdr) { return ((VideoCaptureVFW*)(VideoCaptureVFW_ins[31]))->_on_frame(hwnd,hdr); }
LRESULT PASCAL(*VideoCaptureVFW_on_frame[_VideoCaptureVFW_ins])(HWND hwnd,LPVIDEOHDR hdr)=
    {
    VideoCaptureVFW00_on_frame,
    VideoCaptureVFW01_on_frame,
    VideoCaptureVFW02_on_frame,
    VideoCaptureVFW03_on_frame,
    VideoCaptureVFW04_on_frame,
    VideoCaptureVFW05_on_frame,
    VideoCaptureVFW06_on_frame,
    VideoCaptureVFW07_on_frame,
    VideoCaptureVFW08_on_frame,
    VideoCaptureVFW09_on_frame,
    VideoCaptureVFW10_on_frame,
    VideoCaptureVFW11_on_frame,
    VideoCaptureVFW12_on_frame,
    VideoCaptureVFW13_on_frame,
    VideoCaptureVFW14_on_frame,
    VideoCaptureVFW15_on_frame,
    VideoCaptureVFW16_on_frame,
    VideoCaptureVFW17_on_frame,
    VideoCaptureVFW18_on_frame,
    VideoCaptureVFW19_on_frame,
    VideoCaptureVFW20_on_frame,
    VideoCaptureVFW21_on_frame,
    VideoCaptureVFW22_on_frame,
    VideoCaptureVFW23_on_frame,
    VideoCaptureVFW24_on_frame,
    VideoCaptureVFW25_on_frame,
    VideoCaptureVFW26_on_frame,
    VideoCaptureVFW27_on_frame,
    VideoCaptureVFW28_on_frame,
    VideoCaptureVFW29_on_frame,
    VideoCaptureVFW30_on_frame,
    VideoCaptureVFW31_on_frame,
    };
//---------------------------------------------------------------------------
VideoCaptureVFW::VideoCaptureVFW()
    {
    hcap=NULL;
    hown=NULL;
    ins_ix=-1; ins_use=0;
    on_frame=NULL;
    mem=new TMemoryStream();
    bmp=new Graphics::TBitmap;
    }
//---------------------------------------------------------------------------
VideoCaptureVFW::~VideoCaptureVFW()
    {
    capDriverDisconnect(hcap);
    res_callback_on_frame();
    if (mem) delete mem;
    if (bmp) delete bmp;
    }
//---------------------------------------------------------------------------
void VideoCaptureVFW::set_owner(HWND _hown)
    {
    hown=_hown;
    hcap=capCreateCaptureWindow("",WS_CHILD|WS_VISIBLE,0,0,1,1,hown,1);
    }
//---------------------------------------------------------------------------
AnsiString VideoCaptureVFW::get_video_drivers()
    {
    const int _size=256;
    char drv_name[_size];
    char drv_ver[_size];
    char dev_name[_size];
    AnsiString s0,s1,list;
    int i;
    list="";
    for (i=0;;i++)
        {
        if (!capGetDriverDescription(i,drv_name,_size,drv_ver,_size)) break;
        s0=drv_name;
        s1=drv_ver;
        list+=s0+" "+s1+"\n";
        }
    return list;
    }
//---------------------------------------------------------------------------
void VideoCaptureVFW::set_video_driver(int ix)
    {
    if (hcap==NULL) return;
    capDriverConnect(hcap,ix);
    capDriverGetCaps(hcap,&driver_cp,sizeof(CAPDRIVERCAPS));
    capCaptureGetSetup(hcap,&capture,sizeof(CAPTUREPARMS));
//  capture.dwRequestMicroSecPerFrame=10;   // 1/fps [us]
    capCaptureSetSetup(hcap,&capture,sizeof(CAPTUREPARMS));
    capPreviewRate(hcap,1);         // set preview [ms]
    capPreviewScale(hcap,FALSE);    // stretching off
    //preview_start();
    }
//---------------------------------------------------------------------------
void VideoCaptureVFW::set_callback_on_frame()
    {
    if (ins_ix<0) ins_ix=VideoCaptureVFW_ins_get();
    if (ins_ix<0) return;
    VideoCaptureVFW_ins[ins_ix]=this;
    ins_use|=_vfw_callbach_onframe;
    capSetCallbackOnFrame(hcap,(void*)(VideoCaptureVFW_on_frame[ins_ix]));
    }
//---------------------------------------------------------------------------
void VideoCaptureVFW::res_callback_on_frame()
    {
    if (ins_ix<0) return;
    if (int(ins_use&_vfw_callbach_onframe))
        {
        ins_use^=_vfw_callbach_onframe;
        capSetCallbackOnFrame(hcap,NULL);
        }
    if (ins_use) return;
    VideoCaptureVFW_ins[ins_ix]=NULL;
    ins_ix=-1;
    }
//---------------------------------------------------------------------------
LRESULT VideoCaptureVFW::_on_frame(HWND hwnd,LPVIDEOHDR hdr)
    {
    fps.update();
    int e=0;
    if (hdr->dwBytesUsed<16) return 0;  // ignore too small images
    ((TDirectMemoryStream*)(mem))->SetMemory(hdr->lpData,hdr->dwBytesUsed);
    if ((hdr->lpData[6]=='J')   // JPEG signature
      &&(hdr->lpData[7]=='F')
      &&(hdr->lpData[8]=='I')
      &&(hdr->lpData[9]=='F'))
        {
        e=1;
        TJPEGImage *jpg=new TJPEGImage;
        jpg->LoadFromStream(mem);
        bmp->Assign(jpg);
        delete jpg;
        } else
    if ((hdr->lpData[0]=='B')   // BMP signature
      &&(hdr->lpData[1]=='M'))
        {
        e=1;
        bmp->LoadFromStream(mem);
        }
    else{                       // others
        e=1;
        copy_to_clipboard();
        try {
            bmp->LoadFromClipboardFormat(CF_BITMAP,Clipboard()->GetAsHandle(CF_BITMAP),NULL);
            }
        catch(char *str)
            {
            e=0;
            int hnd=FileCreate("unsuproted_format.dat");
            FileWrite(hnd,hdr->lpData,hdr->dwBytesUsed);
            FileClose(hnd);
            }
        }
    if (e)
        {
        if (on_frame) on_frame(*this);
        }
    return 0;
    }
//---------------------------------------------------------------------------
#endif
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------

I have a small test app with this source code:

//$$---- Form CPP ----
//---------------------------------------------------------------------------
#include <vcl.h>
#pragma hdrstop

#include "Unit1.h"
//---------------------------------------------------------------------------
// select API:
    #define _capture_VFW
//  #define _capture_DirectShow
//---------------------------------------------------------------------------
#ifdef _capture_VFW
#include "VideoCaptureVFW.h"
#endif
#ifdef _capture_DirectShow
#include "DirectX92\\VideoCaptureDirectShow.cpp"
#endif
//---------------------------------------------------------------------------
#pragma package(smart_init)
#pragma resource "*.dfm"
TForm1 *Form1;
Graphics::TBitmap *bmp=new Graphics::TBitmap;
int _callback=0;
int _frame=0;
int _update=false;
//---------------------------------------------------------------------------
#ifdef _capture_VFW
VideoCaptureVFW vfw;
void on_frame_VFW(VideoCaptureVFW &cap)
    {
    if (_callback) Form1->Canvas->Draw(0,26,cap.bmp);
    else if (!_frame) { bmp->Assign(cap.bmp); _frame=1; }
    }
#endif
//---------------------------------------------------------------------------
#ifdef _capture_DirectShow
VideoCaptureDirectShow dsh;
void on_frame_DirectShow(VideoCaptureDirectShow &cap)
    {
    if (_callback) Form1->Canvas->Draw(0,26,cap.bmp);
    else if (!_frame) { bmp->Assign(cap.bmp); _frame=1; }
    }
#endif
//---------------------------------------------------------------------------
__fastcall TForm1::TForm1(TComponent* Owner):TForm(Owner)
    {
    #ifdef _capture_VFW
    pan_VFW->Visible=true;
    vfw.set_owner(this->Handle);
    cb_driver->Items->Clear();
    cb_driver->Items->Text=vfw.get_video_drivers();
    cb_driver->ItemIndex=0;
    vfw.on_frame=on_frame_VFW;
    vfw.set_video_driver(cb_driver->ItemIndex);
    vfw.grab_start();
    #endif

    #ifdef _capture_DirectShow
    pan_DirectShow->Visible=true;
    cb_device->Items->Clear();
    cb_device->Items->Text=dsh.get_devices();

    dsh.on_frame=on_frame_DirectShow;
    _update=1;
    cb_device->ItemIndex=0;
    _update=0;
    cb_device->OnChange(this);
/*
    dsh.Select(0);
    dsh.Start();
    dsh.Stop();
*/
    #endif
    }
//---------------------------------------------------------------------------
void __fastcall TForm1::FormDestroy(TObject *Sender)
    {
    #ifdef _capture_VFW
    vfw.grab_stop();
    #endif

    #ifdef _capture_DirectShow
    dsh.Stop();
    #endif
    delete bmp;
    }
//---------------------------------------------------------------------------
void __fastcall TForm1::Timer1Timer(TObject *Sender)
    {
    if ((!_callback)&&(_frame)) { Canvas->Draw(0,26,bmp); _frame=0; }

    #ifdef _capture_VFW
    Caption=AnsiString().sprintf("frame: %2i fps: %2.1lf",vfw.fps.frame,vfw.fps.fps);
    #endif

    #ifdef _capture_DirectShow
    Caption=AnsiString().sprintf("frame: %2i fps: %2.1lf",dsh.fps.frame,dsh.fps.fps);
    #endif
    }
//---------------------------------------------------------------------------
//--- VFW -------------------------------------------------------------------
//---------------------------------------------------------------------------
void __fastcall TForm1::bt_dialog_sourceClick(TObject *Sender)
    {
    #ifdef _capture_VFW
    vfw.dlg_source();
    #endif
    }
void __fastcall TForm1::bt_dialog_formatClick(TObject *Sender)
    {
    #ifdef _capture_VFW
    vfw.dlg_format();
    #endif
    }
void __fastcall TForm1::bt_dialog_displayClick(TObject *Sender)
    {
    #ifdef _capture_VFW
    vfw.dlg_display();
    #endif
    }
void __fastcall TForm1::cb_driverChange(TObject *Sender)
    {
    #ifdef _capture_VFW
    vfw.set_video_driver(cb_driver->ItemIndex);
    vfw.grab_start();
    #endif
    }
//---------------------------------------------------------------------------
//--- DirectShow ------------------------------------------------------------
//---------------------------------------------------------------------------
void __fastcall TForm1::cb_deviceChange(TObject *Sender)
    {
    #ifdef _capture_DirectShow
    if (_update) return;
    _update=1;
    dsh.Select(cb_device->ItemIndex);
    cb_format->Items->Clear();
    cb_format->Items->Text=dsh.get_formats();
    if (cb_format->Items->Count)
     cb_format->ItemIndex=0;
    _update=0;
    cb_format->OnChange(this);
    #endif
    }
//---------------------------------------------------------------------------
void __fastcall TForm1::cb_formatChange(TObject *Sender)
    {
    #ifdef _capture_DirectShow
    if (_update) return;
    _update=1;
    dsh.set_format(cb_format->Text);
    _update=0;
    #endif
    }
//---------------------------------------------------------------------------

Its single form app with few buttons and combo box lists for dialog boxes and configuration (you can mimic them or ignore them). Did not share the DirectShow as its too big for 30K limit and slower anyway (but ist just a header + lib file no 3th party components). I have these VCL components on the form:

TTimer *Timer1; // 10ms info text update
TPanel *pan_VFW; // just to hold the components for VFW
TSpeedButton *bt_dialog_source; // these 3 buttons configure VFW ...
TSpeedButton *bt_dialog_format; 
TSpeedButton *bt_dialog_display;
TComboBox *cb_driver; // this selects VFW device
TPanel *pan_DirectShow; // just to hold DirectShow components
TComboBox *cb_device; // this selects DirectShow device
TComboBox *cb_format; // this selects DirectShow format

I encapsulated the VFW and DirectShow stuff into configuration #define so you can ignore the DirectShow stuff completely.

Now when you use this you can play with resolution and formats to compare the fps to your DirectShow component grabber.

As you can see I do not use any 3th party components to grab image data from camera the VideoCaptureVFW.h is the only stuff you need.

[Edit1]

Here is the link to the Demo containing both VFW and DirectShow examples (source and win32 binaries) in Embarcadero BDS2006 C++.

[Edit2] your unsupported format

The file size is exactly 1920*1080*3 Bytes which hints raw 24bpp RGB feed. When I tried to visualize it it works (and yes Y is flipped) see tyhe code (no components on form this time):

//$$---- Form CPP ----
//---------------------------------------------------------------------------
#include <vcl.h>
#pragma hdrstop

#include "Unit1.h"
//---------------------------------------------------------------------------
#pragma package(smart_init)
#pragma resource "*.dfm"
TForm1 *Form1;
Graphics::TBitmap *bmp=NULL;
//---------------------------------------------------------------------------
__fastcall TForm1::TForm1(TComponent* Owner):TForm(Owner)
    {
    int xs=1920,ys=1080,x,y,a,hnd,siz;
    BYTE *p,*dat=NULL;
    // load frame
    hnd=FileOpen("maybe_RGB24.dat",fmOpenRead);
    siz=FileSeek(hnd,0,2); dat=new BYTE[siz];
        FileSeek(hnd,0,0);
        FileRead(hnd,dat,siz);
        FileClose(hnd);
    // convert RGB24 to 32bpp bitmap
    bmp=new Graphics::TBitmap;
    bmp->HandleType=bmDIB;
    bmp->PixelFormat=pf32bit;
    bmp->SetSize(xs,ys);
    for (a=0,y=ys-1;y>=0;y--)       // flip y
        {
        p=(BYTE*)bmp->ScanLine[y];
        for (x=0;x<xs;x++)
            {
            p[0]=dat[a]; a++;
            p[1]=dat[a]; a++;
            p[2]=dat[a]; a++;
            p[3]=0;
            p+=4;
            }
        }
    delete dat;
    // resize form
    ClientWidth=xs;
    ClientHeight=ys;
    }
//---------------------------------------------------------------------------
void __fastcall TForm1::FormDestroy(TObject *Sender)
    {
    if (bmp) delete bmp; bmp=NULL;
    }
//---------------------------------------------------------------------------
void __fastcall TForm1::FormPaint(TObject *Sender)
    {
    Canvas->Draw(0,0,bmp);
    }
//---------------------------------------------------------------------------

And resulting screen:

beware I renamed the filename to maybe_RGB24.dat. You could decode this file-format by the frame size alone but its info should be somewhere like in the

AM_MEDIA_TYPE* mt;

structure but do not ask me where exactly I have no clue as I did code this a long time ago and do not use DirectX since (as all my attempts in the past reveals its inferiority to other apis does not matter if its sound or gfx or grabbing...)

Thanks for the reply and the code, I will download tomorrow and give it a try. One frustrating bit about the directshow stuff is it is hard to find anywhere that defines what performance should be. — Barry Andrews, Apr 26 '19 at 12:26
@BarryAndrews I think that is device driver dependent but its almost the same as VFW but slower due to bigger overhead. In VFW you can limit the fps by setting interval of grabbing in `[ms]` the fastest setting is `1 ms` meaning 1000 Hz but that does not mean you will be grabbing at that speed as device itself is controlling that so playing with resolution, pixel format and exposure time can adjust the speed ... — Spektre, Apr 26 '19 at 14:23
Hello, I got the VFW working but it only detects the following driver "Microsoft WDM Image Capture (Win32) Version: 10.0.17763.1" my capture device isn't available. The Image capture doesn't have any source, format or display dialogs available, so the application doesn't work for me. So looks like VFW is not an option. — Barry Andrews, Apr 27 '19 at 05:58
@BarryAndrews Not all cameras provide VFW drivers anymore (in my experience only with better vendors like Logitech you can expect them). See **[edit1]** I added demo (complete project with source and win32 binaries for both VFW and DirectShow) so you can compare it to your grabber. If they are the same speed its because of the API itself if yours is slower then the grabber of yours is at fault or you are doing something wrong. If yours is faster then its the same on my side. Let me know how it went. fps is shown on the Caption of the window just be sure to compare with the same conditions — Spektre, Apr 27 '19 at 07:37
@BarryAndrews IIRC the driver you detected in VFW is image acquisition API from Windows ... I never used it but IIRC it should be capable of loading/converting images in all the formats registered in Windows... — Spektre, Apr 27 '19 at 07:43
Thanks heaps for this. DS works and gives me around 29.0 to 29.6 fps but an inverted bitmap image. The VFW gives me a clipboard format not supported error. This is the case with your exe or if I change your project to work with C++ Builder 10.3 and build it myself. The exception occurs in VFW.h bmp->LoadFromClipboardFormat(CF_BITMAP, Clipboard()->GetAsHandle(CF_BITMAP), NULL); What I would like to be able to do is capture the video to disk using this as well, hence my original FFMPEG question. — Barry Andrews, Apr 30 '19 at 06:25
@BarryAndrews The error means your camera is feeding image in format I do not support in my code so not BMP nor JPG but something else and as the compiler I done this in is pretty old it does not support newer formats unless encoded (I had to add even JPG support for the capturing). Glad to see you have ~30fps so it must be at least some form of JPG or wavelet or god knows what anyway looks like the 30fps is your device or API limit and I am afraid you will not be able to go pass that. — Spektre, Apr 30 '19 at 06:53
@BarryAndrews I would like to see the unsupported frame can you save it into binary file? you know save `dat[siz]` as binary file inside the `HRESULT __stdcall VideoCaptureDirectShow::CallbackHandler::SampleCB(double time, IMediaSample *sample)` function before the line `cap->bmp->HandleType=bmDIB;`. If you do not know how take a look into my VFW source and look for `"unsuproted_format.dat"` and post it for me (I like ulozto.net as it does not need registration) along with description of the frame (chosen format, resolution) — Spektre, Apr 30 '19 at 07:16
I will try and get this tomorrow, don't have my camera with me today. It is actually LRESULT VideoCaptureVFW::_on_frame(HWND hwnd,LPVIDEOHDR hdr) which is giving the exception when copying to the clipboard, but the catch(char * str) doesn't seem to be entered after the exception and hence the .dat file is not created, I will try changing the catch to catch(Exception& e) and see if that works, I did break point and look at the hdr->lpData but it didn't look like a sensible string. — Barry Andrews, May 01 '19 at 07:33
Directshow capture is working, just inverted image. 30fps is all my current camera outputs so it's good to see the directshow is handling it. Makes me wonder why MITOV TVDSCapture component cannot handle preview on it though as this should also be DirectShow. — Barry Andrews, May 01 '19 at 07:34
I went to add the code where you said, however, there is no SampleCB in the code you sent me? Nor is there any cap->bmp->HandleType=bmDIB; did you mean for me to output the frame in on_frame_DirectShow where it does bmp->Assign(cap.bmp)? — Barry Andrews, May 01 '19 at 07:47
@BarryAndrews The code is inside file `DirectX92\\VideoCaptureDirectShow.cpp` I posted in the demo btw. I am rendering the feed by VCL/GDI canvas and bitmap so no DirectX fancy overlays ... meaning that is not the bottleneck. My bet is that the component of yours has some dellays/sleeps for sync/safety or is hitting some vertical sync issue ... Also the unsupported frame might be direct feed without any header, I saw some like that for YUV pixelformats. Do not forget to chose correct define to select VFW/DirectShow if you using my project — Spektre, May 01 '19 at 13:34
I have captured the frame as you suggested, and saved at https://ulozto.net/!6ZHrs8S433Wb/unsupported-format-zip for you to download. — Barry Andrews, May 02 '19 at 00:41
@BarryAndrews Thanks for the frame looks I got it see [edit2] but that should work (but flipped) and do not throw unsupported format anywhere unless the `cap->bpp` is not matching ... or you did not overwrite the `bmp` memory somewhere. If the assign is troubling you can use `bmp->Canvas->Draw(0,0,cap->bmp);` instead but prior to it set the `bmp` resolution and `pixelformat`... Nice to see I am not the only one left on Win7 :) I hate the Win8/10 no decent IDE works there. Btw the RAW image is distorted due to lossy compression so camera most likely use JPG — Spektre, May 02 '19 at 08:20
@BarryAndrews just a taught maybe the grabbing is OK and the problem is on `void on_frame_DirectShow(VideoCaptureDirectShow &cap)` in the main code check if both bmps are valid ones (you know breakpoint and watch both of them if they have valid resolution, pixel format and are not NULL pointers or gibberish) ... — Spektre, May 02 '19 at 08:25
@BarryAndrews heh :) I just decoded `YUY2` frame (that is why I had the unsupported format dat in the VFW for one of mine cameras) based on this https://stackoverflow.com/a/4494004/2521214 ... didnt get to it for 3 years until now :) — Spektre, May 02 '19 at 09:44
did you have code for the YUY2 or do I look into the examples you referenced? — Barry Andrews, May 03 '19 at 02:43
in order to capture to AVI file, do I just IBaseFilter *pMux; hr = capture->SetOutputFileName(&MEDIASUBTYPE_Avi, "Example.avi", &pMux, NULL); hr = capture->RenderStream(&PIN_CATEGORY_CAPTURE, &MEDIATYPE_Video, samplegrabberfilter, NULL, pMux); ? — Barry Andrews, May 03 '19 at 04:43
@BarryAndrews 1. the linked code is enough for YUY2, I do not have it implemented in the grabber yet (as I do not have the YUY2 camera anymore) I have it just in separate app to decode/verify unsupported frames. 2. As I mentioned I do not use `DirectX` and coded the grabber years ago (so I do not remeber) so I have no clue and would need to study the api again to verify that which I am not willing to do as my use of Direct is appart this grabbing stuff none. You can still save the frames to file on your own, but for that you need encoding engine to some video format like your FFMPEG I assume — Spektre, May 03 '19 at 06:05
@BarryAndrews I see you got another down vote maybe you should edit your question a bit and include your current code and progress... so the question does not look like a request for code or debugging question. The causal voters usually do not read comments ... — Spektre, May 03 '19 at 06:08
@spectre it has drifted a bit from my original question where I wanted to add ffmpeg to existing directshow components to improve ability to write to disc and preview. I will look at changing the question, just need to work out the wording. I got the writing to avi file working on one camera but get exception on another so need to research more. — Barry Andrews, May 03 '19 at 13:13

Direct Show Video capture performance

1 Answers1