1

I am looking for an effective way to grab image data off video files. I am currently testing FilgraphManagerClass.GetCurrentImage() from the Interop.QuartzTypeLib library. This does what I need but is painfully slow. I need to process all frames of each video. What better options do I have?

Requirements

  • Must be frame accurate. <-- Very important!
  • Gives me access to the decoded pixel buffer (array of int or byte[]), ideally RGB24 or RGB32.
  • The buffer can be grabbed in realtime or faster. I do not need to display the video, I only need to analyze the pixels.
  • Handle mp4 files (h264/aac). I can rewrap or frame serve via AviSynth if needed but no retranscoding can be involved.

Any suggestions would be welcome.

Some code as requested:

FilgraphManagerClass graphClass = new FilgraphManagerClass();
graphClass.RenderFile(@"C:\tmp\tmp.avs");

int sz = (graphClass.Width * graphClass.Height + 10) * 4;    
int[] buffer = new int[sz - 1];

I am then stepping through each frame. I have something like this in the loop:

graphClass.GetCurrentImage(ref sz, out buffer[0]);
//DoStuff(buffer);
graphClass.CurrentPosition += graphClass.AvgTimePerFrame;
BlueVoodoo
  • 3,626
  • 5
  • 29
  • 37
  • Show some code. `GetCurrentImage` is close but still not the right way to do it. In C# domain you typically use `Sample Grabber` with a callback to get your `SampleCB` method called providing video frames one by one. – Roman R. Jul 21 '12 at 17:23
  • @RomanR. I have included some code now. – BlueVoodoo Jul 21 '12 at 17:42

2 Answers2

5

IBasicVideo::GetCurrentImage method you are using is basically intended for snapshots, and works with legacy video rendering in legacy modes only. That is, (a) it is NOT time accurate, it can get you duplicate frames or, the opposite, lose frames; and (b) it assumes that you display video.

Instead you want to build a filter graph of the following kind: File Source -> ... -> Sample Grabber Filter -> Null Renderer. Sample Grabber, a standard component, can be provided with a callback so that it calls you with any frame data that comes through it.

Then you remove clock from the graph by calling SetReferenceClock(null) on the filter graph so that it run as fast as possible (as opposed to realtime). Then you Run the graph and all video frames are supplied to your callback.

To accomplish the task in C# you need to use DirectShow.NET library. It's Capture\DxSnap sample provides a brief example how to use Sample Grabber. They do it through BufferCB instead of SampleCB and it works well too. Other samples there are also using this approach.

You will find other code snippets very close to this task:

Regarding MP4 files you should take into consideration the following:

  1. Support for MPEG-4 is limited in Windows, and you might need third party components installed to make the files playable. If GraphEdit can read them, then you can too.
  2. Windows Media Player might be using, and is likely to, a newer API and you should rather look at GraphEdit
  3. Be sure to use Win32/x86 platform on your application to avoid running into scenario that your app is running in 64-bit domain, while support for MP4 only exists in 32-bit components/libraries installed
Community
  • 1
  • 1
Roman R.
  • 68,205
  • 6
  • 94
  • 158
  • Thanks for your reply. I was actually trying that library earlier but unfortunately ran into some issues right away. The sample I was testing was Editing\DxScan but it didn't want to accept my mp4 file. The error was "No combination of intermediate filters could be found to make the connection.". If I open up the file in GraphEdit or Windows Media Player it works however. If I try DxSnap instead, it returns a "Timeout waiting to get picture". I'll read up on the library. Do you know if there is a way to manually construct the filter chain that I see in GraphEdit? – BlueVoodoo Jul 21 '12 at 18:24
  • See my update above on MP4s. I suppose you had #3 issue. If GraphEdit can do MP4, you can too. – Roman R. Jul 21 '12 at 18:32
  • I did have an 64-bit related issue on a test AVI file which went away when I set the project to x86. But for my mp4 file, I am still out of luck. It works in GraphEdit but not when debugging DxScan. Will go for a break now but keep on trying when I get back. – BlueVoodoo Jul 21 '12 at 18:50
  • With `RenderFile` you offload all work to find matching components to the filter graph. Something perhaps does not work as expected there. In GraphEdit you see what you need and you can do some graph building manually (`AddFilter`, `Connect` etc). – Roman R. Jul 21 '12 at 18:58
  • This has something to do with the way the sample grabber is configured. If I comment out ConfigureSampleGrabber(sampGrabber); it gets further. Perhaps I can't set it to RGB24 or similar. I'll accept this as an answer for now. This is a separate issue. In terms of speed and what I need to do. This is what I have been looking for. Thanks for you help so far. – BlueVoodoo Jul 21 '12 at 20:48
  • Worth knowing is that in GraphEdit, I really can do the full connection. Output-->SampleGrabber-->Null Renderer. I just seem to be able to do the same thing in my code. No idea why at this point. – BlueVoodoo Jul 21 '12 at 21:45
  • ...and the problem is finally solved. Installed FFDShow which works better compared to whatever I had before. – BlueVoodoo Jul 21 '12 at 21:58
0

You could also look at creating an allocator-presenter using Windows Media Foundation. This will give you the decoded video frame as a GPU texture and you could also use CUDA or OpenCL to perform the processing required (if possible) which would help your processing speed immensely.

Ani
  • 10,826
  • 3
  • 27
  • 46