5

I'm trying to use the MediaFoundation API to encode a video but I'm having problems pushing the samples to the SinkWriter.

I'm getting the frames to encode through the Desktop Duplication API. What I end up with is an ID3D11Texture2D with the desktop image in it.

I'm trying to create an IMFVideoSample containing this surface and then push that video sample to a SinkWriter.

I've tried going about this in different ways:

  • I called MFCreateVideoSampleFromSurface(texture, &pSample) where texture is the ID3D11Texture2D, filled in the SampleTime and SampleDuration and then passed the created sample to the SinkWriter.
    SinkWriter returned E_INVALIDARG.

  • I tried creating the sample by passing nullptr as the first argument and creating the buffer myself using MFCreateDXGISurfaceBuffer, and then passing the resulting buffer into the Sample.
    That didn't work either.

  • I read through the MediaFoundation documentation and couldn't find detailed information on how to create the sample out of a DirectX texture.

I ran out of things to try.
Has anyone out there used this API before and can think of things I should check, or of any way on how I can go about debugging this?

ANeves
  • 6,219
  • 3
  • 39
  • 63
Tiago Magalhães
  • 51
  • 5
  • I think that you have made a mistake with using of `MFCreateVideoSampleFromSurface` and `MFCreateDXGISurfaceBuffer`. These function create buffers for writing into the surface of DirectX and visualization, and they create objects with interface `IMF2DBuffer`, but encoders and writers need objects with `IMFMediaBuffer`. They cannot understand `IMF2DBuffer`. Try use function 'MFCreateSampleCopierMFT' - it created MFT object for copying from `IMFMediaBuffer` to `IMF2DBuffer` on [MSDN](https://msdn.microsoft.com/en-us/library/windows/desktop/dd388101(v=vs.85).aspx). May be it can do revers work!?! – Evgeny Pereguda Jul 25 '16 at 22:32
  • Are you sure on that? Looking at the documentation for `MFCreateDXGISurfaceBuffer`, it states that it returns a pointer to an IMFMediaBuffer. – Tiago Magalhães Jul 26 '16 at 11:06
  • You can try get data via `IMFMediaBuffer` of buffer from `MFCreateDXGISurfaceBuffer`. – Evgeny Pereguda Jul 26 '16 at 11:39
  • 1
    The best way to get data out of a Sample/Buffer backed by a DXGI surface seems to be MFGetService. Another thread on this indicates that these two APIs should be able to handle this task: http://stackoverflow.com/questions/14527718/whole-screen-capture-and-render-in-directx-performance. I am unable to find any decent example anywhere on the best use of these two methods. We're going to try and go in another direction with this which is a shame since DesktopDuplication + MediaFoundation seemed to be the best solution for our problem. – Tiago Magalhães Jul 27 '16 at 14:24
  • 1
    Your idea of using of DesktopDuplication + MediaFoundation is interesting. In my project CaptureManager SDK on 'CaptureProject' site I used DirectX9 + MediaFoundation for capture live screen video, but it copy image from Desktop Manager (In task Manager program and Desktop Manager take 15 percents each other). I think that using of DesktopDuplication can be more efficient. – Evgeny Pereguda Jul 27 '16 at 23:20
  • 1
    @Tiago Magalhães did you find any solution for the problem? I too got stuck with it as there are no resources I could find that is relevant. – iamrameshkumar Jan 27 '18 at 13:21
  • Sorry Ram, got no solution for this and we ended up going with a very different solution to the overall problem. Like you said, very little resources and we ran out of time. But if you do come up with something, I would love to hear from you, it's a problem I still intend to go back to one of these days. – Tiago Magalhães Jan 28 '18 at 14:58

1 Answers1

2

First of all you should learn to use mftrace tool. Very likely, it will tell you the problem right away.

But my guess is, following problems are likely.

  1. Probably, some other attributes are required besides SampleTime / SampleDuration.

  2. Probably, SinkWriter needs a texture it can read on CPU. To fix that, when a frame is available, create a staging texture of the same format + size, call CopyResource to copy desktop to staging texture, then pass that staging texture to MF.

  3. Even if you use a hardware encoder so the CPU never tries to read the texture data, I don’t think it’s a good idea to directly pass your desktop texture to MF.

When you set a D3D texture for sample, no data is copied anywhere, the sample merely retains the texture.

MF works asynchronously, it may buffer several samples in its topology nodes if they want to.

DD gives you data synchronously, you may only access the texture between AcquireNextFrame and ReleaseFrame calls.

Soonts
  • 20,079
  • 9
  • 57
  • 130
  • I tried not to bog the question down with too many details but I am copying the desktop texture before passing it to MF by getting the desktop texture descriptor and creating a new texture using that before using CopyResource. But thanks a lor for the mftrace pointer, will take a look at that. – Tiago Magalhães Jul 26 '16 at 10:50
  • I used mfctrace and figured out that the samples being sent to the sink are being reported as having 0B size so something is going wrong with their creation. – Tiago Magalhães Jul 27 '16 at 14:21
  • @Soonts, I have even tried creating a staging texture and did call CopyResource, have no impact in the output. Getting simply a green screen. https://stackoverflow.com/questions/58503662/getting-green-screen-in-ffplay-streaming-desktop-directx-surface-as-h264-vide – iamrameshkumar Nov 02 '19 at 06:28