How use hardware(nvenc) encoding in Media Foundation (Windows 10)

Question

Sorry for my English. I have question about hardware encoding with MF in Windows 10. I have Nvidia gtx 650 (with nvenc). I develop the application for encoding live frame stream to h264 file in c#.

I used this code ( https://codereview.stackexchange.com/questions/136144/h-264-image-encoding-using-media-foundation-net ) as example. I created IMFSinkWriter object(sinkWriter), as in example, with MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS=true

private int InitializeSinkWriter(String outputFile, int videoWidth, int videoHeight)
{            
    IMFMediaType mediaTypeIn = null;
    IMFMediaType mediaTypeOut = null;
    IMFAttributes attributes = null;

    int hr = 0;

    if (Succeeded(hr)) hr = MFExtern.MFCreateAttributes(out attributes, 1);
    if (Succeeded(hr)) hr = attributes.SetUINT32(MFAttributesClsid.MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, 1);
    if (Succeeded(hr)) hr = attributes.SetUINT32(MFAttributesClsid.MF_LOW_LATENCY, 1);

    // Create the sink writer 
    if (Succeeded(hr)) hr = MFExtern.MFCreateSinkWriterFromURL(outputFile, null, attributes, out sinkWriter);

    // Create the output type
    if (Succeeded(hr)) hr = MFExtern.MFCreateMediaType(out mediaTypeOut);
    if (Succeeded(hr)) hr = mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
    if (Succeeded(hr)) hr = mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.H264);
    if (Succeeded(hr)) hr = mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_AVG_BITRATE, videoBitRate);
    if (Succeeded(hr)) hr = mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int) MFVideoInterlaceMode.Progressive);
    if (Succeeded(hr)) hr = MFExtern.MFSetAttributeSize(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
    if (Succeeded(hr)) hr = MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
    if (Succeeded(hr)) hr = MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
    if (Succeeded(hr)) hr = sinkWriter.AddStream(mediaTypeOut, out streamIndex);

    // Create the input type 
    if (Succeeded(hr))  hr = MFExtern.MFCreateMediaType(out mediaTypeIn);
    if (Succeeded(hr)) hr = mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
    if (Succeeded(hr)) hr = mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.RGB24);
    if (Succeeded(hr)) hr = mediaTypeIn.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int)MFVideoInterlaceMode.Progressive);
    if (Succeeded(hr)) hr = MFExtern.MFSetAttributeSize(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
    if (Succeeded(hr)) hr = MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
    if (Succeeded(hr)) hr = MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
    if (Succeeded(hr)) hr = sinkWriter.SetInputMediaType(streamIndex, mediaTypeIn, null);

    // Start accepting data
    if (Succeeded(hr))  hr = sinkWriter.BeginWriting();

    COMBase.SafeRelease(mediaTypeIn);
    COMBase.SafeRelease(mediaTypeOut);

    return hr;            
}

I did, as in example, the following steps for each frame encoding: 1. Created IMFMediaBuffer object(buffer) and copied there the frame 2. Created IMFSample object(sample) and sample.AddBuffer(buffer) 3. Wrote sample to IMFSinkWriter object(sinkWriter)

As result have I 100% load on CPU (this is not hardware encoding!!!). How may I use the same code to do the hardware encoding?

How do you know that hardware encoder was not used apart from the CPU usage (which might be cause by other parts of your code)? You need to run your program with mftrace to see which encoder has been initialized. — VuVirt, Nov 29 '16 at 07:41

score 3 · Answer 1 · edited May 23 '17 at 11:54

Sink Writer API offers you automatic conversion (esp. encoding) of data, which includes in turn H.264 encoding when you request it. You do request it by doing AddStream with MFMediaType.H264 followed by SetInputMediaType with MFMediaType.RGB24.

You, however, don't have fine grained flexibility of encoder choice. It is behavior by design that API picks "best" encoder for you. Your applying MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS means that you are okay to use hardware MFT but it does not mean you force the option. You don't even know if hardware MFT is available option in first place, do you? Again, it's behavior by design that API avoids ambiguity known as "codec hell" by reserving the right to select best applicable encoder itself. Documentation explains this this way: If a certified hardware encoder is present, it will generally be used instead of the inbox system encoder for Media Foundation related scenarios.

Maxed out CPU load is a hint that compression might be software, yet it could have been caused by other reasons too. In your situation you need to check out what H.264 encoders are available, debug and trace your application. To force hardware H.264 encoding with Media Foundation when the API does not pick the encoder up automatically, you have no other choices but to compress video to H.264 yourself (using MFT which corresponds to unlikely scenario that encoder exists as MFT but API does not pick it up, or encode otherwise esp. using vendor specific SDK), then feed the compressed data to Sink Writer API to produce a well-formatted MP4 file, with possibly audio mixed in.

How use hardware(nvenc) encoding in Media Foundation (Windows 10)

1 Answers1

Linked