Extending Media Foundation Encoder to support 10-bit video encoding

Question

The HEVC Media Foundation Encoder in Windows will only encode 8-bit video. My GFX NVidia card also supports 10-bit HDR and alpha-mode video encoding, so I decided to create my own IMFTransform to use the NVidia SDK.

I 've registered my DLL using MFTRegister:

    MFT_REGISTER_TYPE_INFO aMediaTypesIn[] =
    {
        {   MFMediaType_Video, MFVideoFormat_ARGB32 },
        {   MFMediaType_Video, MFVideoFormat_RGB32 },
        {   MFMediaType_Video, MFVideoFormat_RGB32 },
        {   MFMediaType_Video, MFVideoFormat_RGB10 },
        {   MFMediaType_Video, MyFakeFmt },
        
    };

    MFT_REGISTER_TYPE_INFO aMediaTypesOut[] =
    {
        {   MFMediaType_Video, MFVideoFormat_H264 },
        {   MFMediaType_Video, MFVideoFormat_H265 },
        {   MFMediaType_Video, MFVideoFormat_HEVC },
    };

    // Size of the array.
    const DWORD cNumMediaTypesI = ARRAY_SIZE(aMediaTypesIn);
    const DWORD cNumMediaTypesO = ARRAY_SIZE(aMediaTypesOut);

    hr = MFTRegister(
        GUID_NVidiaEncoder,     // CLSID.
        MFT_CATEGORY_VIDEO_ENCODER,  // Category.

The MyFakeFmt is a non existing input type to fool the Sink Writer to pick my encoder when calling SetInputMediaType instead of the predefined Microsoft's transform. This works OK.

int wi = 1920;
int he = 1080;
int fps = 30;
int br = 4000;
auto fmt = MFVideoFormat_H264;
bool Our = 1;
const wchar_t* fil = L"r:\\1.mp4";
std::vector<DWORD> frame;
frame.resize(wi * he);

// Test
CComPtr<IMFSinkWriter> wr;
DeleteFile(fil);

CComPtr<IMFAttributes> attrs;
MFCreateAttributes(&attrs, 0);
auto hr = MFCreateSinkWriterFromURL(fil, 0, attrs, &wr);
DWORD str = (DWORD)-1;
CComPtr<IMFMediaType> mt2;
MFCreateMediaType(&mt2);
mt2->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
mt2->SetGUID(MF_MT_SUBTYPE, fmt);
MFSetAttributeRatio(mt2, MF_MT_FRAME_RATE, fps, 1);
hr = MFSetAttributeSize(mt2, MF_MT_FRAME_SIZE,wi, he);
MFSetAttributeRatio(mt2, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
mt2->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);
mt2->SetUINT32(MF_MT_VIDEO_NOMINAL_RANGE, MFNominalRange_Normal);
mt2->SetUINT32(MF_MT_AVG_BITRATE, br*1000);
hr = wr->AddStream(mt2, &str);
CComPtr<IMFMediaType> mt1;
MFCreateMediaType(&mt1);
mt1->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
mt1->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_ARGB32);
hr = MFSetAttributeSize(mt1, MF_MT_FRAME_SIZE, wi, he);
// Force our selection
if (Our)
{
    mt1->SetGUID(MF_MT_SUBTYPE, MyFakeFmt);
    hr = wr->SetInputMediaType(str, mt1, 0);
}
mt1->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_ARGB32);
hr = wr->SetInputMediaType(str, mt1, 0);
hr = wr->BeginWriting();
for(int i = 0 ; i < 15 ; i++)
{
    auto i2 = i % 5;
    if (i2 == 0)            Frm(frame, wi, he, 0xFFFFFFFF);
    if (i2 == 1 || i2 == 4)         Frm(frame, wi, he, 0xFF0000FF); // some colors
    if (i2 == 2)            Frm(frame, wi, he, 0xFFFF00FF); // 
    if (i2 == 3)            Frm(frame, wi, he, 0xFF00FFFF); //

    CComPtr<IMFSample> s;
    MFCreateSample(&s);
    int secs = 1;

    hr = s->SetSampleDuration(10 * 1000 * 1000 * secs);
    hr = s->SetSampleTime(10 * 1000 * 1000 * i);

    CComPtr<IMFMediaBuffer> b;
    MFCreateMemoryBuffer((DWORD)(frame.size() * 4), &b);        
    b->SetCurrentLength((DWORD)(frame.size() * 4));
    BYTE* by = 0;
    DWORD ml = 0, cl = 0;
    b->Lock(&by, &ml, &cl);
    memcpy(by, frame.data(), frame.size() * 4);
    b->Unlock();
    hr = s->AddBuffer(b);
    b = 0;
    hr = wr->WriteSample(str, s);
}

hr = wr->Finalize();
wr = 0;

The problems start with the call to Finalize to end the writing. At that point, everything seems to work normally. Note that I have tested the NVidia IMFTransform I 've created with input frames and it encodes and outputs them correctly as raw data.

When I call Finalize and the type is MFVideoFormat_H264 , the call succeeds. However the generated mp4 plays weirdly:

For some reason also, MediaInfo shows 1 FPS. Why?

When the output is MFVideoFormat_HEVC, then Finalize fails with `0xc00d4a45 : Sink could not create valid output file because required headers were not provided to the sink.'.

I 've also tried to convert the raw .h264 file I 'm saving with ffmpeg to mp4, and this works. The mp4 generated plays correctly.

Adding a MF_MT_MPEG_SEQUENCE_HEADER didn't help (besides, I think this is only needed for H.264)

const char* bl4 = "\x00\x00\x00\x01\x67\x42\xC0\x28\x95\xB0\x1E\x00\x89\xF9\x70\x16\xC8\x00\x00\x03\x00\x08\x00\x00\x03\x01\xE0\x6D\x04\x42\x37\x00\x00\x00\x01\x68\xCA\x8F\x20";
    mt2->SetBlob(MF_MT_MPEG_SEQUENCE_HEADER, (UINT8*)bl4, 39);

What do you make of all that? Thanks

Semi-orthgonal question: Have you considered querying to see if the buffer supports the [`IMF2DBuffer`](https://learn.microsoft.com/en-us/windows/win32/api/mfobjects/nn-mfobjects-imf2dbuffer) interface? That's what you're supposed to use when filling in video frames according to the docs on `IMFBuffer`, specifically you're supposed to call `Lock2D` because it is more efficient and gives better info on padding you might be overwriting and breaking things. — Mgetz, May 05 '22 at 13:05
Whether you have manually provided the sample description box (stsd) description using the [MF_MT_MPEG4_SAMPLE_DESCRIPTION](https://learn.microsoft.com/en-gb/windows/win32/medfound/mf-mt-mpeg4-sample-description?redirectedfrom=MSDN) GUID when configuring the properties before creating the sink? I suggest you could refer to the thread: https://stackoverflow.com/questions/25129263/cant-add-audio-stream-to-mp4-file-using-sinkwriter-from-media-foundation — Jeaninez - MSFT, May 06 '22 at 06:33

Extending Media Foundation Encoder to support 10-bit video encoding

0 Answers0