Media Foundation set video interlacing and decode

Question

I have an MOV file and I want to decode it and have all frames as separate images.

So I try to configure an uncompressed media type in the following way:

// configure the source reader
IMFSourceReader* m_pReader;
MFCreateSourceReaderFromURL(filePath, NULL, &m_pReader);

// get the compressed media type
IMFMediaType* pFileVideoMediaType;
m_pReader->GetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, &pFileVideoMediaType);

// create new media type for uncompressed type
IMFMediaType* pTypeUncomp;
MFCreateMediaType(&pTypeUncomp);

// copy all settings from compressed to uncompressed type
pFileVideoMediaType->CopyAllItems(pTypeUncomp);

// set the uncompressed video attributes
pTypeUncomp->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB8);
pTypeUncomp->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, TRUE);
pTypeUncomp->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);

// set the new uncompressed type to source reader
m_pReader->SetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, pTypeUncomp);

// get the full uncompressed media type
m_pReader->GetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, &pTypeUncomp);

I noticed that even I explicitly set the MF_MT_INTERLACE_MODE to MFVideoInterlace_Progressive the final configuration is still configured with the old mode MFVideoInterlace_MixedInterlaceOrProgressive.

Afterwards, I loop through all samples and look at their size:

IMFSample* videoSample = nullptr;
IMFMediaBuffer* mbuffer = nullptr;
LONGLONG llTimeStamp;
DWORD streamIndex, flags;

m_pReader->ReadSample(
            MF_SOURCE_READER_FIRST_VIDEO_STREAM,
            0,                               // Flags.
            &streamIndex,                    // Receives the actual stream index. 
            &flags,                          // Receives status flags.
            &llTimeStamp,                    // Receives the time stamp.
            &videoSample)                    // Receives the sample or NULL.

videoSample->ConvertToContiguousBuffer(&mbuffer);

BYTE* videoData = nullptr;
DWORD sampleBufferLength = 0;

mbuffer->Lock(&videoData, nullptr, &sampleBufferLength);
cout << sampleBufferLength << endl;

And I get quite different sizes for the samples: from 31bytes to 18000bytes. Even changing the format to MFVideoFormat_RGB32 does not change affect the sample sizes.

This question seems to have the same issue but the solution is not fixing it.

Any help on why I can't change the interlacing and how to properly decode video frames and get image data out of samples?

Many thanks in advance.

Maybe you should try to call GetNativeMediaType first instead of m_pReader->GetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, &pFileVideoMediaType);. Did you also check if each sample is interlaced: pSample->GetUINT32(MFSampleExtension_Interlaced, &isSampleInterlaced); — VuVirt, Apr 20 '17 at 09:04
You can check the ConfigureDecoder function here: https://msdn.microsoft.com/en-us/library/windows/desktop/dd389281(v=vs.85).aspx#setting_output_formats. You can also check for MF_SOURCE_READERF_CURRENTMEDIATYPECHANGED in the receives status flags when calling ReadSample, to see if the type has been changed. — VuVirt, Apr 20 '17 at 09:12
@VuVirt, thanks for your comments. Getting native media type did not change anything. Checking a sample for interlacing ended up with an error. And there was no media type change. Is there something else in your mind? In case you can find couple of minutes to look at my code : https://github.com/mbaros100/Media-Foundation-video-decoder. Thank you very much for your help. — mbaros, Apr 20 '17 at 13:01
try without pTypeUncomp->CopyAllItems. Just build the uncompressed video media type manually by using all the neccessary attributes, as shown in this link: https://msdn.microsoft.com/en-us/library/windows/desktop/ff485865(v=vs.85).aspx — VuVirt, Apr 20 '17 at 13:14
@VuVirt, I have followed the link you sent me and did as it says. No any positive change. I have updated the code in the repo. — mbaros, Apr 20 '17 at 14:21
You shouldn't set MFVideoFormat_Base for the subtype. Try with MFVideoFormat_RGB32. — VuVirt, Apr 20 '17 at 15:10

score 3 · Accepted Answer · edited Aug 31 '20 at 15:39

3

In order to make SourceReader convert the samples to RGB you need to create it like this:

IMFAttributes* pAttr = NULL;
MFCreateAttributes(&pAttr, 1);
pAttr->SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, TRUE);
pAttr->SetUINT32(MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING, TRUE);
    
IMFSourceReader* m_pReader;
throwIfFailed(MFCreateSourceReaderFromURL(filePath, pAttr, &m_pReader), Can't create source reader from url");
pAttr->Release();

Later, you shouldn't break from the cycle when MF_SOURCE_READERF_CURRENTMEDIATYPECHANGED occurs. Now you'll have all samples with the same size. Otherwise you can use MFVideoFormat_NV12 subtype and then you won't need to specify MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING attribute when creating the reader.

edited Aug 31 '20 at 15:39

Andreas

9,245
9
49
97

answered Apr 20 '17 at 16:13

VuVirt

1,887
11
13

1

Worked for me. Thank you very much. – mbaros Apr 20 '17 at 22:29
let me ask you one more question. For some files I have got that buffersize exactly matching image height*width*4 and the data was correct. For some other files I got the buffer size bigger than the height*width*4. Do you know what is this additional data standing for ? – mbaros Apr 21 '17 at 21:42
VuVirt, I saw your answer to http://stackoverflow.com/questions/43014780/extarct-rgb32-byte-data-from-imfsample/43027190#43027190, which is very nice. I tried to lock my IMFSample to 2D data as you said, but I ended up with an error. Can you also please share a piece of code that show how to correctly lock and access sample data with 2D? Thanks in advance – mbaros Apr 21 '17 at 23:04
1

@mbaros What error? Some samples do not expose IMF2DBuffer. You might need to lock IMFMediaBuffer directly in this case. Or query for D3D texture. Follow the links in the other answer, there is enough source code and explanations to start with. As for the bigger width, it is called stride: https://msdn.microsoft.com/en-us/library/windows/desktop/aa473780(v=vs.85).aspx. Thanks! – VuVirt Apr 22 '17 at 08:00
VuVirt look. Most probably it's the case that samples do not expose IMF2DBuffer. I do use IMFMediaBuffer. I have tested 3 media files. In all cases stride = width*4. The issue is that for some cases sample buffer length is more that h*w*4. And taking first h*w*4 bytes of data gives wrong output image. Whenever my buffer length is h*w*4 the data is correct. – mbaros Apr 23 '17 at 20:49
You should use the stride obtained by the Lock call and iterate the image line by line by incrementing stride bytes on each iteration. – VuVirt Apr 24 '17 at 07:13
If you're using `IMFMediaSession` for playback in **Direct3D11/DXGI** mode, then by default your `IMFMediaSink` won't see the full range of deinterlacing and color formats. Fortunately, it's trivial to remedy this by simply setting `MF_TOPOLOGY_ENABLE_XVP_FOR_PLAYBACK` to `UINT32/TRUE` on your `IMFTopology`. Presto, the pipeline will include the `XVP` processor MFT to do color-conversion, deinterlacing, etc., and your `IMFMediaSink` will be presented with many more options during format negotiation. – Glenn Slayden Aug 04 '17 at 22:59
Oh, and by the way, `MF_TOPOLOGY_ENABLE_XVP_FOR_PLAYBACK` doesn't seem to be documented anywhere. I'm posting this note here in case it helps anyone, since it took me forever and a hideous struggle before stumbling upon such an easy fix. For example, the Microsoft D3D11 `IMFMediaSink` sample--by demonstrating the explicit manual hook-up and calling of external XVP--would lead you to believe that when the D3D9 `EVR` model was abandoned in D3D11, the `IMFMediaSink` had to take on the burden of those chores.Not entirely so, apparently. – Glenn Slayden Aug 05 '17 at 00:11

Media Foundation set video interlacing and decode

1 Answers1