I suspect you are doing too much of the work (which hopefully is good news). First keep in mind that the example does not encode the output as RGB. RGB was simply the input type to the sink, and you can feel free to leave it that way.
The container is MFTranscodeContainerType_ASF (more on that in a sec),
and the subtypte is MFVideoFormat_WMV3,
and the result is a well compressed windows media video file.
When the sink writer is created, in your case the container is automatically created, since the attribute wasn't specified. In fact, the only attribute added is MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS on the following lines:
MFCreateAttributes(&spAttr, 10);
spAttr->SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, true);
hr = MFCreateSinkWriterFromURL(L".wmv",
spByteStream.Get(),
spAttr.Get(),
&sinkWriter);
Which begs the question - 'why are 10 attributes allocated?' ...
(shrug) oh well, doesn't matter.
The point being, you send RGB frames to the sink, and transforms are being done to complete the encoding to your specification. Containers may be found here:
http://msdn.microsoft.com/en-us/library/windows/desktop/dd388919(v=vs.85).aspx
and prior to creating the sink, you would specify a given container similar to the following:
spAttr->SetGUID(MF_TRANSCODE_CONTAINERTYPE, MFTranscodeContainerType_ASF);
// or MFTranscodeContainerType_MPEG4, MFTranscodeContainerType_AVI, etc
You previously set the subtype using the following line:
hr = mediaTypeOut->SetGUID(MF_MT_SUBTYPE, encodingFormat);
// where encodingFormat was hardcoded to MFVideoFormat_WMV3 in the ctor
Keep in mind however, some formats expect bottom-up images. Try changing the file extension to ".mp4", container to MFTranscodeContainerType_AVI, and the subtype to MFVideoFormat_NV12 and you will see your same video upside-down. And your file will be huge comparing to the wmv since all of the YUV values are placed into the file.
Getting the container and format details correct requires a good deal of care, and some details are difficult to research.
Hope this helps.