Debug .NET OutOfMemoryException with windbg

Question

I need help to debug a OutOfMemoryException in a .net dll that convert rtf text in raw text or in html.

Here is the code for conversion, (http://matthewmanela.com/blog/converting-rtf-to-html/)

public string ConvertRtfToHtml(string rtfText)
{
    if (rtfText.Equals("")) return "";
    try
    {
        var thread = new Thread(ConvertRtfInSTAThread);
        var threadData = new ConvertRtfThreadData { RtfText = rtfText };
        thread.SetApartmentState(ApartmentState.STA);
        thread.Start(threadData);
        thread.Join();

        return threadData.HtmlText;
    }
    catch (Exception e)
    {
        GestionErreurConv.EnregistrerErreur("Convert", "ConvertRtfToHtml", e.Message);
        return rtfText;
    }
}

private void ConvertRtfInSTAThread(object rtf)
{
    try
    {
        var threadData = (ConvertRtfThreadData)rtf;
        var converter = new RtfToHtmlConverter();
        threadData.HtmlText = converter.ConvertRtfToHtml(threadData.RtfText);
    }
    catch (Exception e)
    {
        GestionErreurConv.EnregistrerErreur("Convert", "ConvertRtfToHtml", e.Message);
    }
}

public class RtfToHtmlConverter
{
    private const string FlowDocumentFormat = "<FlowDocument>{0}</FlowDocument>";

    public string ConvertRtfToHtml(string rtfText)
    {
        var xamlText = string.Format(FlowDocumentFormat, ConvertRtfToXaml(rtfText));
        var converter = new HtmlFromXamlConverter();
        return converter.ConvertXamlToHtml(xamlText, false);
    }

    private string ConvertRtfToXaml(string rtfText)
    {
        string returnString;

        try
        {
            var richTextBox = new RichTextBox
            {
                UndoLimit = 0,
                IsUndoEnabled = false
            };

            var textRange = new TextRange(richTextBox.Document.ContentStart, richTextBox.Document.ContentEnd);

            //Create a MemoryStream of the Rtf content
            using (var rtfMemoryStream = new MemoryStream())
            {
                using (var rtfStreamWriter = new StreamWriter(rtfMemoryStream))
                {
                    rtfStreamWriter.Write(rtfText);
                    rtfStreamWriter.Flush();
                    rtfMemoryStream.Seek(0, SeekOrigin.Begin);

                    //Load the MemoryStream into TextRange ranging from start to end of RichTextBox.
                    textRange.Load(rtfMemoryStream, DataFormats.Rtf);
                }
            }

            using (var rtfMemoryStream = new MemoryStream())
            {
                textRange = new TextRange(richTextBox.Document.ContentStart, richTextBox.Document.ContentEnd);
                textRange.Save(rtfMemoryStream, DataFormats.Xaml);
                rtfMemoryStream.Seek(0, SeekOrigin.Begin);
                using (var rtfStreamReader = new StreamReader(rtfMemoryStream)) { 
                    returnString = rtfStreamReader.ReadToEnd();
                }
            }

            // Libération mémoire
            GC.Collect();
            GC.WaitForPendingFinalizers();

            return returnString;
        }
        catch (Exception)
        {
            // Libération mémoire
            GC.Collect();
            GC.WaitForPendingFinalizers();
            return rtfText;
        }
    }
}

/// <summary>
/// HtmlToXamlConverter is a static class that takes an HTML string
/// and converts it into XAML
/// </summary>
public class HtmlFromXamlConverter
{
    #region Public Methods

    /// <summary>
    /// Main entry point for Xaml-to-Html converter.
    /// Converts a xaml string into html string.
    /// </summary>
    /// <param name="xamlString">
    /// Xaml strinng to convert.
    /// </param>
    /// <returns>
    /// Html string produced from a source xaml.
    /// </returns>
    public string ConvertXamlToHtml(string xamlString, bool asFullDocument)
    {
        var htmlStringBuilder = new StringBuilder(100);

        using (var xamlReader = new XmlTextReader(new StringReader(xamlString)))
        using (var htmlWriter = new XmlTextWriter(new StringWriter(htmlStringBuilder)))
        {
            if (!WriteFlowDocument(xamlReader, htmlWriter, asFullDocument))
            {
                return "";
            }

            return htmlStringBuilder.ToString();
        }
    }

    #endregion Public Methods

    // ---------------------------------------------------------------------
    //
    // Private Methods
    //
    // ---------------------------------------------------------------------

    #region Private Methods
    /// <summary>
    /// Processes a root level element of XAML (normally it's FlowDocument element).
    /// </summary>
    /// <param name="xamlReader">
    /// XmlTextReader for a source xaml.
    /// </param>
    /// <param name="htmlWriter">
    /// XmlTextWriter producing resulting html
    /// </param>
    private bool WriteFlowDocument(XmlTextReader xamlReader, XmlTextWriter htmlWriter, bool asFullDocument)
    {
        if (!ReadNextToken(xamlReader))
        {
            // Xaml content is empty - nothing to convert
            return false;
        }

        if (xamlReader.NodeType != XmlNodeType.Element || xamlReader.Name != "FlowDocument")
        {
            // Root FlowDocument elemet is missing
            return false;
        }

        // Create a buffer StringBuilder for collecting css properties for inline STYLE attributes
        // on every element level (it will be re-initialized on every level).
        var inlineStyle = new StringBuilder();

        if (asFullDocument)
        {
            htmlWriter.WriteStartElement("HTML");
            htmlWriter.WriteStartElement("BODY");
        }

        WriteFormattingProperties(xamlReader, htmlWriter, inlineStyle);

        WriteElementContent(xamlReader, htmlWriter, inlineStyle);

        if (asFullDocument)
        {
            htmlWriter.WriteEndElement();
            htmlWriter.WriteEndElement();
        }
        return true;
    }

    /// <summary>
    /// Reads attributes of the current xaml element and converts
    /// them into appropriate html attributes or css styles.
    /// </summary>
    /// <param name="xamlReader">
    /// XmlTextReader which is expected to be at XmlNodeType.Element
    /// (opening element tag) position.
    /// The reader will remain at the same level after function complete.
    /// </param>
    /// <param name="htmlWriter">
    /// XmlTextWriter for output html, which is expected to be in
    /// after WriteStartElement state.
    /// </param>
    /// <param name="inlineStyle">
    /// String builder for collecting css properties for inline STYLE attribute.
    /// </param>
    private void WriteFormattingProperties(XmlTextReader xamlReader, XmlTextWriter htmlWriter, StringBuilder inlineStyle)
    {
        // Clear string builder for the inline style
        inlineStyle.Remove(0, inlineStyle.Length);

        if (!xamlReader.HasAttributes)
        {
            return;
        }

        bool borderSet = false;

        while (xamlReader.MoveToNextAttribute())
        {
            string css = null;

            switch (xamlReader.Name)
            {
                // Character fomatting properties
                // ------------------------------
                case "Background":
                    css = "background-color:" + ParseXamlColor(xamlReader.Value) + ";";
                    break;
                case "FontFamily":
                    css = "font-family:" + xamlReader.Value + ";";
                    break;
                case "FontStyle":
                    css = "font-style:" + xamlReader.Value.ToLower() + ";";
                    break;
                case "FontWeight":
                    css = "font-weight:" + xamlReader.Value.ToLower() + ";";
                    break;
                case "FontStretch":
                    break;
                case "FontSize":
                    css = "font-size:" + xamlReader.Value + "px;";
                    break;
                case "Foreground":
                    css = "color:" + ParseXamlColor(xamlReader.Value) + ";";
                    break;
                case "TextDecorations":
                    if (xamlReader.Value.ToLower() == "strikethrough")
                        css = "text-decoration:line-through;";
                    else
                        css = "text-decoration:underline;";
                    break;
                case "TextEffects":
                    break;
                case "Emphasis":
                    break;
                case "StandardLigatures":
                    break;
                case "Variants":
                    break;
                case "Capitals":
                    break;
                case "Fraction":
                    break;

                // Paragraph formatting properties
                // -------------------------------
                case "Padding":
                    css = "padding:" + ParseXamlThickness(xamlReader.Value) + ";";
                    break;
                case "Margin":
                    css = "margin:" + ParseXamlThickness(xamlReader.Value) + ";";
                    break;
                case "BorderThickness":
                    css = "border-width:" + ParseXamlThickness(xamlReader.Value) + ";";
                    borderSet = true;
                    break;
                case "BorderBrush":
                    css = "border-color:" + ParseXamlColor(xamlReader.Value) + ";";
                    borderSet = true;
                    break;
                case "LineHeight":
                    break;
                case "TextIndent":
                    css = "text-indent:" + xamlReader.Value + ";";
                    break;
                case "TextAlignment":
                    css = "text-align:" + xamlReader.Value + ";";
                    break;
                case "IsKeptTogether":
                    break;
                case "IsKeptWithNext":
                    break;
                case "ColumnBreakBefore":
                    break;
                case "PageBreakBefore":
                    break;
                case "FlowDirection":
                    break;

                // Table attributes
                // ----------------
                case "Width":
                    css = "width:" + xamlReader.Value + ";";
                    break;
                case "ColumnSpan":
                    htmlWriter.WriteAttributeString("COLSPAN", xamlReader.Value);
                    break;
                case "RowSpan":
                    htmlWriter.WriteAttributeString("ROWSPAN", xamlReader.Value);
                    break;

                // Hyperlink Attributes
                case "NavigateUri":
                    htmlWriter.WriteAttributeString("HREF", xamlReader.Value);
                    break;

                case "TargetName":
                    htmlWriter.WriteAttributeString("TARGET", xamlReader.Value);
                    break;
            }

            if (css != null)
            {
                inlineStyle.Append(css);
            }
        }

        if (borderSet)
        {
            inlineStyle.Append("border-style:solid;mso-element:para-border-div;");
        }

        // Return the xamlReader back to element level
        xamlReader.MoveToElement();
    }

    private string ParseXamlColor(string color)
    {
        if (color.StartsWith("#"))
        {
            // Remove transparancy value
            color = "#" + color.Substring(3);
        }
        return color;
    }

    private string ParseXamlThickness(string thickness)
    {
        string[] values = thickness.Split(',');

        for (int i = 0; i < values.Length; i++)
        {
            if (double.TryParse(values[i], out double value))
            {
                values[i] = Math.Ceiling(value).ToString();
            }
            else
            {
                values[i] = "1";
            }
        }

        switch (values.Length)
        {
            case 1:
                return thickness;
            case 2:
                return values[1] + " " + values[0];
            case 4:
                return values[1] + " " + values[2] + " " + values[3] + " " + values[0];
            default:
                return values[0];
        }
    }

    /// <summary>
    /// Reads a content of current xaml element, converts it
    /// </summary>
    /// <param name="xamlReader">
    /// XmlTextReader which is expected to be at XmlNodeType.Element
    /// (opening element tag) position.
    /// </param>
    /// <param name="htmlWriter">
    /// May be null, in which case we are skipping the xaml element;
    /// witout producing any output to html.
    /// </param>
    /// <param name="inlineStyle">
    /// StringBuilder used for collecting css properties for inline STYLE attribute.
    /// </param>
    private void WriteElementContent(XmlTextReader xamlReader, XmlTextWriter htmlWriter, StringBuilder inlineStyle)
    {
        bool elementContentStarted = false;

        if (xamlReader.IsEmptyElement)
        {
            if (htmlWriter != null && !elementContentStarted && inlineStyle.Length > 0)
            {
                // Output STYLE attribute and clear inlineStyle buffer.
                htmlWriter.WriteAttributeString("STYLE", inlineStyle.ToString());
                inlineStyle.Remove(0, inlineStyle.Length);
            }
            elementContentStarted = true;
        }
        else
        {
            while (ReadNextToken(xamlReader) && xamlReader.NodeType != XmlNodeType.EndElement)
            {
                switch (xamlReader.NodeType)
                {
                    case XmlNodeType.Element:
                        if (xamlReader.Name.Contains("."))
                        {
                            AddComplexProperty(xamlReader, inlineStyle);
                        }
                        else
                        {
                            if (htmlWriter != null && !elementContentStarted && inlineStyle.Length > 0)
                            {
                                // Output STYLE attribute and clear inlineStyle buffer.
                                htmlWriter.WriteAttributeString("STYLE", inlineStyle.ToString());
                                inlineStyle.Remove(0, inlineStyle.Length);
                            }
                            elementContentStarted = true;
                            WriteElement(xamlReader, htmlWriter, inlineStyle);
                        }
                        Debug.Assert(xamlReader.NodeType == XmlNodeType.EndElement || xamlReader.NodeType == XmlNodeType.Element && xamlReader.IsEmptyElement);
                        break;
                    case XmlNodeType.Comment:
                        if (htmlWriter != null)
                        {
                            if (!elementContentStarted && inlineStyle.Length > 0)
                            {
                                htmlWriter.WriteAttributeString("STYLE", inlineStyle.ToString());
                            }
                            htmlWriter.WriteComment(xamlReader.Value);
                        }
                        elementContentStarted = true;
                        break;
                    case XmlNodeType.CDATA:
                    case XmlNodeType.Text:
                    case XmlNodeType.SignificantWhitespace:
                        if (htmlWriter != null)
                        {
                            if (!elementContentStarted && inlineStyle.Length > 0)
                            {
                                htmlWriter.WriteAttributeString("STYLE", inlineStyle.ToString());
                            }
                            htmlWriter.WriteString(xamlReader.Value);
                        }
                        elementContentStarted = true;
                        break;
                }
            }
        }
    }

    /// <summary>
    /// Conberts an element notation of complex property into
    /// </summary>
    /// <param name="xamlReader">
    /// On entry this XmlTextReader must be on Element start tag;
    /// on exit - on EndElement tag.
    /// </param>
    /// <param name="inlineStyle">
    /// StringBuilder containing a value for STYLE attribute.
    /// </param>
    private void AddComplexProperty(XmlTextReader xamlReader, StringBuilder inlineStyle)
    {
        if (inlineStyle != null && xamlReader.Name.EndsWith(".TextDecorations"))
        {
            inlineStyle.Append("text-decoration:underline;");
        }

        // Skip the element representing the complex property
        WriteElementContent(xamlReader, /*htmlWriter:*/null, /*inlineStyle:*/null);
    }

    /// <summary>
    /// Converts a xaml element into an appropriate html element.
    /// </summary>
    /// <param name="xamlReader">
    /// On entry this XmlTextReader must be on Element start tag;
    /// on exit - on EndElement tag.
    /// </param>
    /// <param name="htmlWriter">
    /// May be null, in which case we are skipping xaml content
    /// without producing any html output
    /// </param>
    /// <param name="inlineStyle">
    /// StringBuilder used for collecting css properties for inline STYLE attributes on every level.
    /// </param>
    private void WriteElement(XmlTextReader xamlReader, XmlTextWriter htmlWriter, StringBuilder inlineStyle)
    {
        if (htmlWriter == null)
        {
            // Skipping mode; recurse into the xaml element without any output
            WriteElementContent(xamlReader, /*htmlWriter:*/null, null);
        }
        else
        {
            string htmlElementName;
            switch (xamlReader.Name)
            {
                case "Run" :
                case "Span":
                case "InlineUIContainer":
                    htmlElementName = "SPAN";
                    break;
                case "Bold":
                    htmlElementName = "B";
                    break;
                case "Italic" :
                    htmlElementName = "I";
                    break;
                case "Paragraph" :
                    htmlElementName = "P";
                    break;
                case "BlockUIContainer":
                case "Section":
                    htmlElementName = "DIV";
                    break;
                case "Table":
                    htmlElementName = "TABLE";
                    break;
                case "TableColumn":
                    htmlElementName = "COL";
                    break;
                case "TableRowGroup" :
                    htmlElementName = "TBODY";
                    break;
                case "TableRow" :
                    htmlElementName = "TR";
                    break;
                case "TableCell" :
                    htmlElementName = "TD";
                    break;
                case "List" :
                    string marker = xamlReader.GetAttribute("MarkerStyle");
                    if (marker == null || marker == "None" || marker == "Disc" || marker == "Circle" || marker == "Square" || marker == "Box")
                    {
                        htmlElementName = "UL";
                    }
                    else
                    {
                        htmlElementName = "OL";
                    }
                    break;
                case "ListItem" :
                    htmlElementName = "LI";
                    break;
                case "Hyperlink":
                    htmlElementName = "A";
                    break;
                default :
                    htmlElementName = null; // Ignore the element
                    break;
            }

            if (htmlWriter != null && htmlElementName != null)
            {
                htmlWriter.WriteStartElement(htmlElementName);

                WriteFormattingProperties(xamlReader, htmlWriter, inlineStyle);

                WriteElementContent(xamlReader, htmlWriter, inlineStyle);

                htmlWriter.WriteEndElement();
            }
            else
            {
                // Skip this unrecognized xaml element
                WriteElementContent(xamlReader, /*htmlWriter:*/null, null);
            }
        }
    }

    // Reader advance helpers
    // ----------------------

    /// <summary>
    /// Reads several items from xamlReader skipping all non-significant stuff.
    /// </summary>
    /// <param name="xamlReader">
    /// XmlTextReader from tokens are being read.
    /// </param>
    /// <returns>
    /// True if new token is available; false if end of stream reached.
    /// </returns>
    private bool ReadNextToken(XmlReader xamlReader)
    {
        while (xamlReader.Read())
        {
            switch (xamlReader.NodeType)
            {
                case XmlNodeType.Element: 
                case XmlNodeType.EndElement:
                case XmlNodeType.None:
                case XmlNodeType.CDATA:
                case XmlNodeType.Text:
                case XmlNodeType.SignificantWhitespace:
                    return true;

                case XmlNodeType.Whitespace:
                    if (xamlReader.XmlSpace == XmlSpace.Preserve)
                    {
                        return true;
                    }
                    // ignore insignificant whitespace
                    break;

                case XmlNodeType.EndEntity:
                case XmlNodeType.EntityReference:
                    //  Implement entity reading
                    //xamlReader.ResolveEntity();
                    //xamlReader.Read();
                    //ReadChildNodes( parent, parentBaseUri, xamlReader, positionInfo);
                    break; // for now we ignore entities as insignificant stuff

                case XmlNodeType.Comment:
                    return true;
                case XmlNodeType.ProcessingInstruction:
                case XmlNodeType.DocumentType:
                case XmlNodeType.XmlDeclaration:
                default:
                    // Ignorable stuff
                    break;
            }
        }
        return false;
    }

    #endregion Private Methods
}

}

This dll is used by a Windows Service and ConvertRtfToHtml method is called many times.

This is windbg informations :

    0:016> !sos.clrstack
OS Thread Id: 0x220c (16)
Child SP       IP Call Site
127beb0c 755bc232 [GCFrame: 127beb0c] 
127bebcc 755bc232 [HelperMethodFrame_2OBJ: 127bebcc] System.Environment.GetResourceFromDefault(System.String)
127bec50 10fa493c System.Environment.GetResourceString(System.String, System.Object[])
127bec60 10fa48af System.Exception.get_Message()
127bec70 069077d9 *** WARNING: Unable to verify checksum for Convertisseur.dll
SQWebContributeur.Convertisseur.Convert.ConvertRtfInSTAThread(System.Object) [D:\SOLU-QIQ\Projets SVN\DLL Maison\Convertisseur\Convertisseur\Convert.cs @ 268]
127bed94 069052d4 System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
127beda0 063e2c17 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
127bee10 063e2177 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
127bee24 06905162 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
127bee3c 069050e3 System.Threading.ThreadHelper.ThreadStart(System.Object)
127bef80 730eebf6 [GCFrame: 127bef80] 
127bf164 730eebf6 [DebuggerU2MCatchHandlerFrame: 127bf164] 


0:016> !pe -nested
Exception object: 019bbfc0
Exception type:   System.Runtime.InteropServices.COMException
    Message:          Espace insuffisant pour traiter cette commande. (Exception de HRESULT : 0x80070008)
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    00000000 00000001 UNKNOWN!System.Environment.GetResourceFromDefault(System.String)+0x2
    127BEC50 10FA493C UNKNOWN!System.Environment.GetResourceString(System.String, System.Object[])+0xc
    127BEC60 10FA48AF UNKNOWN!System.Exception.get_Message()+0x4f
    127BEC70 069077D9 Convertisseur_ae70000!SQWebContributeur.Convertisseur.Convert.ConvertRtfInSTAThread(System.Object)+0xe9
    127BED94 069052D4 UNKNOWN!System.Threading.ThreadHelper.ThreadStart_Context(System.Object)+0x9c
    127BEDA0 063E2C17 UNKNOWN!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x107
    127BEE10 063E2177 UNKNOWN!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x17
    127BEE24 06905162 UNKNOWN!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x3a
    127BEE3C 069050E3 UNKNOWN!System.Threading.ThreadHelper.ThreadStart(System.Object)+0x4b

StackTraceString: <none>
HResult: 80070008

Nested exception -------------------------------------------------------------
Exception object: 019b9dc4
Exception type:   System.OutOfMemoryException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    00000000 00000001 UNKNOWN!System.GC._WaitForPendingFinalizers()+0x2
    127BEB68 10FA11DF UNKNOWN!System.GC.WaitForPendingFinalizers()+0x4f
    127BEB98 0C55631D Convertisseur_ae70000!SQWebContributeur.ClassesHelp.RtfToHtmlConverter.ConvertRtfToXaml(System.String)+0x385
    127BED00 069078C9 Convertisseur_ae70000!SQWebContributeur.ClassesHelp.RtfToHtmlConverter.ConvertRtfToHtml(System.String)+0x51
    127BED38 0690779C Convertisseur_ae70000!SQWebContributeur.Convertisseur.Convert.ConvertRtfInSTAThread(System.Object)+0xac

StackTraceString: <none>
HResult: 8007000e

!eeheap -gc command show that 4 Mo are used by garbage collector :

0:016> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x019b9db8
generation 1 starts at 0x019b9cec
generation 2 starts at 0x01861000
ephemeral segment allocation context: none
 segment     begin  allocated      size
01860000  01861000  01bec808  0x38b808(3717128)
Large object heap starts at 0x02861000
 segment     begin  allocated      size
02860000  02861000  028ba260  0x59260(365152)
Total Size:              Size: 0x3e4a68 (4082280) bytes.
------------------------------
GC Heap Size:    Size: 0x3e4a68 (4082280) bytes.

!dumpheap -stat command shows that only 2 Mo are free :

00e6a430      865      2382762      Free

Here is perfmon data :

I don't know what to do to resolve this exception. I try to add GC.Collect() to force GC without any effect.

The VM have 8 Go of physical memory and on another VM with 4 Go, exception not occurs. I don't know how can I resolve this exception.

Thanks for your help

Answer 1

First of all, you did a good job extracting all the inner exceptions in order to figure out that this crash is caused by a OOM exception. Not all developers have that skill.

!eeheap -gc command show that 4 Mo are used by garbage collector

That's correct - and a strong indicator that garbage collecting on its own will not really help. Even if it could free the 4 MB, you have gained almost nothing.

(But: more on that later)

!dumpheap -stat command shows that only 2 Mo are free

While this statement is not wrong, it's also not complete.

a) there are 2 MB free, but those 2 MB are split into 865 different regions. So it might still be impossible to allocate a single 2 MB block

b) these 2 MB are free from .NET point of view. If .NET does not have enough memory free, it will request more memory from the operating system. That request may then succeed or fail, depending on how much memory the OS is available to give.

With that knowledge in mind, you need to ask

why wasn't the operating system able to give .NET more memory?

The reason likely is: because it has given all memory away already. In a 32 bit process, that's 2 GB, 3 GB or 4 GB, depending on configurations and settings (mainly Large Address Aware). That's not much, and especially, it's not available as a contiguous block. In many cases, you only have 700 MB.

Where could the OS have given the memory? To COM objects in your case (because we have a COM exception, but that may be misleading). And these COM objects seem to be native (otherwise they would allocate managed memory). Looking at .NET memory will not help then.

However, there is one exception: if your .NET code is the reason that COM objects are not freed, then your .NET code is indirectly responsible for the native memory leak. So, what you should be looking for is the amount of RCW objects. If you have a lot of them, you need to somehow get rid of them.

If that's not the reason, maybe your RTF is simply too large and does not fit into the largest region of free memory.

I have once made up a graph for tackling down OOM exceptions that tells you where to start.

With !address -summary you take a look at the point of view of the operating system.

You likely have a small <unknown> value, since the .NET usage is small.

If Heap has a large value, the memory is gone via the Windows Heap Manager (e.g. C++) and you have a native leak (possibly caused by COM objects not being released).

You can also have a look at the section "largest region by size", where you'll find a value for Free. That's the maximum value someone can get with a single request. Maybe it's not enough to fit your data.