Memory leak in XmlSerializer as used in VB.net

Question

I seem to have a memory leak when using xmlSerializer in VB.net 2010.
The code below is the guts of a function which is called every 20 seconds or so forever.
Uncommenting the line 'serializer.Serialize(writer, response)' sees the memory usage climb.

 Using writer As New StreamWriter(fileName)
    Dim request = invoker.CreateRequest()
    Dim response = invoker.Invoke(authorization, request)
    If Not response Is Nothing Then 'maybe token timeout
        Dim serializer = New XmlSerializer(GetType(TResponse)) ', New XmlRootAttribute("TResponse"))
        Try
            '  serializer.Serialize(writer, response)

        Catch e As InvalidOperationException
            Dim msg As String = "ERROR: Failed to serialize the response:" & vbCrLf & e.Message
            AddToRtfEventLog(category.ToUpper & " DATA " & msg, Drawing.Color.Red)
        End Try
        serializer = Nothing
        AddToStatus(Convert.ToString("Response saved as ") & fileName & vbCrLf, Drawing.Color.LightGray, True)
    End If
    Return response
End Using

From internet reading, I gather this is a bug in the VB xmlSerializer and a possible solution is to use it in a caching manner. Unfortunately I have no idea how to do that so I wonder if someone can help me out here. Firstly, am I doing it right in the first place (it works fine, just the memory issue) and Secondly, if I am, can you show me how to fix the leak please?

Thanks
Malcom

Part 2-
With help from Jehof, I've now got the following cache class in place: Public NotInheritable Class XmlSerializerCache Private Sub New() End Sub Private Shared ReadOnly cache As New Dictionary(Of String, XmlSerializer)()

Public Shared Function Create(type As Type, root As XmlRootAttribute) As XmlSerializer
    Dim key = [String].Format(CultureInfo.InvariantCulture, "{0}:{1}", type, root.ElementName)

    If Not cache.ContainsKey(key) Then
        cache.Add(key, New XmlSerializer(type, root))
    End If

    Return cache(key)
End Function

End Class

and have replaced the calls with :

 Dim xmlRootAttribute = New XmlRootAttribute("TResponse")
                    Dim serializer = XmlSerializerCache.Create(GetType(TResponse), XmlRootAttribute)
                    Try
                        serializer.Serialize(writer, response)

Thats has improved the leak but is the correct way to do this? To me it seems that I am recreating the cache with each call?