Getting OutOfMemory exception threading

Question

I'm just getting started with multi-threading. I am running a test of my multi-thread code but I'm getting an OutOfMemory exception.

The code is converting PS to PDF using a new thread. The task takes about half a second so for this test, I'm simply sleeping the main thread for a second to make sure I don't have too many tasks running. It did more than 900 before throwing an OutOfMemory exception.

I know I need to use Thread Pool, Semaphore or Task Parallel to limit my threads, but for now I'm just doing a test of my threads.

Dim sr As New StreamReader(PSTempFolder & "PDFWrite.txt")

Do While Not sr.EndOfStream

    'get PS
    Dim FileNamePS As String = sr.ReadLine

    'get folder
    Dim CustFolder As IO.DirectoryInfo
    CustFolder = GetCustFolder(FileNamePS)

    'set PDF path and name
    FileNamePDF = CustFolder.FullName & "\Statement.pdf"

    Dim t As Thread
    Dim n As ConvertPDF = Nothing
    n = New ConvertPDF
    n.DeletePS = False
    n.PSFileName = FileNamePS
    n.PDFFileName = FileNamePDF

    t = New Thread(AddressOf n.callConvertToPDF)
    t.Start()

    'wait
    Thread.Sleep (1000)

Loop

sr.Close()

It seems it must be creating too many threads and not cleaning up the old ones. How do I clean up/dispose of the thread before creating a new one?

I suppose a second solution (in this context) would be simply using the same thread (I think I can do that), but for this question I'm more interested on disposing of the thread and releasing the memory. How do I do that?

Here is the rest of the code:

Class ConvertPDF

    Public PSFileName As String
    Public PDFFileName As String
    Public DeletePS As Boolean = False

    Delegate Function ConvertToPDFdel(ByVal svPsFileName As String, _
                     ByVal svPDFName As String, _
                     ByVal DeletePS As Boolean) As Integer

    Sub callConvertToPDF()
        Dim dlgt As New ConvertToPDFdel(AddressOf ConvertToPDF)
        Dim i As Integer = dlgt.Invoke(PSFileName, PDFFileName, DeletePS)
    End Sub

End Class

Public Function ConvertToPDF(ByVal svPsFileName As String, _
                             ByVal svPDFName As String, _
                             ByVal DeletePS As Boolean) As Integer

    'check for file
    If Not IO.File.Exists(svPsFileName) Then
        Throw New ApplicationException(svPsFileName & " cannot be found")
    End If

    'delete old file
    If IO.File.Exists(svPDFName) Then IO.File.Delete(svPDFName)

    'convert
    Dim myProcInfo As New ProcessStartInfo
    myProcInfo.FileName = DanBSolutionsLocation & "Misc\GhostScript\GSWIN32C.EXE"
    myProcInfo.Arguments = "-sDEVICE=pdfwrite -q -dSAFER -dNOPAUSE -sOUTPUTFILE=""" & svPDFName & """ -dBATCH """ & svPsFileName & """"
    'Debug.Print(myProcInfo.Arguments)

    'do the conversion
    Dim myProc As Process = Process.Start(myProcInfo)

    'wait for finish (no more than 20 seconds)
    myProc.WaitForExit(20000)

    myProcInfo = Nothing
    myProc.Dispose()

    'delete PS
    If DeletePS Then
        If IO.File.Exists(svPDFName) Then IO.File.Delete(svPsFileName)
    End If

End Function

EDIT: I did some more testing between GroverBoy's code and mine and the results are inconclusive. Sometimes one is better sometimes the other. Maybe the two really are the same and the problem is elsewhere.

The new thread starts a new process that takes 0.55 seconds to complete. If the main thread waits 1 second each iteration, that should mean that we'll never have more than one thread or one open file at a time. Why isn't this true?

What actually happens will vary and I'm not sure why. I'm testing with a loop of 100 and a 1 second wait on the main thread. I usually watch the Performance tab of the Task Manager. Sometimes I run the code and number of threads will fluctuate between 2-6 extra and the Commit Charge will fluctuate between 1044M to 1150M. This is what I want.

Other times I run the same code (100 iterations) and the number of threads keeps rising to more than 63 extra. And the Commit Charge keeps rising from 1044M to more than 1272M.

What can I do to ensure that the program will clean up the threads consistently?

Answer 1

Another answer is to use Thread.Join without using GC.Collect. This keeps the main thread waiting until the new thread finished.

t.Start(Params)

Params = Nothing

t.Join()

Using this method the threads and Commit Charge rose a bit and then stayed steady. They did not keep accumulating.

Answer 2

I'm making a wild guess that your code causes OutOfMemoryException because it creates but never destroys 900 (or whatever) instances of ConvertPDF. Of course it's possible that some other of your code (not shown) causes the problem. Anyway here goes ...

Let's suppose that ConvertPDF implements IDisposable, which means that after using it you need to call ConvertPDF.Dispose or, better, use ConvertPDF in a Using clause to call Dispose automatically. Your code isn't structured to do this at an appropriate time, because it has no way of knowing when callConvertToPDF has completed executing. You can restructure so that the worker thread also does the work of initializing and disposing an instance of ConvertPDF.

The code below adds a helper class Paths, to function as a parameter for the worker thread. Warning: I don't really develop in VB.NET so this may not compile :)

Class Paths
    Public FileNamePS As String
    Public FileNamePDF As String
End Class

Sub Main()
    Using sr As New StreamReader(PSTempFolder & "PDFWrite.txt")
        Do While Not sr.EndOfStream
            Dim MyPaths As Paths = New Paths()

            'get PS
            MyPaths.FileNamePS = sr.ReadLine

            'get folder
            Dim CustFolder As IO.DirectoryInfo = GetCustFolder(MyPaths.FileNamePS)

            'set PDF path and name
            MyPaths.FileNamePDF = IO.Path.Combine(CustFolder.FullName, "Statement.pdf")

            Dim t As Thread = New Thread(AddressOf ConvertPStoPdf)

            ' start the thread, passing the parameter that ConvertPStoPdf will need
            t.Start(MyPaths)

            'wait
            Thread.Sleep (1000)
        Loop
    End Using ' automatically disposes StreamReader
End Sub

Sub ConvertPStoPdf(Data As Object)
    ' get Paths instance from weak-typed parameter
    Dim MyPaths As Paths = CType(Data, Paths)

    Using C As ConvertPDF = New ConvertPDF        
        C.DeletePS = False
        C.PSFileName = MyPaths.FileNamePS
        C.PDFFileName = MyPaths.FileNamePDF
        C.callConvertToPDF            
    End Using ' automatically disposes ConvertPDF
End Sub

Answer 3

I found one answer to force reclaiming the memory is using GC.Collect. Rico's blog: When to call GC.Collect()

t.Start (Params)

Params = Nothing

Thread.Sleep (1000)

GC.Collect()
GC.WaitForPendingFinalizers()
GC.Collect()
GC.WaitForPendingFinalizers()

Code is same as what is used to release Excel from this page.

I realize rule #1 is don't use GC.Collect. So is there a better answer?

With this method the threads didn't accumulate and the Commit Charge didn't rise. I won't be getting out-of-memory exception with this. But I would be glad to hear of a better answer. I really don't want to use Thread.Sleep in production code.