Getting many system out of memory exceptions

Question

My program reads data off of a datatable using linq queries and writes it to a local file. I have the stream and the streamwriter wrapped in using statements but I'm getting many system out of memory exceptions. Am I supposed to call the dispose method on the datatables as well in my code? That is the only solution I can think of

My code:

public static async void getTrainingData(string symbol, string market)
    {
        decimal returnPct = 0, shortRating = 0, rsi = 0, mfi = 0, williamsR = 0, macd = 0, sma20 = 0, sma50 = 0, sma200 = 0;
        string path = "training.csv";
        List<Task> taskList = new List<Task>();
        int count = 0;

        try
        {
            using (Calculations calc = createCalcClass(symbol, market))
            {
                using (FileStream fs = new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.Write, 4096, FileOptions.Asynchronous))
                using (TextWriter writer = new StreamWriter(fs))
                {
                    DateTime date;

                            var amexQuery = from c in calc.dailyAmexTable.AsEnumerable()
                                            orderby c.Date descending
                                            select c;

                            count = Convert.ToInt32(calc.dailyAmexAdapter.ScalarQuerySymbol(symbol));
                            for (int i = 0; i < count - 14; i++)
                            {
                                if (i != count - 15)
                                {
                                    returnPct = calc.calculateReturnPercentage(amexQuery.ElementAtOrDefault(i + 1).AdjustedClose, amexQuery.ElementAtOrDefault(i).AdjustedClose);
                                    date = amexQuery.ElementAtOrDefault(i + 1).Date;

                                    if (returnPct >= 10)
                                    {
                                        calc.calculatePctGainsLosses(14, date);
                                        shortRating = calculateRating(calc, ratingType.short_rating, date);
                                        rsi = calc.calculateRSI(14, date);
                                        mfi = calc.calculateMFI(14, date);
                                        williamsR = calc.calculateWilliamsR(14, date);
                                        macd = calc.calculateMACDDivergence(date);
                                        sma20 = calc.calculateReturnPercentage(calc.calculateSMA(20, date),
                                            calc.getSpecificValues(Calculations.valueType.Most_Recent_Close, 0, date));
                                        sma50 = calc.calculateReturnPercentage(calc.calculateSMA(50, date),
                                            calc.getSpecificValues(Calculations.valueType.Most_Recent_Close, 0, date));
                                        sma200 = calc.calculateReturnPercentage(calc.calculateSMA(200, date),
                                            calc.getSpecificValues(Calculations.valueType.Most_Recent_Close, 0, date));
                                        Task write = TextWriter.Synchronized(writer).WriteLineAsync(returnPct + "," + symbol + "," + shortRating + "," +
                                            rsi + "," + mfi + "," + williamsR + "," + macd + "," + sma20 + "," + sma50 + "," + sma200);
                                        taskList.Add(write);
                                        // add the necessary info to the training data file
                                        // stock rating, pct change, rsi, mfi, williams r, macd, 20 day sma, 50 day sma, 200 day sma
                                    }
                                }
                            }

                    await writer.FlushAsync();
                }

                // wait for all tasks to complete
                Task.WaitAll(taskList.ToArray());
                Console.WriteLine("Return percentages for " + symbol + " saved to training file.");
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
    }

Example output:

example output

In reference to your specific question about disposing datatables: http://stackoverflow.com/questions/913228/should-i-dispose-dataset-and-datatable — JohnnyHK, Jul 03 '14 at 02:49
Yes thank you I did read that article. That was the only idea that came to mind but it still seems to have that exception thrown so I am at a loss what else I could be missing in my code — DarthVegan, Jul 03 '14 at 02:57
Have you profiled your application to learn where the memory is going? — JohnnyHK, Jul 03 '14 at 02:59
Ok correction I did find the option in visual studio and I'm running it now but I'm not sure where you go to see where the memory leak could be — DarthVegan, Jul 03 '14 at 03:02
Difficult to say without knowing what all your functions are doing. You need to try and narrow the problem down. — Matt Burland, Jul 03 '14 at 03:07
For one thing, the `amexQuery.Count()` calls are demanding the entire list into memory at once. Admittedly it is rather difficult to return all but the last N elements of a list lazily. — jaket, Jul 03 '14 at 04:04
You won't see memory leaks in C# easily. Just check whether there are too many objects allocated. If you are not using Ultimate edition you will likely need to dump heaps between different timestamps and diff them with something like http://thinkexception.blogspot.de/2010/06/tool-to-bompare-two-windbg-dumpheap.html — Samuel, Jul 03 '14 at 05:19
Ok I updated my above code and to show you guys what the output is. Any ideas on why I'm still getting out of memory exceptions and why the output looks like that? — DarthVegan, Jul 04 '14 at 03:23
Found this question while I was googling why FileMode.Append can cause OutOfMemory Exceptions. When I switched to FileMode.Open my OutOfMemory Exceptions were gone — Chizh, Apr 08 '16 at 19:07

Getting many system out of memory exceptions

0 Answers0