Why does ReadAllLines work in WPF but not in ConsoleApp

Question

I am doing some benchmarking on different ways to read a CSV file and found a "wierd" problem. The problem being that when I use this method in a console application:

        var lines = File.ReadAllLines(fileName); // OutOfMemoryException
        foreach (var line in lines)
        {
            //doing stuff
        }

I get an OutOfMemoryException, but when I use the same method in my WPF project it works fine. The file im testing this on is 730MB and I know not to use ReadAllLines on bigger CSV files but why does this method work in the WPF application but not in the console application?

How do you know it works? Do you actually get to the foreach loop in the WPF application? — Dirk, Sep 19 '14 at 09:16
Really, for a file that big you would be better to stream it rather than handle it all in one go — Sayse, Sep 19 '14 at 09:19
I get to the foreach loop and I can even show all the lines in a datagrid and as I said I know not to use ReadAllLines on a file that size. — Simon, Sep 19 '14 at 09:29
Meta-comment: I find this question difficult to judge. You claim that `RealAllLines` for huge files works in WPF, but not in a console app. Apart from your claim, we have no example programs (including one that generates the large input file) here that exhibit the problem, i.e. we cannot reproduce the issue. Your question would be much better if you added a minimal WPF application, a minimal console application, and a small program that generates a 730 MB input file that will cause the problem in the console program. — stakx - no longer contributing, Sep 19 '14 at 09:33

score 5 · Accepted Answer · edited Sep 19 '14 at 09:35

5

You're using a 32 bit process and suffering from address space fragmentation. This means that the runtime cannot allocate enough contiguous memory, although in total, enough memory would be free. WPF might just tend to have a better memory layout.

Change the project to be 64 bit.

Or, stream the file using File.ReadLines.

edited Sep 19 '14 at 09:35

stakx - no longer contributing

83,039
20
168
268

answered Sep 19 '14 at 09:18

usr

168,620
35
240
369

Or File.ReadLines(). There is a related post, http://stackoverflow.com/questions/9475355/64bit-console-app-threadpool-out-of-memory. – Peter - Reinstate Monica Sep 19 '14 at 09:21

score 3 · Answer 2 · answered Sep 19 '14 at 09:21

I suspect your WPF application is compiled on Any CPU and you console application on x32 target CPU. If you run the applications on a x64 machine, the WPF application can use a lot more memory compared to the x32 compiled console application, which is limited to roughly 1.5 GB.

Futhermore i suggest to change your approach, and if possible, do not process the entire file at once.

Why does ReadAllLines work in WPF but not in ConsoleApp

2 Answers2