StreamReader Gets Stuck on Large Files

Question

I'm trying to read a 50GB text file with a streamreader. It doesn't even get to reading the file, because my program freezes on the second line(of code, not the file).

string filename = @"C:\wikipedia\dump.xml";
StreamReader wikipediaReader = new StreamReader(filename);
string line = "";
while((line = wikipediaReader.ReadLine()) != null)
{
    Console.WriteLine(line);
    Console.ReadKey();
}

Are you sure its frozen and not working? If you put long running operations on the main thread the program will appear frozen. — eddie_cat, Sep 08 '14 at 21:43
@eddie_cat It gets stuck on the second line. I left it for about half an hour. — user1599078, Sep 08 '14 at 21:43
When you leave it running, does your process memory usage increase? — Blorgbeard, Sep 08 '14 at 21:53
@MarkLarter Not sure. I don't understand why it would change anything, it doesn't even get to reading the file. — user1599078, Sep 08 '14 at 22:03
Is there any chance that the file is locked in some way? Like maybe it is open in an editor or something. — Stilgar, Sep 08 '14 at 22:03
What happens if you first open the file as a READONLY stream and then attach the streamreader to that stream? — lboshuizen, Sep 08 '14 at 22:04
I was wondering, if the file contains Unix newlines instead of Windows newlines, would @JoelCoehoorn's comment nail it? — Mark Larter, Sep 08 '14 at 22:10
I would break and look at the call stack and see where it is that it is stuck inside of the `StreamReader` constructor. That might give us a clue. — Mike Zboray, Sep 08 '14 at 22:12
This is your anti-malware software getting involved. No, you cannot see it from the call stack, it is off in the woods trying to figure out what to do with 50 jiggabytes of potential malware. — Hans Passant, Sep 08 '14 at 22:21
@HansPassant Wouldn't we see an exception because the file was locked or inaccessible in that case? — Mike Zboray, Sep 08 '14 at 22:24
I should probably tell you that when I run your code or my code, the only way to stop the program is to log out and back in. I've tried task manager, ending the process tree, and command prompt. — user1599078, Sep 08 '14 at 22:26
@user - XML files are very good at hosing your machine, any anti-malware will put them on the top of the list. Their DTD is a great DoS attack. — Hans Passant, Sep 08 '14 at 22:28
@user1599078 What does taskmanager tell you about diskactivity or cpu? — lboshuizen, Sep 08 '14 at 22:28
File extensions like xml are a convenience, not a definitive statement about what's in a file. How does the antivirus software know it's really just xml unless it looks? — Joel Coehoorn, Sep 08 '14 at 22:29
Anyway, would it hurt to try this with the AV software disabled, or with an exemption for the file? — Joel Coehoorn, Sep 08 '14 at 22:30
Then very likely something is reading the file, either the Framework or indeed some AV software. @HansPassant could certainly have a point here — lboshuizen, Sep 08 '14 at 22:32
Well, there you go. Who needs a virus when the anti-dote is indistinguishable from the problem. At least mention the name so everybody learns what to avoid. — Hans Passant, Sep 08 '14 at 23:01
@HansPassant Since we have a solution, you should post your comment as an answer. — Mephy, Sep 08 '14 at 23:41

score 2 · Accepted Answer · answered Sep 24 '14 at 01:51

2

It could be anti-virus software trying to scan the file.

answered Sep 24 '14 at 01:51

Joel Coehoorn

399,467
113
570
794

Community Wiki so I don't take credit for @HansPassant's work. I just wanted to get this out here as a real answer. – Joel Coehoorn Sep 24 '14 at 01:51

score 0 · Answer 2 · answered Sep 10 '14 at 11:54

0

You can also try using MemoryMappedFile to read file

answered Sep 10 '14 at 11:54

Araym

19
4

score -1 · Answer 3 · edited May 23 '17 at 11:49

-1

we have to use splitter and set the start position and endposition while reading the file

i advice you to check This Link.

if you still have issue , please let me know i will help you with the working code

edited May 23 '17 at 11:49

Community

1
1

answered Sep 10 '14 at 11:57

Fanish

27
5

StreamReader Gets Stuck on Large Files

3 Answers3