System out of memory exception large txt file into an array

Question

The following code works fine with small txt files , but if we have large txt files its giving outofmemory exception at string[] array = File.ReadAllLines("hash.txt");

hash.txt file is a 500 mb

I tried few suggestions from internet but i didn't get that worked.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
namespace Hash_Parser
{
    internal class Program
    {
        private static List<string> users = new List<string>();
        private static Dictionary<string, int> hash_original = new Dictionary<string, int>();
        private static List<string> hash_found = new List<string>();
        private static List<string> pass = new List<string>();
        private static string hash_path = "split.txt";
        private static void split()
        {
            Console.WriteLine("Splitting...");
            StreamWriter streamWriter = new StreamWriter("user.txt");
            StreamWriter streamWriter2 = new StreamWriter("hash.txt");
            string[] array = File.ReadAllLines(Program.hash_path);
            for (int i = 0; i < array.Length; i++)
            {
                string text = array[i];
                string[] array2 = text.Split(new char[]
               {
                   ':'
               }, 2);
                if (array2.Count<string>() >= 2)
                {
                    streamWriter.WriteLine(array2[0]);
                    streamWriter2.WriteLine(array2[1]);
                }
            }
            streamWriter.Close();
            streamWriter2.Close();
            Console.WriteLine("Saved as user.txt and hash.txt");
        }
        private static void populate()
        {
            Console.WriteLine("Populating lists...");
            Program.users.AddRange(File.ReadAllLines("user.txt"));
            Program.pass.AddRange(File.ReadAllLines("pass.txt"));
            Program.hash_found.AddRange(File.ReadAllLines("found.txt"));
            int num = 0;
            string[] array = File.ReadAllLines("hash.txt");
            for (int i = 0; i < array.Length; i++)
            {
                string key = array[i];
                Program.hash_original.Add(key, num);
                num++;
            }
        }
        private static void seek()
        {
            StreamWriter streamWriter = new StreamWriter("userpass.txt");
            int num = 0;
            int num2 = 100;
            foreach (string current in Program.hash_found)
            {
                if (Program.hash_original.ContainsKey(current))
                {
                    streamWriter.WriteLine(Program.users[Program.hash_original[current]] + ":" + Program.pass[num]);
                }
                num++;
                if (num >= num2)
                {
                    Console.Title = string.Concat(new object[]
                   {
                       "Processed: ",
                       num,
                       " : ",
                       Program.hash_found.Count
                   });
                    num2 += 1000;
                }
            }
            Console.Title = string.Concat(new object[]
           {
               "Processed: ",
               num,
               " : ",
               Program.hash_found.Count
           });
            streamWriter.Close();
        }
        private static void Main(string[] args)
        {
            Console.WriteLine("Split hash /split");
            Console.WriteLine("'split.txt'\n");
            Console.WriteLine("Parse hashes /parse");
            Console.WriteLine("'user.txt' | 'found.txt' | 'hash.txt' | 'pass.txt'");
            string a = Console.ReadLine();
            if (a == "/split")
            {
                Program.split();
            }
            else
            {
                if (a == "/parse")
                {
                    Program.populate();
                    Console.WriteLine("Processing...");
                    Stopwatch stopwatch = new Stopwatch();
                    stopwatch.Start();
                    Program.seek();
                    stopwatch.Stop();
                    Console.WriteLine("Saved as userpass.txt");
                    Console.WriteLine("Time elapsed: " + stopwatch.Elapsed);
                    Console.ReadKey();
                }
            }
        }
    }
}

Thnaks for ur help.

Even if you read the file line by line you would still get out of memory exception? — John Odom, May 04 '16 at 21:57
If having 500MB provokes this error try to load it in chunks — Luis Filipe, May 04 '16 at 21:58
You can also read the file one line at a time, process it, and write one line to the output. — Babak Naffas, May 04 '16 at 21:58
Why not loop through the file and do smaller sections at a time? — user3272686, May 04 '16 at 21:58

score 4 · Answer 1 · edited May 23 '17 at 12:16

4

Try this code :

foreach (var line in File.ReadLines(_filePath))
{
    //Don't put "line" into a list or collection.
    //Just make your processing on it.
}

Quoted Text: Just use File.ReadLines which returns an IEnumerable and doesn't load all the lines at once to the memory.

Quote Link : https://stackoverflow.com/a/13416225/3041974

I hope it helps.

edited May 23 '17 at 12:16

Community

1
1

answered May 04 '16 at 21:58

rootturk

316
4
20

Worked...THanks a lot – pgems sri May 05 '16 at 05:10

score 0 · Answer 2 · answered May 04 '16 at 22:44

Please be aware of process limits in .NET

http://www.codeproject.com/Articles/483475/Memory-Limits-in-a-NET-Process

For instance, a 32 bit system cannot have more than 4 GB of physical memory. Needless to say that 2^32 will give you a virtual address space with 4.294.967.296 different entries, and that’s precisely where the 4GB limit comes from. But even having those 4GB available on the system, your application will actually be able to see 2GB only. Why?

Because on 32 bits systems, Windows splits the virtual address space into two equal parts: one for User Mode applications, and another one for the Kernel (system applications). This behavior can be overridden by using the "/3gb" flag in the Windows boot.ini config file. If we do so, the system will then reserve 3GB for user applications, and 1 GB for the kernel.

What is the process MEM Usage in Task Manager?

System out of memory exception large txt file into an array

2 Answers2