0

I am Reading a binary file into memory and I need that data through the whole execution. However, I am getting an error that the datastream was too long. Any idea how to solve this issue? I have compiled the application as a x64 application. I need the whole the data in memory due to the fact that the binary file consists of data and pointers. The pointer could point out data, through the whole file. Using pointers instead of store data several times will save discspace. 

try
        {
            using (FileStream fileStream = File.OpenRead(args[0]))
            {
                MemoryStream memoryStream = new MemoryStream();
                fileStream.CopyTo(memoryStream);
                memoryStream.Seek(0, SeekOrigin.Begin);
Phero
  • 11
  • 3
  • `I need that data through the whole execution` Why can't you process it while reading from file instead of loading whole file to memory? Remember old days. People processed very large data with only 64K of RAM. – Eser Apr 14 '18 at 19:12
  • In this particular case I need all the data. When I pass the binary file, there are two parts; data and pointers. Every field of data, is stored once, so there will be no redundancy; the data is database tables stored in a binary file. The pointers will then point out the data, row by row and solve the issue if the fielddata occurs several times. This will make the file much smaller. – Phero Apr 15 '18 at 14:49
  • So, if your data is 10TB, you need at least 10TB of memory? doesn't it sound weird? – Eser Apr 15 '18 at 16:15
  • Perhaps it does. I have not seen bigger files than 10 Gigabyte. – Phero Apr 15 '18 at 20:16

0 Answers0