0

Following is the code which causing the issue.

public static byte[] ToByteArrayWithBom(this string s, Encoding encoding) {
  var data = encoding.GetBytes(s);
  long str = data.Length; // 323603364
  var premble = encoding.GetPreamble();
  var concatData = premble.Concat(data);
  return concatData.ToArray();
  // return encoding.GetPreamble().Concat(data).ToArray();
}

The function will work if there are less amount of data passed to this function.
When we are going to pass large amount of data then it is throw exception. This is the place where we are facing problem concatData.ToArray();. When we are converting var data into .ToArray() then it is thrown exception.
I am not sure is there any max limit of .ToArray() which will cause this issue or there is any other problem.

Exception Detail:

System.OutOfMemoryException: Exception of type System.OutOfMemoryException was thrown
at System.Linq.Buffer…ctor(IEnumerable1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable1 source)

VillageTech
  • 1,968
  • 8
  • 18
Shrimant Telgave
  • 1
  • 2
  • You will have to return the data as a stream – meJustAndrew Dec 19 '19 at 09:15
  • 3
    You need 300MB, twice. That should be possible on a 64bit platform but first make sure you didn't select "prefer 32bit" or anything similar. – H H Dec 19 '19 at 09:16
  • 2
    I doubt it´s a limitation of `ToArary`. Simple answer is: you´re out of memory. For 32bit-platform this is 2GB (or with LargeAdressAware enabled 3GB). So even with 100GB of memory, your process can at most allocate 3GB. – MakePeaceGreatAgain Dec 19 '19 at 09:18
  • If you want to write the data back to the *file* (the same text but with BOM - Byte Order Mark), you can return `IEnumerable` instead of `byte[]` (without `.ToArray()`) but still use `File.WriteAllBytes(@"c:\myFile.txt", ToByteArrayWithBom(...))` – Dmitry Bychenko Dec 19 '19 at 09:21
  • @HenkHolterman I have checked it 64 bit platform. the same function is working for less amount of data. – Shrimant Telgave Dec 19 '19 at 09:30
  • @DmitryBychenko I will check but do you know what is root cause of the issue. The given problem is not for "Byte[]" return type. problem when converting var to .ToArray() . – Shrimant Telgave Dec 19 '19 at 09:33
  • This is a similar question. I think these are the answers you are looking for: https://stackoverflow.com/questions/8563933/c-sharp-out-of-memory-exception – Mardukar Dec 19 '19 at 09:44
  • In a 64 bit app you should not have this problem... But as a BOM only makes sense in a UTF8 file (stream), why are you keeping this in memory (2x) at all? The best answer might be not to us ToArray() – H H Dec 19 '19 at 10:19
  • @HenkHolterman I have used UTF8 file. code like this. var rptData= helper.WriteString(rows).ToByteArrayWithBom(Encoding.UTF8); – Shrimant Telgave Dec 19 '19 at 13:27
  • That does not look like 'using a file'. – H H Dec 19 '19 at 13:31

1 Answers1

0

You really don't want to use Linq just to prepend a BOM to a huge byte array. Writing bytes into a memory buffer, directly from the string encoding process would be much better.

Better still would be rebuilding your entire data path around streams.

Jeremy Lakeman
  • 9,515
  • 25
  • 29