0

Is there a way to tell Windows to give a program more memory? I'd rather do that than re-code a straightforward program that otherwise works well.

We have a C# program that runs every 15 minutes and compares a new incoming file to the previous file. Both the new and old file are "|" separated and read into memory. We haven't tried optimizing, just read everything into dictionaries and do a straightforward comparison.

We're getting this error,

Message: Exception of type 'System.OutOfMemoryException' was thrown. at 
System.String.SplitInternal(Char[] separator, Int32 count, 
                            StringSplitOptions options) 
at System.String.Split(Char[] separator)

Each of these files have about a 1000 lines and each line has a base64 encoded large image as one of the pipe-separated fields. Each line can be 500K-700K so the total size of each file is around 600MB. Maybe an occasional line exceeds 1MB.

We estimate the program uses 1.5GB of RAM or so. I'd like to just give it more RAM instead of trying to optimize the code. Alternatively it would be nice to know if the program as a whole is running out of memory or one particular huge line may cause SplitInternal to throw this exception.

Windows Server 2012R2.

Edits: The offending line of code is just "string[] data = info.Split('|');" inside of a loop. After this exception is caught, the program continues and processes many other lines just fine.

Not averse to rewriting it, but if there's an easier way I'd like to try that first. We confirmed it was compiled with the "Prefer 32 bit" checkbox, so first we will try unchecking that and see what happens. After all, 2GB isn't really a whole lot these days on a server...

royappa
  • 663
  • 1
  • 9
  • 20
  • 3
    you have a memory leak there. Adding up more RAM isn't the solution rather check where in your app code you are leaking it – Rahul Oct 17 '17 at 11:50
  • You should really change the code to something that does not require 1.5 GB of strings in Memory. Beside OOM Exception do very often mean something different than actually being out of physical Memory, like Addresses in a 32 Bit process, or fragmentation, or.... – quadroid Oct 17 '17 at 11:53
  • `Alternatively it would be nice to know if the program as a whole is running out of memory or one particular huge line may cause SplitInternal to throw this exception.` Another nice thing to know might be the code you are running... – mjwills Oct 17 '17 at 12:03
  • 1
    The problem with allocating huge strings is that they require a continuous slice of memory. So even if you got 2 Gb of free memory, you might run into OutOfMemoryException if the free memory is fragmented. – Georg Patscheider Oct 17 '17 at 12:14
  • Georg, your comment is most helpful. We read the files line-by-line, so you're saying at some point one line may be, say, 1MB in length and there isn't a 1MB contiguous spot to place it? That sounds tough to fix by adding RAM or messing with virtual memory settings. Even reading the file field-by-field is difficult because after all it's ONE field (the base64 image) which is huge. – royappa Oct 17 '17 at 12:18
  • 1
    You need to show the code causing the exception, in your question. – mason Oct 17 '17 at 12:19
  • It's just "string[] data = info.Split('|');" – royappa Oct 17 '17 at 12:22

1 Answers1

1

Does the application project have "Prefer 32-bit" set?

What is the purpose of the "Prefer 32-bit" setting in Visual Studio 2012 and how does it actually work?

If it does, disable it, and you should now be able to use most of the RAM available, assuming there's no problems elsewhere.

  • Yes, that was set. Will try without and see what happens. Thanks! – royappa Oct 17 '17 at 12:23
  • Our partner figured out that sometimes images were up to 4MB large and simultaneously reduced the image size to 10K..! So it's not clear if this would have worked, however, we most definitely want to use 64 bit for everything so this is the most useful answer. Thank you - and everyone! – royappa Oct 18 '17 at 08:17
  • @royappa you might want to check out some code like this https://stackoverflow.com/a/6655492/310988 where you chop up the stream in to multiple strings as it's read. This would save you having to read in the full string, and then create smaller strings from that large string. –  Oct 18 '17 at 08:19