1

I have a .netcore app running in cloud foundry (4GB RAM). Following piece of code is throwing out of memory exception. I can't really find the record causing the issue because following code is being called inside a job.

I did find article over internet about "MemoryStream" that it keeps on un-allocate and re-allocate memory until object size matches with stream size But couldn't find solution to fix my problem.

IFormatter frm = new BinaryFormatter();
Stream sm = new MemoryStream();
using (sm)
{
    frm.Serialize(sm, myobj);
    sm.Seek(0, SeekOrigin.Begin);
    return (T)frm.Deserialize(sm);
}

The code is called in a job which loops through a list of objects and tries to clone each object. Some of the objects can be really big but definitely not even close to 4 GB which is size of memory.

Environment.Is64BitOperatingSystem is true so I am definitely running in 64-bit mode.

More updates: I did some more investigation and found that there is one object. if I copy json from mongo to notepad it's size is 1 MB. Code is creating more than 7K clone of this object. what is the best way to handle it.

Resolved: Looked into logic and found a way to avoid cloning project. I guess that was the only way to solve my problem.

Anil
  • 1,669
  • 5
  • 19
  • 44
  • What is `myObj`? Maybe it's indeed too large when serialized. Can you also share some code around this? Also, for what is the snippet doing a serialization round-trip? – Ray Aug 26 '22 at 23:48
  • Are you running as a 32 or 64 bit application? You can check `Environment.Is64BitOperatingSystem` to find this out. – dbc Aug 27 '22 at 00:12
  • If you are running in 32 bit, switch to 64. You might also try a [MemoryStream replacement](https://stackoverflow.com/q/12713886/3744182). – dbc Aug 27 '22 at 00:19
  • 1
    You will also want to read this (it strongly suggests you don't use BinaryFormatter): https://learn.microsoft.com/en-us/dotnet/api/system.runtime.serialization.formatters.binary.binaryformatter?view=net-6.0#remarks – Flydog57 Aug 27 '22 at 02:11
  • 1
    Presumably this code is trying to clone an object? If so you should find better methods to do this. – Charlieface Aug 27 '22 at 22:54
  • @Charlieface yes this code is trying to close the object. – Anil Aug 29 '22 at 04:20
  • @dbc Code is build using command "RUN dotnet publish -c Release" in docker file. There is no other parameter to define 32 or 64 bit. – Anil Aug 29 '22 at 04:23
  • @Ray code is called in a job which loop through a list of objects and trying to clone each object. yes some of the objects can be really big but definitely not even close to 4 GB which is size of memory. – Anil Aug 29 '22 at 04:27
  • @Anil - Can you do `Console.WriteLine($"Environment.Is64BitOperatingSystem={Environment.Is64BitOperatingSystem}");` on startup to see if you are running as a 32 or 64 bit app? – dbc Aug 29 '22 at 04:55
  • 1
    @dbc Environment.Is64BitOperatingSystem it returns true – Anil Aug 29 '22 at 17:03
  • Then I think we need to see a [mcve] to help you. Possibly one of the objects you are cloning is disposable (e.g. a bitmap) but you aren't disposing of it properly. – dbc Aug 29 '22 at 17:11
  • 1
    Your object may not be all that large, but you may be exhausting memory due to a large number of objects in memory, or heap fragmentation. Is this code in a for loop in a single method? Consider breaking the method down into smaller methods that properly "dispose" of the objects in a more timely manner so that memory pressure is mitigated. – Mike Hofer Aug 29 '22 at 18:32
  • @MikeHofer I did some more investigation and found that there is one object. if I copy json from mongo to notepad it's size is 1 MB. Code is creating more than 7K clone of this object. what is the best way to handle it. – Anil Sep 02 '22 at 17:49
  • I would begin by asking why you need to create 7,000 clones of an object of that size. If you can eliminate that need, do so. If you can't, find a way to make one clone, serialize it to disk and remove it from memory, then move on to the next one. In an absolute worst case scenario, periodically call GC.Collect() to free unused memory, though I stress that this should be your LAST RESORT. – Mike Hofer Sep 02 '22 at 18:24
  • You might also consider breaking the large object up into smaller pieces, so you only have to clone the smallest piece. For example, if the object contains image data, or large blobs of text, move those into separate locations in your MongoDB and have your object reference them. Don't load large objects during deserialization; load them on-demand. – Mike Hofer Sep 02 '22 at 18:26
  • 1
    @MikeHofer just edited my question. After looking more closely into the logic found a way to avoid cloning whole original object instead created a new object with required property. – Anil Sep 02 '22 at 18:26

0 Answers0