1

Is it possible in .Net to create "custom" heap, allocate objects on this heap and dispose it deterministically without running GC?

Why?

I have a lot of large text files loaded into memory as strings and they should be in memory until end of processing. After finishing processing I would like to dispose all these strings but they are not disposable in .Net. Since there are lot of big objects created during loading the GC works hard and most of objects finish sitting on Gen2 (and LOH) that can take too much time until GC will perform automatic collection. I can call GC.Collect() to free memory but pressure on GC during loading is still a problem.

I would like to tell the runtime to allocate all these strings in no-GC-aware memory block and dispose this block deterministically after processing finish.

Artur
  • 4,595
  • 25
  • 38
  • There is NO *no-GC-aware memory block* – Rahul Feb 06 '19 at 08:03
  • maybe the fixed keyword can help you ? https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/fixed-statement – MikNiller Feb 06 '19 at 08:07
  • You may want to see https://stackoverflow.com/questions/2161895/reading-large-text-files-with-streams-in-c-sharp – Rahul Feb 06 '19 at 08:08
  • 2
    What's the problem you are running into? Is the processing time too slow? How slow is too slow? How have you determined the GC is the bottleneck? Are you processing one file at a time? Do you really need to load the entire file into one single string to process it? – MineR Feb 06 '19 at 08:18
  • 1
    There are many ways to create unmanaged memory, i think the jury is still out on why you need to do it, and why advantages it will give you – TheGeneral Feb 06 '19 at 08:19
  • @MikNiller The fixed keyword will change the consumer code to operate with char pointers instead of strings that is not possible. – Artur Feb 06 '19 at 08:32
  • @MineR I helped my friend to investigate some performance and memory leak problem. No, I didn't ran profiler to prove my point, it was mostly static code analysis. Unfortunately I can't perform checks and measurements as it's out of my scope right now, but the question is still interesting me and I would like to find an answer with minimum code changes outside of files loading code. – Artur Feb 06 '19 at 08:40
  • @Rahul you are right, the perfect solution will work with streams and enumerate file instead of loading all at once. But it will require a lot of code changes that is not possible in this stage. – Artur Feb 06 '19 at 08:42
  • 2
    If you don't want to change code there is nothing you can do – TheGeneral Feb 06 '19 at 08:51
  • I think you should look for the way strings are handled in .NET. In short, every string is kept in memory and reused as needed so IMHO you should try to avoid keeping big strings in memory and instead use a binary format or a stream to read the text files when needed. – Isma Feb 06 '19 at 09:05
  • @MichaelRandall it's not what I want or don't want. It's what I can or can't do. There are always restrictions when we work on some problem solution that we have to respect (human resources, deadlines, third party, "it works - don't touch")... – Artur Feb 06 '19 at 09:09
  • https://codereview.stackexchange.com/questions/238085/generic-heap-implementation-in-c this link help you to write own heap in C# – divyang4481 Apr 25 '20 at 16:48

0 Answers0