0

In my .Net (C#) application I have a lot of objects which needs to be accessible very quickly. Because of some external limitations I don't have control over, I'm limited to one 32bit process. So I'm quickly hitting the memory limit of 4GB.

Is there any data structure I can use which can store these objects, and write them (temporary) to disk when memory usage becomes too large? Ideally, it would keep the objects which are accessed most frequently in memory and flush less used objects to disk.

I looked at several embedded databases, but they all seem to be storing either everything in memory or everything on disk.

Ideas?

Coder14
  • 1,305
  • 1
  • 9
  • 26
  • Memory on PC is stored on hard-drive when when memory limits are exceeded which is called swap space. The disk is slow compared to memory so swapping is not a choice if you need to get the data quickly. changing Process has nothing really to do with issue. Putting swap space on a solid state disk will speed up swapping if you are currently using a hard-drive. – jdweng Feb 08 '18 at 17:06
  • If you are having high memory usage without getting an OutOfMemoryException, it may be due to a slow resource leak. I had a similar issue that turned out to be from missing `using` blocks on database calls. The memory usage dropped from about 600MB avg to about 70MB avg after fixing this. – NightOwl888 Feb 08 '18 at 17:16
  • My guess is that you have misdiagnosed your problem – David Heffernan Feb 08 '18 at 17:33
  • 1
    I do get an OutOfMemory exception if I load too many objects. The software is running on the customers pc so I have no influence on the used hardware. I know what is using memory and I can't change that. There's no leak. – Coder14 Feb 08 '18 at 19:07
  • A 32-bit process has a 2GB limit for user memory. You could access up to 4GB if you are on a 64-bit OS or 3GB on a 32-bit OS - maybe that would be enough? (See [answer](https://stackoverflow.com/a/464473/2557128) here.) – NetMage Feb 08 '18 at 23:28
  • How would you decide what to unload to disk? Do you have temporal or relational locality? I think you would need to write your own caching structure. – NetMage Feb 08 '18 at 23:31
  • 1
    Have you looked at [Sqlite](https://www.sqlite.org/)? Yes, the data is on disk, but it use caching, so it's likely that frequently-accessed records will remain in memory. – Jim Mischel Feb 09 '18 at 05:23

0 Answers0