5

I'm developing a logger/sniffer using Delphi. During operation I get hugh amounts of data, that can accumulate during stress operations to around 3 GB of data. On certain computers when we get to those levels the application stops functioning and sometimes throws exceptions.

Currently I'm using GetMem function to allocate the pointer to each message.

Is there a better way to allocate the memory so I could minimize the chances for failure? Keep in mind that I can't limit the size to a hard limit.

What do you think about using HeapAlloc, VirtualAlloc or maybe even mapped files? Which would be better?

Thank you.

Ran
  • 391
  • 1
  • 4
  • 10
  • 1
    Can you use an existing high-volume logger, like [Flume](https://cwiki.apache.org/FLUME/)? – Barend Sep 12 '11 at 18:07
  • 1
    allocate less memory or switch to 64 bit delphi – David Heffernan Sep 12 '11 at 18:10
  • @Barend it's a realtime in-house logger that gets information from all our processes + USB data (my company works with cell phones and during communication all the usb packets are added to the logger). I don't think a ready made logger will be able to handle that correctly. – Ran Sep 12 '11 at 18:13
  • @David why will moving to 64bit delphi help me? – Ran Sep 12 '11 at 18:23
  • 2
    You are running out of address space rather than memory, I guess. Changing allocators won't help much. You can't do anything about 4gb address space limit on 32 bit windows. – David Heffernan Sep 12 '11 at 18:27

1 Answers1

5

Your fundamental problem is the hard address space limit of 4GB for 32 bit processes. Since you are hitting problems at 3GB I can only presume that you are using /LARGEADDRESSAWARE running on 64 bit Windows or 32 bit Windows with the /3GB boot switch.

I think you have a few options, including but not limited to the following:

  1. Use less memory. Perhaps you can process in smaller chunks or push some of the memory to disk.
  2. Use 64 bit Delphi (just released) or FreePascal. This relieves you of the address space constraint but constrains you to 64 bit versions of Windows.
  3. Use memory mapped files. On a machine with a lot of memory this is a way of getting access to the OS memory cache. Memory mapped files are not for the faint hearted.

I can't advise definitively on a solution since I don't know your architecture but in my experience, reducing your memory footprint is often the best solution.

Using a different allocator is likely to make little difference. Yes it is true that there are low-fragmentation allocators but they surely won't really solve your problem. All they could do would be make it slightly less likely to arise.

David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490
  • 3
    Memory mapped files may be one way to move forward, but you may end up allocating the max possible. And you'd have complex mappings for >3Gb. It strikes me that the best option would be to tokenise the log messages to reduce memory requirement. So you might have "error" and "starting" and "opening" reduced to a single byte - it may take some static analysis to work out what is common, but could halve the requirement quickly. – mj2008 Sep 12 '11 at 18:48