13

In C/C++ under Linux, I need to allocate a large (several gigabyte) block of memory, in order to store real-time data from a sensor connected to the ethernet port and streaming data at about 110MB/s. I'd like to allocate the largest amount of memory possible, to maximise the length of data sequence that I can store. However, I also need to make sure that there will be no disk-swapping, since the resulting delay and limited bandwidth of disk access causes the sensor's (very limited) buffer to overflow.

What is the best way to determine how much memory to allocate? Am I limited to just allocating a slightly smaller block than the reported free memory, or can I interface more directly with the linux virtual memory manager?

Chris Johnson
  • 10,469
  • 4
  • 31
  • 35
  • Purely out of interest, what sensor is this you are using? – Konrad Jul 14 '10 at 10:58
  • Duplicate of http://stackoverflow.com/questions/2513505/ – Björn Pollex Jul 14 '10 at 11:54
  • @Space_C0wb0y: it isn't really. – Hasturkun Jul 14 '10 at 12:00
  • Get a better sensor. Or switch to a proper real time OS. virtual memory OS's are not going to let you do what you want, unless you do something weird like using a ram disk as swap space. I have seen 4GB + of RAM on the other side of a IDE interface. Proper powered RAM. Very fast. Very useful for subverting a Virtual Memory based OS and making it all cached in RAM. – Chris Becke Jul 14 '10 at 12:07
  • 1
    @Chris Becke: Linux doesn't force you to do anything wierd to make buffers unswappable; there's a straightforward system call to lock them into physical memory, or you can configure the system with no swap disk at all if you want. And I've found its realtime behaviour to be quite satisfactory for realtime constraints on the order of a few milliseconds or longer. – Mike Seymour Jul 14 '10 at 12:14
  • @Konrad: it's a Dalsa Genie HM1400 camera @Space_C0wb0y: thanks for the link to the related question @Chris Becke: The traditional way of solving this problem is a hardware unit with exactly what you suggest - an embedded realtime OS and lots of RAM. Out of my budget though... – Chris Johnson Jul 14 '10 at 14:22

4 Answers4

10

Well, under linux you can use mlock()/mlockall() to keep an adress range in physical memory and prevent it from being swapped out. The process using mlock needs a couple of privileges to do so, "man mlock" has the details. I am not sure about the maximum mlock'able block (it might differ from what seems to be "free"), so probably a binary search could help (lock a range, if that fails reduce the size of the area etc..)

On the other hand, 110MB/s is not really a problem for a Solid-State-Drive. A 60GB SSD with 280MB/s write speed costs about $200 on the corner. Just copy the sensor data into a small write buffer and stream that to the SSD.

Nordic Mainframe
  • 28,058
  • 10
  • 66
  • 83
3

If the computer system is dedicated to receiving data from your sensor, you can simply disable swap. Then allocate as big buffer as you can, leaving enough memory in the system only for essential tools.

Tomek Szpakowicz
  • 14,063
  • 3
  • 33
  • 55
  • Sadly it's not a dedicated system. However, it has plenty of RAM for it's other uses, so disabling swap might be an option. – Chris Johnson Jul 14 '10 at 14:24
  • Check memory usage for some time. You might see that swap is hardly ever used unless some application starts leaking memory like crazy. I've been running swap-less desktop systems without any problems. But for more general solution use mlock(2) as Luther advised. And don't allocate everything as one continuous buffer. Memory manager might have problem allocating (or reallocating) it. Use many smaller buffers linked into one by some interface. This way you can e.g. tune buffer size at runtime. – Tomek Szpakowicz Jul 14 '10 at 14:42
  • Thumbs up for "buffor" spelling :P! – Cray May 22 '11 at 01:29
0

If you malloc the needed amount of memory and write to it at that speed, you'll still get a performance hit due to all the page faults (i.e. mapping each page of virtual memory to physical memory, which also may include swapping out memory of other processes).

In order to avoid that, you could memset the entire allocated buffer to 0 before you start reading from the sensor, so that all the needed virtual memory is mapped to physical memory.

If you only use the available physical memory, you should suffer no swapping at all. Using more would cause memory of other processes to be swapped to the disk - if these processes are idle, it shouldn't pose any problem. If they're active (i.e. using their memory once in a while), some swapping would occur - probably in a much lower rate than the hard-drive bandwidth. The more memory you use, more active processes' memory would be swapped out, and more HD activity would occur - at this point the maximal amount of memory you could use with decent performance is pretty much a result of trial and error.

By using more than the physical memory available, you'll definitely cause swapping at the rate of memory writes, and there's no way to avoid that.

adamk
  • 45,184
  • 7
  • 50
  • 57
  • 2
    There's no need to use `memset()` to trick the system into allocating physical memory. Use `mlock()` to do it explicitly, and make sure that it's never swapped out. – Mike Seymour Jul 14 '10 at 11:37
0

What is the best way to determine how much memory to allocate?

Due to how virtual memory is used, non-swappable kernel memory, it is nearly impossible to identify how much of installed memory can be accessed by an application.

Best I can come up with is to allow user to configure how much memory to use for buffering.

Am I limited to just allocating a slightly smaller block than the reported free memory,

Reported free memory is not really "free physical memory." Unfortunately.

or can I interface more directly with the linux virtual memory manager?

That can be done by using a custom device driver, allocating memory directly in kernel space and providing access to it via mmap(). Generally not recommended, yet would works in specialized cases such as yours.

However, I also need to make sure that there will be no disk-swapping

At pace of the Linux kernel development, knowledge becomes obsolete quite fast, so take with grain of salt what I'm saying here. You can try to play with the following:

  1. SysV shared memory. It is generally not swapped. See man shmget.

  2. tmpfs - in-memory file system. The memory was pinned to RAM at least in early 2.6 kernels and thus was not swappable. To use it as memory, create a file on tmpfs, write() something into the file (to force the memory being actually allocated) and then mmap() the file.

Dummy00001
  • 16,630
  • 5
  • 41
  • 63
  • Just for the records, *tmpfs* is in fact swappable. There is another variant *rootfs* that will never be swapped. But just because of this you have to be really careful with it and should not overstress it. – Jens Gustedt Jul 14 '10 at 11:46