Memory allocation threshold (mmap vs malloc)

Question

Id like to point out I'm new to this so I'm trying to understand / explain it best i can.

I am basically trying to figure out if its possible to keep memory allocation under a threshold due to memory limitation of my project.

Here is how memory is allocated currently using third party libsodium:

alloc_region(escrypt_region_t *region, size_t size)
{
    uint8_t *base, *aligned;
#if defined(MAP_ANON) && defined(HAVE_MMAP)
    if ((base = (uint8_t *) mmap(NULL, size, PROT_READ | PROT_WRITE,
#ifdef MAP_NOCORE
                                 MAP_ANON | MAP_PRIVATE | MAP_NOCORE,
#else
                                 MAP_ANON | MAP_PRIVATE,
#endif
                                 -1, 0)) == MAP_FAILED)
        base = NULL; /* LCOV_EXCL_LINE */
    aligned  = base;
#elif defined(HAVE_POSIX_MEMALIGN)
    if ((errno = posix_memalign((void **) &base, 64, size)) != 0) {
        base = NULL;
    }
    aligned = base;
#else
    base = aligned = NULL;
    if (size + 63 < size)
        errno = ENOMEM;
    else if ((base = (uint8_t *) malloc(size + 63)) != NULL) {
        aligned = base + 63;
        aligned -= (uintptr_t) aligned & 63;
    }
#endif
    region->base    = base;
    region->aligned = aligned;
    region->size    = base ? size : 0;

    return aligned;
}

So for example, this currently calls posix_memalign to allocate (e.g.) 32mb of memory. 32mb exceeds my 'memory cap' given to me (but does not throw memory warnings as the memory capacity is far greater, its just what I'm 'allowed' to use)

From some googling, I'm under the impression i can either use mmap and virtual memory. I can see that the function above already has some mmap implemented but is never called.

Is it possible to convert the above code so that i never exceed my 30mb memory limit?

From my understanding, if this allocation would exceed my free memory, it would automatically allocate in virtual memory? So can i force this to happen and pretend that my free space is lower than available?

Any help is appreciated

UPDATE

/* Allocate memory. */
    B_size = (size_t) 128 * r * p;
    V_size = (size_t) 128 * r * N;
    need   = B_size + V_size;
    if (need < V_size) {
        errno = ENOMEM;
        return -1;
    }
    XY_size = (size_t) 256 * r + 64;
    need += XY_size;
    if (need < XY_size) {
        errno = ENOMEM;
        return -1;
    }
    if (local->size < need) {
        if (free_region(local)) {
            return -1;
        }
        if (!alloc_region(local, need)) {
            return -1;
        }
    }
    B  = (uint8_t *) local->aligned;
    V  = (uint32_t *) ((uint8_t *) B + B_size);
    XY = (uint32_t *) ((uint8_t *) V + V_size);

I afraid no easy answer.... eg for file reading or writing usually mmap. — 0___________, Jul 11 '17 at 14:00
The above code won't try to allocate more than 30Mb _unless you ask it to_. Is there a reason you can't just avoid requesting too much memory in the first place? — Useless, Jul 11 '17 at 14:06
I'm calling an encryption method which causes a termination due to memory issue, as I'm crossing my given 30mb threshold. So i traced the allocation to this function. Are you saying that if i just tell it to allocate 20mb it can still perform the operation with just 20mb allocated? I have updated my original post to show the call to alloc_region — nicwhitts, Jul 11 '17 at 15:52
why not write your own malloc that keeps track of how much is allocated — pm100, Jul 11 '17 at 15:58
@Useless i think your right now reviewing code, I'm having errors when trying to allocate less memory now though. I think a new question would be needed. — nicwhitts, Jul 11 '17 at 19:29

Basile Starynkevitch · Answer 1 · 2017-07-11T16:12:51.313

I am basically trying to figure out if its possible to keep memory allocation under a threshold due to memory limitation of my project.

On Linux or POSIX systems, you might consider using setrlimit(2) with RLIMIT_AS:

         This is the maximum size of the process's virtual memory
          (address space) in bytes.  This limit affects calls to brk(2),
          mmap(2), and mremap(2), which fail with the error ENOMEM upon
          exceeding this limit.

Above this limit, the mmap would fail, and so would fail for instance the call to malloc(3) that triggered that particular use of mmap.

I'm under the impression i can either use mmap

Notice that malloc(3) will call mmap(2) (or sometimes sbrk(2)...) to retrieve (virtual) memory from the kernel, thus growing your virtual address space. However, malloc often prefer to reused previously free-d memory (when available). And free usually won't call munmap(2) to release memory chunks but prefer to keep it for future malloc-s. Actually most C standard libraries segregate between "small" and "large" allocation (in practice a malloc for a gigabyte will use mmap, and the corresponding free would mmap immediately).

See also mallopt(3) and madvise(2). In case you need to lock some pages (obtained by mmap) into physical RAM, consider mlock(2).

Look also into this answer (explaining that the notion of RAM used by a particular process is not that easy).

For malloc related bugs (including memory leaks) use valgrind.

Memory allocation threshold (mmap vs malloc)

1 Answers1

Linked