0

I want to allocate virtually contiguous range memory so that I can use the locality property (spatial locality) when accessing data, with consideration for better performance. I found out at the following page that I need to use vmalloc for better memory locality access (please correct me if I am wrong and have to use kmalloc instead).

What is the difference between vmalloc and kmalloc?

I downloaded the vmalloc package from http://www2.research.att.com/~gsf/cgi-bin/download.cgi?action=list&name=vmalloc

I followed the installation procedure for building the libmalloc.a static library from the source files, and then I copied the generated libvmalloc.a library to /usr/local/lib and /usr/lib directory on my mac.

In my C program, I tried to include the vmalloc.h header file through various approaches as follow: 

#include <vmalloc.h>

.

#include <linux/vmalloc.h>

.

#include "vmalloc.h"

but none of them worked. I always got the vmalloc.h: No such file or directory error message. I used -L/usr/local/lib -lvmalloc flags when compiling my C++ program. I also got the same error when trying the same thing on my desktop computer (under CentOS operating system).
Here is my makefile: 

SHELL = /bin/sh
PROJECT = hw2

TARGET = mkhashtbl
CFLAGS = -O3
LFLAGS = -O3 -L/usr/local/lib -lvmalloc

TFILE = $(PROJECT).tgz
ZFILE = $(PROJECT).zip
PACKFILES = Makefile hashtbl.h hashtbl.c mkhashtbl.c timer.c

all:    $(TARGET)

run:    all
    - ./$(TARGET)

I also tried to modify my Linker flags as follows: 

LFLAGS = -O3 -L/usr/local/lib -lvmalloc

and I still got the same error. What could be wrong in this case? Is there anything wrong in the way I linked the library, or does vmalloc only work for some versions of Linux? If it is the latter case, I am sure that I should still be able to at least include the header file.

EDIT

My real problem is actually the following: 

hashtbl_cache *    hashTable_cache;             /* Hash table cache */

    int i;
    hashTable_cache = (hashtbl_cache*)malloc(tbl_size* sizeof(hashtbl_cache));

    for( i = 0 ; i < tbl_size; i++ )
    {
          hashTable_cache[i]. ht_elements = (hashelt_cache*)malloc( hashTable->data_total_size[i] * sizeof(hashelt_cache) ) ;
    }

I want to make sure that all the ht_elements in every cache are created in order. I read from a forum that vmalloc is good for creating cache aware application, as data are created in virtual memory. Is there any other approach to make sure that the allocation of all elements of my cache array are created in contiguous order, hence will enable me to do a fast lookup? One more thing, the size of every elements in every cache is not the same, so I guess using calloc is not a solution, but I may be wrong.

Community
  • 1
  • 1
all_by_grace
  • 2,315
  • 6
  • 37
  • 52
  • 2
    The `vmalloc` in that question is a Linux kernel function. Unless you are hacking the kernel or writing a device driver, it is not relevant to you. The `vmalloc` at the AT&T Research site is something else entirely. Neither one does what you think it does. Just use `malloc`. – Nemo Jul 19 '11 at 00:18
  • Yeah,I tried using malloc. And now I want to create a cache-aware hashtable. If I use malloc, each element of my hashtable will be located on different memory address right? Thus, I won't be able to use the locality property when accessing the data in my hashtable. My assumption is that if I use vmalloc, all the data entries of the hashtable would be created in 1 continuous large block, so accessing the data would be faster. Another question: if it is a linux kernel function, how come I cannot include it on my CentOS OS computer? – all_by_grace Jul 19 '11 at 00:29
  • Where is the file `vmalloc.h` in your file system? When you know that, you will know what value to add to your `-I` options. Note that the `-L` option is relevant to finding libraries when linking, not to finding headers when compiling. Also, where is the library (`libvmalloc.so` or `libvmalloc.a`). If you can't find the header except in the build area of libvmalloc, you will need to reinstall it from the build, or redo the build and ensure that the header is installed (or manually install it). – Jonathan Leffler Jul 19 '11 at 01:35
  • I have the libvmalloc.a inside the /usr/local/lib directory. The vmalloc.h is included in the build of libvmalloc.a. Thanks for clarifying the different between -I and -L. – all_by_grace Jul 19 '11 at 01:45

2 Answers2

0

When compiling, specify -Ipath for gcc to know where 'vmalloc.h' is.

Or, specify the environment variable C_INCLUDE_PATH (for C) or CPLUS_INCLUDE_PATH (for C++) before calling gcc.

GCC will search header files follow this way:

  1. -Ipath
  2. Environment variables: C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, OBJC_INCLUDE_PATH

  3. Default paths (if you didn't specify prefix when installing gcc), probably like:

    /usr/include

    /usr/local/include

    /usr/lib/gcc-lib/i386-linux/2.95.2/include

    /usr/lib/gcc-lib/i386-linux/2.95.2/../../../../include/g++-3

    /usr/lib/gcc-lib/i386-linux/2.95.2/../../../../i386-linux/include

Stan
  • 787
  • 5
  • 16
  • I have done that in my .bashrc and .bash_profile. So the include path is automatically fetched. Even I also used -I and -L flags to specify the path manually. So this is not the problem. – all_by_grace Jul 19 '11 at 01:19
  • It seems gcc stop at the preprocess stage and it still cannot find the right include path. Are you sure what you specify to -I option is the right path? PS: The -L -l option is for linking but not compiling, the problem occurred before they take effect. – Stan Jul 19 '11 at 01:25
0

Nemo's comment should have been given as an answer:

The vmalloc in that question is a Linux kernel function. Unless you are hacking the kernel or writing a device driver, it is not relevant to you. The vmalloc at the AT&T Research site is something else entirely. Neither one does what you think it does. Just use malloc.

It's obvious that you don't know what "virtually contiguous range" means or you'd realize that malloc must give you that; otherwise it would not work at all.

Premature optimization is the root of all evil. Especially when you don't have any idea what the optimization you're trying to make means.

R.. GitHub STOP HELPING ICE
  • 208,859
  • 35
  • 376
  • 711
  • Thanks for clarifying. I edited my question and provide a clarification of what I want to achieve. What do you think is the best way to achieve that? Thanks – all_by_grace Jul 19 '11 at 02:17
  • The way you allocate objects in close (virtual) address proximity is to perform one large allocation with `malloc` and then carve it up with pointer arithmetic or by using it as an array of the type you need. Note that you cannot then `free` the individual pieces, only the whole chunk as originally returned by `malloc`. – R.. GitHub STOP HELPING ICE Jul 19 '11 at 02:53