Limiting memory usage for a single process in OSX /Darwin

Question

I am trying to modify some JNI code to limit the amount of memory that a process can consume. Here is the code that I am using to test setRlimit on linux and osx. In linux it works as expected and the buf is null.

This code sets the limit to 32 MB and then tries to malloc a 64 MB buffer, if buffer is null then setrlimit works.

    #include <sys/time.h>
    #include <sys/resource.h>                                                     
    #include <stdio.h>                                                  
    #include <time.h>                                                   
    #include <stdlib.h>                                                  
    #include <unistd.h>                                                   
    #include <sys/resource.h>

    #define errExit(msg)     do { perror(msg); exit(EXIT_FAILURE); \
                                       } while (0)

       int main(int argc) {


        pid_t pid = getpid();

        struct rlimit current;
        struct rlimit *newp;

        int memLimit = 32 * 1024 * 1024;
        int result = getrlimit(RLIMIT_AS, &current);
        if (result != 0)
                errExit("Unable to get rlimit");

        current.rlim_cur = memLimit;
        current.rlim_max = memLimit;

        result = setrlimit(RLIMIT_AS, &current);
        if (result != 0)
                errExit("Unable to setrlimit");


        printf("Doing malloc \n");
        int memSize = 64 * 1024 * 1024;

        char *buf = malloc(memSize);

        if (buf == NULL) {
                printf("Your out of memory\n");
        } else {
                printf("Malloc successsful\n");
        }

        free(buf);
}

On linux machine this is my result

memtest]$ ./m200k                             
Doing malloc 
Your out of memory

On osx 10.8

./m200k 
Doing malloc 
Malloc successsful

My question is that if this does not work on osx is there a way to acomplish this task in darwin kernel. The man pages all seem to say it will work but it does not appear to do so. I have seen that launchctl has some support for limiting memory but my goal is to add this ability in code. I tried using ulimit also but this did not work either and am pretty sure ulimit uses setrlimit to set limits. Also is there a signal I can catch when setrlimit soft or hardlimit is exceeded. I haven't been able to find one.

Bonus points if it can be accomplished in windows also.

Thanks for any advice

Update
As pointed out the RLIMIT_AS is explicitly defined in the man page but is defined as the RLIMIT_RSS, so if referring to the documentation RLIMIT_RSS and RLIMIT_AS are interchangable on OSX.

/usr/include/sys/resource.h on osx 10.8

#define RLIMIT_RSS      RLIMIT_AS       /* source compatibility alias */

Tested trojanfoe's excellent suggestion to use RLIMIT_DATA which is described here

    The RLIMIT_DATA limit specifies the maximum amount of bytes the process
 data segment can occupy. The data segment for a process is the area in which 
 dynamic memory is located (that is, memory allocated by malloc() in C, or in C++,
 with new()). If this limit is exceeded, calls to allocate new memory will fail. 

The result was the same for linux and osx and that was the malloc was successful for both.

chinshaw@osx$ ./m200k 
Doing malloc 
Malloc successsful


chinshaw@redhat ./m200k
Doing malloc 
Malloc successsful