-2

I have a program which would use cudaMallocHost() to allocate the pinned memory, but I forget to use cudaFreeHost() to free the pinned memory...

I run this program once and then exit, but next time I want to run the same program, it would throw segmentation fault when I called cudaMallocHost.

I suspected that is due to the memory is pinned when I first run the program, and when I try to run the program one more time, the OS couldn't find any more memory that could be pinned...

My question is, is there any CUDA API that I can call to clear the already pinned memory in host without knowing the host memory address? I lookup the CUDA document but didn't find any, and rebooting didn't help either.

Edit

I run htop and found that there is a 17GB of memory which seems like nobody is using it. I wonder if this is the memory that I pinned?

htop

leonhhh
  • 3
  • 2
  • your pinned memory may not have been allocated by cuda, or allocated by not contiguous pieces, or is reserved for another hardware (some acquisition systems have their own page locked memory). I'm afraid there is no such utility since the operating system don't manage this memory anymore. The only way i know to get this memory back is to reboot the computer. – X3liF Jan 03 '23 at 13:47
  • @X3liF I tried to reboot but when I use `htop`, there is still a 17GB of memory allocated, so I think reboot is not helping. – leonhhh Jan 03 '23 at 14:17
  • it should not happen, pinned memory allocated with cudaMallocHost is not persistent when you restart your computer it must be back, may you share your htop screenshot after having restarted your computer (Especillay sorted by memory usage -> push F6 then select percent_mem) you must have an application using this memory – X3liF Jan 03 '23 at 14:30
  • @X3liF The result of htop result which sorted by memory usage is here (https://i.stack.imgur.com/C2L60.png). I still don't see any process using that much memory, did I miss something or the htop does not work as I think? – leonhhh Jan 03 '23 at 14:50
  • this image is the same as you provide in your question. can you double check the uptime of your machine (should be near to 0 after the reboot) or try the small code i provided to check your htop behaviour. i'm afraid this is not pinned memory you are observing. – X3liF Jan 03 '23 at 14:57
  • 2
    The memory allocated by `cudaMallocHost` is freed by CUDA automatically, when your application exits, whether you call `cudaFreeHost` or not. No, there is no API in CUDA to free memory that your application did not allocate. – Robert Crovella Jan 03 '23 at 14:58
  • Thank you @Robert Crovella for these details – X3liF Jan 03 '23 at 14:59
  • Some Linux distributions put e.g. `/tmp` into RAM using ramfs for better performance. The memory is then shown shown as occupied, but not attributed to a process/thread, I think. I struggeled with this on Fedora before when programs needed a lot of temporary storage. Use `free -h` in the shell to see how much is `used` by processes vs used as `buff/cache`. That being said questions about analyszing RAM usage in linux are offtopic here and should be posted in the [Unix StackExchange](https://unix.stackexchange.com/). – paleonix Jan 03 '23 at 16:06
  • If you do [proper CUDA error checking](https://stackoverflow.com/a/14038590/10107454), you should get a meaningful error message instead of a segfault when running out of memory. – paleonix Jan 03 '23 at 16:23

1 Answers1

1

i made some tests using htop and a small application :

here is the code i used :

#include <cuda_runtime.h>
#include <unistd.h>
#include <vector>

int main(void)
{
    std::vector<void*> arPtrs;
    for(int i=0;i<5;++i)
    {
            void* ptr = nullptr;
            cudaMallocHost(&ptr, 1 * 1024 * 1024*1024);
            arPtrs.push_back(ptr);
            sleep(2);
    }
    return 0;
}

as you can see i don't call the cudaFreeHost on my pointers.

in parallel i monitor the memory with htop. The memory appears released for htop when the application leaves.

May your memory being use by another user so you can't see it ?

X3liF
  • 1,054
  • 6
  • 10