How to reassign cuda gpu device arrays in a for loop without running out of memory?

Question

Look at this code:

 (cudavenv) C:\main\FemtoTest\Library\Python\libImageProcess\trunk\src\libImageProcess>python
Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from numba import cuda
>>> for i in range(26):
...     arr = np.zeros((17, 8025472),dtype=np.uint32)
...     d_arr = cuda.to_device(arr)
...

which runs successfully vs.

(cudavenv) C:\main\FemtoTest\Library\Python\libImageProcess\trunk\src\libImageProcess>python
Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from numba import cuda
>>> class M:
...     def __init__(self):
...             self.arr = np.zeros((17, 8025472),dtype=np.uint32)
...             self.d_arr = None
...
>>> ms = [M() for _ in range(26)]
>>> for m in ms:
...     m.d_arr = cuda.to_device(m.arr)
...
Traceback (most recent call last):
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 741, in _attempt_allocation
    allocator()
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 756, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\devices.py", line 225, in _require_cuda_context
    return fn(*args, **kws)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\api.py", line 110, in to_device
    to, new = devicearray.auto_device(obj, stream=stream, copy=copy)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\devicearray.py", line 693, in auto_device
    devobj = from_array_like(obj, stream=stream)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\devicearray.py", line 631, in from_array_like
    writeback=ary, stream=stream, gpu_data=gpu_data)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\devicearray.py", line 102, in __init__
    gpu_data = devices.get_context().memalloc(self.alloc_size)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 758, in memalloc
    self._attempt_allocation(allocator)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 748, in _attempt_allocation
    allocator()
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 756, in allocator
    driver.cuMemAlloc(byref(ptr), bytesize)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 294, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "C:\Users\alex\AppData\Local\Continuum\anaconda3\envs\cudavenv\lib\site-packages\numba\cuda\cudadrv\driver.py", line 329, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY

I think in the first instance I am reassigning d_arr to the device array each time so it only takes up that much memory. In the second case because there are 26 instances, it creates a new array on the device each time and eventually runs out of memory. What method do I need to call to delete the memory reference when I am done using it in the for loop? So that this can run without issue?

score 2 · Answer 1 · answered Sep 17 '19 at 00:15

You may wish to read section 3.3.8 here.

Deallocation of no-longer-needed CUDA memory is possible when the last reference to it is dropped. In your first case, this happens on each pass through the loop, when d_arr is reassigned. In the second case, it does not, because the references are held in ms.

I believe a proper solution is to cause the reference to be dropped. The pythonic way to do this is to remove the reference:

import numpy as np
from numba import cuda
class M:
    def __init__(self):
            self.arr = np.zeros((17, 8025472),dtype=np.uint32)
            self.d_arr = None

ms = [M() for _ in range(26)]
for m in ms:
    m.d_arr = cuda.to_device(m.arr)
    # do whatever it is you want to do with m.d_arr here
    m.d_arr = None

How to reassign cuda gpu device arrays in a for loop without running out of memory?

1 Answers1