I am performing an operation which involves different operations(subtraction, squaring,broadcasting) on numpy array of higher dimensions. My code is giving Memory Error on performing such operations.
My code below-
from skimage.segmentation import find_boundaries
w0 = 10
sigma = 5
def make_weight_map(masks):
"""
Generate the weight maps as specified in the UNet paper
for a set of binary masks.
Parameters
----------
masks: array-like
A 3D array of shape (n_masks, image_height, image_width),
where each slice of the matrix along the 0th axis represents one binary mask.
Returns
-------
array-like
A 2D array of shape (image_height, image_width)
"""
masks = masks.numpy()
nrows, ncols = masks.shape[1:]
masks = (masks > 0).astype(int)
distMap = np.zeros((nrows * ncols, masks.shape[0]))
X1, Y1 = np.meshgrid(np.arange(nrows), np.arange(ncols))
X1, Y1 = np.c_[X1.ravel(), Y1.ravel()].T
#In the below for loop, I am getting the Memory Error
for i, mask in enumerate(masks):
# find the boundary of each mask,
# compute the distance of each pixel from this boundary
bounds = find_boundaries(mask, mode='inner')
X2, Y2 = np.nonzero(bounds)
xSum = (X2.reshape(-1, 1) - X1.reshape(1, -1)) ** 2
ySum = (Y2.reshape(-1, 1) - Y1.reshape(1, -1)) ** 2
distMap[:, i] = np.sqrt(xSum + ySum).min(axis=0)
ix = np.arange(distMap.shape[0])
if distMap.shape[1] == 1:
d1 = distMap.ravel()
border_loss_map = w0 * np.exp((-1 * (d1) ** 2) / (2 * (sigma ** 2)))
else:
if distMap.shape[1] == 2:
d1_ix, d2_ix = np.argpartition(distMap, 1, axis=1)[:, :2].T
else:
d1_ix, d2_ix = np.argpartition(distMap, 2, axis=1)[:, :2].T
d1 = distMap[ix, d1_ix]
d2 = distMap[ix, d2_ix]
border_loss_map = w0 * np.exp((-1 * (d1 + d2) ** 2) / (2 * (sigma ** 2)))
xBLoss = np.zeros((nrows, ncols))
xBLoss[X1, Y1] = border_loss_map
# class weight map
loss = np.zeros((nrows, ncols))
w_1 = 1 - masks.sum() / loss.size
w_0 = 1 - w_1
loss[masks.sum(0) == 1] = w_1
loss[masks.sum(0) == 0] = w_0
ZZ = xBLoss + loss
return ZZ
To reproduce a problem, a numpy array of dimensions 4,584, 565 could recreate the issue.
Traceback of the error-
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-32-0f30ef7dc24d> in <module>
----> 1 img = make_weight_map(img)
<ipython-input-31-e75a6281476f> in make_weight_map(masks)
34 xSum = (X2.reshape(-1, 1) - X1.reshape(1, -1)) ** 2
35 ySum = (Y2.reshape(-1, 1) - Y1.reshape(1, -1)) ** 2
---> 36 distMap[:, i] = np.sqrt(xSum + ySum).min(axis=0)
37 ix = np.arange(distMap.shape[0])
38 if distMap.shape[1] == 1:
MemoryError:
When using a input of 4,584, 565, the shape are-
X1.shape
(329960,)
X2.shape,Y2.shape# for first iteration
(15239,) (15239,)
The main issue is arising in this line (X2.reshape(-1, 1) - X1.reshape(1, -1)) because the shape of X2 is (15239,1) and X1 IS (1,329960) so first it had to perform a humongous broadcasting operation.
distmap, I am not able to calculate because my code is halting before that.
Also, If I try to perform the following subtraction operation with the above dimension, code is halting there too.
X2.reshape(-1, 1) - X1.reshape(1, -1)
I am using a system of 32 Gb RAM, I also tried running on cloud of 64 Gb RAM. I have checked the below questions and either they are not providing the solutions to my problem or I am not able to apply to my use case.
Python/Numpy MemoryError
Working with big data in python and numpy, not enough ram, how to save partial results on disc?
Memory growth with broadcast operations in NumPy
