How to replace a masked part of an image with another image using openCV python?

Question

The objective is to replace a part of an image with another image. What I plan to do is to get the segment map of the original photo, and replace the selected part of the original with another image.

For example, This is a photo of the original photo's segment map:

And I want to replace that pink segment mask with another guy model (different person):

How should I do so about it? I am thinking of selecting an ROI based on the pink colour (RGB Values of 192,128,128) and bitwise_and the two images? But I'm not too sure on how to do that, or if that's the best approach.

I am aware that the pink mask and the sample guy are not the exact fit, but I just want to be able to fit the model in the pink segment first, and perhaps scale or transform it afterwards.

Any suggestions will be great! Thank you!

Are the backgrounds for all of your images always solid black? — Jello, Mar 01 '19 at 16:44
@Jello actually on second thought, maybe not.. but it will always be a solid colour.. does that help? — Jenny, Mar 01 '19 at 16:46
@Jello alternatively, I can make it such that the background is solid black, but the mask is an actual image (not colour segmented...). so something like the guy model picture. except I'll be replacing one guy with another guy. Does that make sense? — Jenny, Mar 01 '19 at 16:52
Ah, yes that makes sense, you should be able to rescale the replacement image, grab all pixels with nonzero values and slap them in the center of the target image (in this case the target doesn't necessarily need a mask and can just be made all black) — Jello, Mar 01 '19 at 16:55
@Jello That did come across my mind! I have a question though, how do i get the center of the target image? — Jenny, Mar 01 '19 at 16:57

Jello · Accepted Answer · 2019-03-03T15:16:17.167

Load libs and read images

from __future__ import division
import cv2
import numpy as np

guy_img = cv2.imread('path')
target_img = cv2.imread('path')

Get bounding boxes for the guy (every pixel that is non-zero) and for the target mask (the pink region).

a = np.where(guy_img > 0)
b = np.where(target_img == 129)  # picked one of the channels in your image
bbox_guy = np.min(a[0]), np.max(a[0]), np.min(a[1]), np.max(a[1])
bbox_mask = np.min(b[0]), np.max(b[0]), np.min(b[1]), np.max(b[1])

Note that when I loaded the target image in the values were different from the ones you provided. There were also some white pixels along the edges of the image. That was likely just due to the image being uploaded to imgur and downloaded. If your values are correct and the rest of your image is completely black except for the pink region you can just get all non zero pixels the same way I did for the guy image with np.where(target_img > 0).

Now get just the values of the guy and mask regions

guy = guy_img[bbox_guy[0]:bbox_guy[1], bbox_guy[2]:bbox_guy[3],:]
target = target_img[bbox_mask[0]:bbox_mask[1], bbox_mask[2]:bbox_mask[3],:]

Resize the guy to be the same scale / shape as the mask

guy_h, guy_w, _ = guy.shape
mask_h, mask_w, _ = target.shape
fy = mask_h / guy_h
fx = mask_w / guy_w
scaled_guy = cv2.resize(guy, (0,0), fx=fx,fy=fy)

Reassign the masked region of the target image with the guy image values

for i, row in enumerate(range(bbox_mask[0], bbox_mask[1])):
    for j, col in enumerate(range(bbox_mask[2], bbox_mask[3])):
        target_img[row,col,:] = scaled_guy[i,j,:]

cv2.imshow('', target_img)
cv2.waitKey(0)

Hi jello thanks for the help! Regarding getting the center of the photo, the method you’ve provided only works if the human is always standing right in the middle of the image am I right? But for cases where the human may be standing at different location in a photo, how should I approach that? — Jenny, Mar 02 '19 at 00:31
@Jenny, I updated my response, is that more of what you are looking for? — Jello, Mar 02 '19 at 20:56
i really appreciate the updated response and yes it is what i'm looking for! one final question (sorry for bothering), but how did you determine the value 129 in `b = np.where(target_img == 129)`? — Jenny, Mar 03 '19 at 13:24
Check the **note** that i posted above, does that make sense? — Jello, Mar 03 '19 at 19:58

Bhaumik Mistry · Answer 2 · 2019-03-01T17:11:05.173

First you need to consider few things before you go ahead with the implementation.

Consistent background for mask image? consistent color of masked blob?
Size of the mask and the object to add image.

I would suggest you to first find the size, color and location of the ROI on the pink segmented image. Scale the dimensions of the Object(Boy image) according to the mask image, so it gets easier to mask the object image to the pink part.

You can then create an image (lets call is step one Image) with dimension similar to mask image. Add the scaled object image to it at location obtained from the pink blob analysis. This way you will have two images with similar dimension and similar scale.

With the step one image and Pink ROI should be very easily overlap-able visually.

if you want exact pink then replace all the pixels on the step one image with the black pixel if the same location pixel is black on the masked image. Otherwise let it be what it is.

if(mask[i][j] == black){ step_one[i][j] = black }

which leaves the pink part masked out of object image.

Thank you for the explanation! Im able to visualize it in my head. one question, how do I get the location of the ROI of the pink segment? Is there a function that I can use? — Jenny, Mar 01 '19 at 17:21
The blob analysis functions usually gives you radius center and other characteristics of blobs. Take a look at this example links [another question for blob analysis](https://stackoverflow.com/questions/8076889/how-to-use-opencv-simpleblobdetector) [full example](https://www.learnopencv.com/blob-detection-using-opencv-python-c/) — Bhaumik Mistry, Mar 01 '19 at 18:04

How to replace a masked part of an image with another image using openCV python?

2 Answers2