Using OpenCV and Python to replace a segmented part of an image with other images

Question

I want to replace a part of an image with another image. I am able to segment an image into it's respective class, take for example, in the image links, the picture of an eagle has been segmented using solid colours to differentiate between the object and the background.

Original Photo
Segmented Photo

Is there a suitable algorithm that I can experiment with, which detects object by colour, and performs object replacement? For example, I want to replace this eagle with another type of bird (is there any way to resize the new image to fit the current segmented part of the image?).

I'm very new to this concept, so any feedback will be greatly appreciated! I'm by no means asking for solutions, I'm just looking for a starting point (algorithms, sample code, tutorials, etc) for me to explore further. Thank you so much.

score 2 · Accepted Answer · answered Dec 26 '18 at 11:57

You can use this deeplab v3 model which is trained on thousands of images to correctly segment any common object (in your case, birds). This will give you the mask image.

Now, if you try to crop only the foreground and overlay it on any other background it's very easy to do by simply using opencv's seamless clone. which makes the replacement look natural.

But if you are trying to crop only the background and replace it with another bird it could be a little bit tricky I think. Because you need to 1st remove the bird and replace it with a possible background and then place the other bird.

I hope you can start your journey here and improve along the way.

I am indeed using deeplab to segment the images :-)! I am actually more interested in keeping the background, but replacing the foreground with another bird. But thanks for introducing seamless clone, I'll look into it! — Jenny, Dec 26 '18 at 12:37