3

I have 2D png image frame

I want to place a 3D plane at a perfect rotation and position of the sofa as shown in figure final frame

I am using three.js and a perspective camera to manually add the rotation and translation to the plane.

I have the center and corner coordinates of the area where the green plane is placed. I have initial 3D coordinates position - (0,0,0) and rotation - (0,0,0). I have initial points of Object (0,0),(0,100),(140,0),(1 like initial frame

I want to transform initial frame to final frame, using OpenCV and three.js. How to find the angle of the green plane in the final frame from that 2D image to rotate and translate the green plane to the white plane?

Our end goal is to place a 3D object on a plane like this - final image

I tried to solve the same using OpenCV solvePNP but didn't get the desired result it's rotation and translation vector are not working and it is giving the wrong result.

Shailja Atkotiya
  • 39
  • 5
  • 1
    just work with homographies. – Christoph Rackwitz Aug 24 '21 at 18:20
  • you aren't gonna estimate camera parameters (field of view) for this, so no, that's the only solution if you want a perspective projection on there. define the four points, use getPerspectiveTransform. – Christoph Rackwitz Aug 24 '21 at 21:30
  • Thanks, @ChristophRackwitz, We don't want to perspective transform the plane, we are trying to place a 3D object. I have shared the image in the edited question. – Shailja Atkotiya Aug 25 '21 at 04:21
  • 1
    If you are able to detect/track 4 points on that plane you can compute a homography for that plane. From decomposition you can compute the camera's rotation/translation between frames. With that you can project the 3D object. If you cant track the plane (e.g. because it's not a real plane) you will have to compute camera motion/pose in a different way. – Micka Aug 25 '21 at 06:40
  • @Micka firstly we don't have video or multiple frames we have a single image. To find homography and decomposition we need initial camera matrix which we don't have. – Shailja Atkotiya Aug 25 '21 at 07:13
  • Use solvePnp which can decompose the camera pose for a single image – Micka Aug 25 '21 at 07:17
  • We used solvePnp but we don't have camera matrix and distortion coefficient. – Shailja Atkotiya Aug 25 '21 at 07:20
  • Try using some guessed intrinsics (principal point = image center) and 0 distortions . – Micka Aug 25 '21 at 07:26
  • 2
    You can even use a homography to subsample the 3D plane (create NxN image points on that plane) and use calibrateCamera afterwards. You obviously wont get perfect results if you dont have distortion coefficients, intrinsics, etc. – Micka Aug 25 '21 at 07:29
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/236389/discussion-between-shailja-atkotiya-and-micka). – Shailja Atkotiya Aug 25 '21 at 09:15

1 Answers1

2

An example using OpenCV calculate homography and wrap image to select region in python.

First, load your 2D png image and banner image, the banner image will replace the white region.

import cv2
import numpy as np

img = cv2.imread("2D png image.png")
banner = cv2.imread("replace region img.jpg")

Then, hand selects four corners(sofa corners in your image) to define replace a region.

#point sequence:top_left corner, top right corner, bottom left corner, bottom left corner
pt = np.array([[65,180], [122, 192], [122, 277], [67, 251]])

Next, get four corners of our banner image.

#point sequence same as hand select :top_left corner, top right corner, bottom left corner, bottom left corner
pts_banner = np.array([[0, 0], [banner.shape[1] - 1, 0], [banner.shape[1] - 1, banner.shape[0] - 1], [0, banner.shape[0] - 1]])

Next using OpenCV calculate the homography matrix and wrap image to the replace region.

homographyMat, status = cv2.findHomography(pts_banner, pt)
result1 = cv2.warpPerspective(banner, homographyMat, (img.shape[1], img.shape[0]))

Finally, blackout select region and add wrap image to input image.

cv2.fillConvexPoly(img, pt, (0,0,0))
result2 = img + result1
cv2.imwrite("result.png", result2)

banner image:

enter image description here

output:

enter image description here

Reference:

https://anishdubey.com/virtual-billboard-homography-perspective-geometric-transformation-image-opencv

yanzzz
  • 86
  • 5
  • 1
    Thanks, @yanzzz , but this is not exactly what we want. Along with the plane, we also want to place a 3D object which has the rotation orientation (normal vector) as the plane has. – Shailja Atkotiya Aug 25 '21 at 04:14