0

I have a 3D human model in Unity which is viewed by a camera at a certain pose (p1) and a 2D image rendered from this view (I1) is recorded. Then, I change the camera pose (p2) slightly such that the model is viewed by the camera from another viewpoint and a second image (I2) is recorded. Considering that I know both p1 and p2, how can I get to I2 from the reference image I1 ?

I have a dataset of pairs of camera poses (p1,p2) for which I repeat this procedure several times to get the images. After rendering in Unity, I record the images for offline processing using OpenCV.

Remarks:

  • There is a single object in the scene rendered on a skybox.
  • Difference between p1 and p2 are quite small, so there is practically no disocclusions at any part of the object.
  • The camera may move between p1 and p2 translationally such that the object appears bigger/smaller in I2 compared to I1.
  • Since I'm doing this Unity, I can control the camera intrinsics by using a physical camera (if that helps at all).
chronosynclastic
  • 1,585
  • 3
  • 19
  • 40
  • its unclear what you want to achieve... the only question: `how can I get to I2 from the reference image I1 ?` makes not much sense as according to the text before it you already know `I2` ... did you mean how to cross match feature points between I1 and I2 to allow for 3D stereo scopy reconstruction? – Spektre Aug 30 '21 at 11:01
  • Sorry for the confusion. I have I2 because I can generate that by changing the camera pose in Unity but in the real application, it won't be available. I only have the camera poses p1 and p2 in world coordinates and the image I1 (viewed from the pose p1). I'm interested in correcting rendering errors in an AR system caused by using the latest head pose of the user available on the device. Therefore I'm simulating this in Unity by capturing the poses from the device and inputting to Unity to control the camera position. – chronosynclastic Aug 30 '21 at 11:07
  • 1
    without depth information is this not doable. There are alogrithms for 3D reconstruction from 2D image based on known sizes of objects/features on the image , or infering lighting conditions or using sequence of images during camera motion. See [2. reconstruct the scene into 3D mesh](https://stackoverflow.com/a/35752786/2521214) from there its just a matter of rendering mesh or PCL – Spektre Aug 30 '21 at 14:31
  • I thought that since the change between camera poses is very small, it could be possible to slightly warp the original image by assuming that the object is planar and the two images are related by a homography. Between two poses, I observe mostly very small translational jumps around the scene and no disocclusions. Basically I want to treat the object like a plane and apply a perspective projection to modify its scale and position very slightly. – chronosynclastic Aug 30 '21 at 14:50
  • If you consider your image as planar then its just matter of rendering single textured quad with new camera orientation (just rotated by difference between poses) no big deal however if you want also the small differences due to non planar surface in view then you need to reconstruct the 3D mesh. – Spektre Aug 31 '21 at 05:58

0 Answers0