Lets say we have a paper on the floor with coordinates of its corners (47.1317829457364, 320, 144.62015503876, 837.953488372093, 547.720930232558, 741.209302325581, 361.674418604651, 331.162790697674). Now I want to rotate it naturally in the floor plane on arbitrary angle.
I think that this could be done using homography/perspective transformation somehow. But not sure how.
I can think of two ways to solve this. In any cases, you will need both the pixel coordinates of the corners and their corresponding real world coordinates, e.g. in millimeters.
The first, easier way, is to look into rectification of the plane / perspective correction. Here is an example with a plastic badge instead of a sheet of paper
Perspective correction in OpenCV using python
Once you have established the homograph H, you would be able to compute a "frontal" view of the sheet of paper (you do not actually need to warp the image into the frontal view explicitly). In this space, you can rotate by an arbitrary H_r and project back to the image by the inverse of H.
x_r = inv(H) * H_r * H * x,
where x is an arbitrary input pixel in homogeneous two-space of the image, x_r is the pixel (after in-plane rotation), H is the 3x3 homography that maps pixels to millimeters and H_r is your desired in-plane rotation, eg. 3x3 homography
H_r = [cos(phi) -sin(phi) 0
sin(phi) cos(phi) 0
0 0 1]
with a rotation angle phi.
The second more powerful approach is to use a calibrated camera. This allows you to establish a projection matrix from the homography and work in 3D space. Since you have four points in the plane already, I would like to refer you to an answer like this one:
Computing camera pose with homography matrix based on 4 coplanar points
Once you establish a 3D coordinate system on your sheet of paper, you can simulate arbitrary 3D transformations. One way to think of this is a plane induced homography. You virtually rotate the camera around the paper like so:
P' = P*T
where T could be any 3D transformation like rotation and translation.
