1

Using an iPhone camera with a TrueDepth sensor I am able to capture accurate depth data in images of my face. I am capturing depth from the front, left and right sides (about 30 degrees rotation) and with the head tilted up a bit to capture under the chin (so 4 depth images in total). We are only capturing depth here, so no color information. We are cropping out unimportant data by using an ellipse frame

We are also using ARKit to give us the transform of the face anchor which is the same as the transform of the face, ref: https://developer.apple.com/documentation/arkit/arfaceanchor. It isn't possible to capture a depth image and the face transform at the same time, since they come from different capture sessions. So we have to take the depth image, and then quickly switch sessions while the user holds their face still to get the face anchor transforms. The world alignment is set the .camera so the face anchor transform should be relative to the camera, not the world origin.

We end up with 4 point clouds that look like this: left-right: chin up, left 30, front on, right 30 chin up, left, front, right

We also end up with 4 transforms. We are trying to stitch they point clouds back together to make a smooth mesh of the face using open3d in python.

The process so far is as follows:

  1. read point clouds and transforms
  2. apply inverse transforms to point clouds to return to original position w.r.t camera

I was expecting these point clouds to roughly be at the same position, but this is happening instead: after transform 1 after transform 2

As you can see the faces are still offset from one another: after transforms 3

Am I using the transforms wrong?

The python code and example point clouds and transforms are here: https://github.com/JoshPJackson/FaceMesh but the important bit is below:

dir = './temp4/'

frontPcd = readPointCloud('Front.csv', dir)
leftPcd = readPointCloud('Left.csv', dir)
rightPcd = readPointCloud('Right.csv', dir)
chinPcd = readPointCloud('Chin.csv', dir)

frontTransform = readTransform('front_transform.csv', dir)
leftTransform = readTransform('left_transform.csv', dir)
rightTransform = readTransform('right_transform.csv', dir)
chinTransform = readTransform('chin_transform.csv', dir)

rightPcd.transform(np.linalg.inv(rightTransform))
leftPcd.transform(np.linalg.inv(leftTransform))
frontPcd.transform(np.linalg.inv(frontTransform))
chinPcd.transform(np.linalg.inv(chinTransform))

I am expecting to get all of the point clouds merging together so I can remove duplicate vertices and then make a mesh

joshua jackson
  • 629
  • 1
  • 6
  • 17

1 Answers1

0

One good method is to find a mathematical reference of you faces. (work only with surfaces)

Many steps to do it:

  • Take one face and create a 2D function to map the face with a function f(x, y). The noise has to point toward z direction.
  • Fit your new function to other faces using lmfit.minimize or curve_fit
  • Use return parameters from the fit to know the offset!
Vincent Bénet
  • 1,212
  • 6
  • 20
  • On the function you linked I'm not sure what the inputs would be? – joshua jackson Mar 30 '21 at 13:19
  • For the 3D surface, I just took the first example but there are many other ways to achieve this. Same thing for the fit, it just examples. Just to give you a guideline. I am more familiar with numpy.polyfit that I already used but for curve detection. – Vincent Bénet Mar 30 '21 at 13:45
  • Small tip: you can define any existing function using a1 + a2 * x + a3 * x^2 + ... to infinite. It is a Tailor theorem! It means you can test wich polynomila deg you need, then take your 1-D fit function to define your 2D fit surface! – Vincent Bénet Mar 30 '21 at 13:51