1

Is it possible to extract colour values (such as average RGB) for different regions of the face using iOS ARKit? I have seen example of overlaying a 3D mesh over the face consisting of triangular pieces. Is there anyway to perform any kind of region wise-colour filtering such as extracting RGB average, histograms, max/min values (same as what CIFilter allows on normal frameS) but through ARKit's 3D face tracker?

I have also seen ARFaceAnchor.BlendShapeLocation allowing coefficients on movements of various elements across the face. Is there anyway to extract the mesh (and specifically its colour and other characteristic) for specific elements across the face such as nose, cheeks, lips, forehead, eyes, etc.?

P. B.
  • 33
  • 4

1 Answers1

1

See my answer here about how to get the realtime texture of the user's face using ARKIt

Once you have the face texture, you can sample the rgb data however and wherever you'd like. If you are only interested in a specific region:

  • Update the code from the linked project in the answer to only sample from a specific region (one very basic approach would be to only draw that part of the mesh you care about)

  • Sample regions from the entire 2D face texture. One challenge with this approach is that all your sampling regions would need to be defined on the 2D texture

Matt Bierner
  • 58,117
  • 21
  • 175
  • 206