Mediapipe, assign the landmarks to the vertices?

Question

Here is the face in fbx format that mediapipe uses for their face mesh model. It has 468 vertices. Here is the visualisation of the indices.

Here is the description of mediapipes face mesh model. It outputs landmark positions.

How do I know which landmark belongs to which vertices? For example in blender. When I import the fbx face, how can I get the same indices like the landmarks of the mediapipe face mesh model?

Answer 1

It seems like indices in the blender with fbx model are same as those provided from mediapipe face mesh solution. These indices are same as those in the mediapipe canonical face model uv visualization. This answer provides example to get a landmark by its index.

Need to have Developer Extras enabled. In edit mode the option is shown under Viewport Overlays > Developer > Indices as shown below to get indices in blender. Alternate option to get indices can be found here.

I have shown an example below with left eye landmark indices as they appear in canonical face mesh uv visualization.

Indices Visualization

Code

Code based on, https://google.github.io/mediapipe/solutions/face_mesh.html.

import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles
mp_face_mesh = mp.solutions.face_mesh

# For static images:

import glob
IMAGE_FILES = glob.glob('img.jpg')


drawing_spec = mp_drawing.DrawingSpec(thickness=1, circle_radius=1)
with mp_face_mesh.FaceMesh(
    static_image_mode=True,
    max_num_faces=1,
    refine_landmarks=True,
    min_detection_confidence=0.5) as face_mesh:
  for idx, file in enumerate(IMAGE_FILES):
    image = cv2.imread(file)
    # Convert the BGR image to RGB before processing.
    results = face_mesh.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

    # Print and draw face mesh landmarks on the image.
    if not results.multi_face_landmarks:
      continue
    annotated_image = image.copy()

    landmarks_list = [
        33, 7, 163, 144, 145, 153, 154, 155,
        133, 173, 157, 158, 159, 160, 161, 246
    ]

    for face_landmarks in results.multi_face_landmarks:
      #print('face_landmarks:', face_landmarks)

      print(len(face_landmarks.landmark))
    #   print(face_landmarks.landmark[7])

      for idx in landmarks_list:
        loc_x = int(face_landmarks.landmark[idx].x * image.shape[1])
        loc_y = int(face_landmarks.landmark[idx].y * image.shape[0])
        print(loc_x, loc_y)
        cv2.circle(annotated_image,(loc_x, loc_y), 2, (255,255,255), 2)

    cv2_imshow(annotated_image)

Output

Demo notebook

https://github.com/quickgrid/AI-Resources/blob/master/code-lab/computer_vision/Mediapipe_Python_Holistic_and_Face_Mesh.ipynb