1

I have five pictures of wound models taken from different angles links for the pictures is provided here

I have used SFM for computing a mesh , the picture of the mesh is presented below enter image description here. I would like to extract only the features associated with the wound region compute volume and depth accordingly.

In order to solve this question, i have used U-Net segmentation to generate a 2D mask of the wound from 2d pictures , an example of 2D mask generated using U-Net is shown below .

I would like to know how i can map this mask onto 3D mesh and extract specific region within the 3D mesh which deals with wound part while removing other regions.

Any other ideas on how to segment the 3D mesh and extract specific region of interest are greatly appreciated, since i don't have different wound models i cannot apply supervised learning using 3D U-Net.

DevanDev
  • 161
  • 1
  • 2
  • 17

1 Answers1

0

Convert the image and mesh to numpy array and follow the steps:

  1. Duplicate the 2D mesh and stack it to make it a 3D mesh. for example, like this:
from skimage.transform import resize
mesh = np.array(mesh)
mesh = resize(mesh, (HEIGHT, WIDTH, 3))
mesh3D = np.array([mesh]*3).reshape(HEIGHT, WIDTH, 3)
  1. Convert the pixel value of the mesh to binary (0,1). Set the part of the mesh where the wound is present to 1 and the rest to 0.
  2. Multiply the mesh with the image.
  3. Part of the mesh where the value is 1, that part of the image will remain as it is and the part of the mesh where the value is 0, that part of the image will be set to 0
MD Mushfirat Mohaimin
  • 1,966
  • 3
  • 10
  • 22
  • I tried reshaping and i get a value error : ValueError: cannot reshape array of size 3686400 into shape (512,512,3), I have chosen HEIGHT , WIDTH to 512 – DevanDev Feb 21 '22 at 14:17
  • @DevanDev you will have to resize your image first, then you will be able to reshape it – MD Mushfirat Mohaimin Feb 21 '22 at 14:18
  • I have done that using cv2.resize but still i get same value error : ValueError: cannot reshape array of size 3145728 into shape (512,512,3) – DevanDev Feb 21 '22 at 14:22
  • @DevanDev I have added a way to resize, check it out – MD Mushfirat Mohaimin Feb 21 '22 at 14:25
  • My mesh file is in .obj and its a 3D mesh and i am using trimesh to read my mesh file as mesh = trimesh.load ('/home/Downloads/OneDrive_1_12-20-2021/texturedMesh.obj') , why should i duplicate my mesh in this case – DevanDev Feb 21 '22 at 14:28
  • @DevanDev save the mesh as jpg or png at the perspective of the mask, and then load the mesh as a numpy array and follow the steps – MD Mushfirat Mohaimin Feb 21 '22 at 14:32
  • Can't we map the 2D mask region onto 3D mesh without performing the stacking it, i say this because when i try your approach it fails badly, my algorithm actually creates a 3D mesh in .obj file format – DevanDev Feb 21 '22 at 14:37
  • @DevanDev I don't know any other way. – MD Mushfirat Mohaimin Feb 21 '22 at 14:37