6

I'm a working a neural network that does face reconstruction from a single image using tensorflow.

I'm trying to figure out how to render the output of the network (which is a 3D mesh) into an image in python.

Most of the libraries I found do 3D rendering in real time, I'm only looking to render and single frame.

I also need something that is fast, because this is potentially going to be part of the pre-processing for the network in real-time (apply the network to live video.

The mesh is quite simple, it's a simple face with about 30k vertices and 50k triangles.

user3208617
  • 101
  • 1
  • 2
  • 4

3 Answers3

8

I've just had a similar problem and solved it with the pyrender and its off-screen renderer.

Take a look at the minimum working example here PyrenderOSMesaSample.ipynb. It is pretty straightforward to build your own mesh and obtain rendered RGB of a scene.

UPD: Here is my MWE

import os
# switch to "osmesa" or "egl" before loading pyrender
os.environ["PYOPENGL_PLATFORM"] = "osmesa"

import numpy as np
import pyrender
import trimesh
import matplotlib.pyplot as plt

# generate mesh
sphere = trimesh.creation.icosphere(subdivisions=4, radius=0.8)
sphere.vertices+=1e-2*np.random.randn(*sphere.vertices.shape)
mesh = pyrender.Mesh.from_trimesh(sphere, smooth=False)

# compose scene
scene = pyrender.Scene(ambient_light=[.1, .1, .3], bg_color=[0, 0, 0])
camera = pyrender.PerspectiveCamera( yfov=np.pi / 3.0)
light = pyrender.DirectionalLight(color=[1,1,1], intensity=2e3)

scene.add(mesh, pose=  np.eye(4))
scene.add(light, pose=  np.eye(4))

c = 2**-0.5
scene.add(camera, pose=[[ 1,  0,  0,  0],
                        [ 0,  c, -c, -2],
                        [ 0,  c,  c,  2],
                        [ 0,  0,  0,  1]])

# render scene
r = pyrender.OffscreenRenderer(512, 512)
color, _ = r.render(scene)

plt.figure(figsize=(8,8)), plt.imshow(color);
M0nZDeRR
  • 216
  • 3
  • 7
0

There are several open source and commercial options available to you depending on your intended result. I infer from your question that you are already leveraging GPUs and you intend on leveraging their performance acceleration to produce your single frame result. With these assumptions, here are my top two recommendations:

1) If you are building a proof-of-concept, I'd leverage Blender (specifically Lux renderer).

  1. Start by exporting your scene using LuxBlend 2.5
  2. Take your TensorFlow generated mesh geometry and inject into the scene file
  3. Call Lux to render these dynamic files for each frame

2) If you are planning to develop a commercial product, I'd recommend using nVidia's iRay renderer. You can get SDK access and leverage the support team's knowledge to help you with optimizing the render's specifics.

I'd be happy to revise this answer based on any feedback (or your GitHub project)

If my assumptions were way off, this answer might be helpful: 3D Scene Renderer for Python ;-)

Community
  • 1
  • 1
Rick Lentz
  • 503
  • 6
  • 18
0

nVidia's iRay is excellent however, 50k plus triangles is a serious task for any rendering engine.

If you're looking for photorealistic quality you'll be looking at several minutes per render at 50k triangles from a simple lighting source.

The more complex your lighting becomes the slower the process. A simple texture based shadow render is much quicker and can be done in real time, the results are limited to the quality of the UV mapped texture.

Do you capture the surface material from the single image or do you apply a pre-mapped UV surface material to your standard 50k triangle model mesh?

Polynechramorph
  • 1
  • 2