After some clarifications the main question seems to be How to combine solid meshes and volumes such that when intersecting, the volume is only rendered up to the surface of the solid object
To answer this, we need to render the scene in two passes
- Render all solid objects into a frame buffer object (FBO) with an attached depth buffer
- Render all volumes with modified step count and initial color.
I'll explain the details in the following sections
Modified step count
Normally, you compute two points for your ray marching algorithm: The near and far intersection of the ray with your cube. This can be accomplished by transforming the ray x = o + t*d
into cube space and taking the minimum t_min
and maximum value t_max
for t
. If you want solid objects to intersect your volume, you need to make sure that these points are adjusted correctly.
- The near point normally doesn't have to be adjusted, since the pixel would not be rendered if the front of the cube was behind the surface of the solid object.
- The far point needs to be adjusted based on the depth value of the current pixel:
- Compute the world space coordinates of the object's surface at the current pixel (by applying the inverse projection and view transform)
- Compute the cube space coordinates
p
of the object's surface by applying the cube's inverse world transformation.
- Compute
t_surf
such that p = o + t_surf * d
, i.e. calculate the distance at which the ray reaches the solid surface
- Take the minimum between
t_max
and t_surf
and make it your new maximum distance t_max
from the ray origin. This way, you ignore all parts of the volume that lie behind the solid surface.
Modified initial color
With the current modification, the volume gets correctly culled at the solid surface, but still we would only see a black background, not the solid surface's color shining through.
To solve this, simply stick to back-to-front composition and set the initial value to the color of the surface (instead of just leaving it black like in the normal case). This way, if the volume is completely transparent at a certain point, you see the solid surface, if it is somewhat opaque, it gets mixed with the volume in front of the surface.