We called it "z-fighting" - but a more common term is z-ordering or z-buffer problems.
I'll try and explain the problem without diagrams 1st.
3D renderers work by drawing the polygons from back to front. It does this by assigning each polygon to one of a discrete number of positions. This works for most scenes.
You get a problem when there aren't enough slots to be able to decide which polygon is in front of another.
To solve it - it will depend on your application, but there are a number of things which should be common to all:
- Make sure the bounding box of the world is as small as possible.
- Increase the colour depth of your display. Both OpenGL and Direct3D can tie the number of z slots to the resolution of the display.
- Increase the number/decrease the size of your triangles. Smaller triangles are going to be less affected by this - but will decrease the performance as there are more to draw.
If you are running on Windows you should set the desktop to the highest colour resolution. You application might pick that up as the default to use.
Can you switch between OpenGL and Direct3D rendering? If so then try both, one might give better results than the other.