I'm doing basic object transparency using depth sort. As depth i use distance (squared) from camera to every center of model's triangles, which i calculate as {(x1+x2+x3)/3, (y1+y2+y3)/3, (z1+z2+z3)/3}. Although result is almost fine, but there are some mistakes.
monkey head without sorting
with sorting
Is there anything i can do about those errors?
There is no way to sort the triangles in a perfect way. Look at the examples at end of the Transparency Sorting article on opengl.org wiki.
@kos: Give a look at Order Independent Transparency with Dual Depth Peeling and Alpha to Coverage.
Unless you have screen-aligned particles, sorting arbitrary triangles is really quite difficult. For a perfect result you have to start splitting triangles.
As @tibur says, you can get away with some simple approximations but order-independent transparency (OIT) is a decent solution too. It does however require OpenGL 3 era graphics features. I have an implementation available on github, which shows some ways to make exact OIT quite fast.
It's worth taking a look at "adaptive transparency", "multi-layer alpha blending" and "hybrid transparency". These are fast approximate solutions but give very good results for common scenes.
Some similar questions...
Mathematically speaking, if you know something about the body it could help sorting.
For instance the faces of a cube can be depth sorted by barycenter (face center) alone. If the center of the square face A will be closer than the center of B, B will never have any part in front of A (again, for a cube.)
In fact, for a convex body, faces can be sorted by orientation alone.
