You certainly need to take the transformations into account for sorting. Applying all the transformations, and then sorting by the resulting depth (z-coordinate), is the most direct approach.
A mostly more efficient way of achieving the same thing is to apply the inverse transformations to your view direction once for each object (or set of objects that use the same transformations), and then calculate the depth of each vertex/triangle as the dot product of the vertex with this inverse-transformed view vector. This will require only one dot product per triangle, instead of applying full transformations to them. And the number of triangles is often orders magnitude larger than the number of objects.
As to which point to use: There's really no solution that will work for all cases. The center of the triangle should e as good as anything. This whole approach is an approximation that will work sufficiently well in many cases, but will not be entirely correct in some scenarios.
To illustrate the fundamental challenges with order-dependent transparency, let's look at a few examples. In the following figure, the view direction is from left to right, and we look at two triangles A and B edge on:
\
\
B
\
\ \
\ \
view -----> \
A
\
\
Visually, it's clear that B is behind A, and needs to be drawn first. Yet:
- The closest point of B is closer to the view point than the closest point of A.
- The farthest point of B is closer to the view point than the farthest point of A.
- The center point of B is closer to the view point than the center point of A.
You can't sort these triangles correctly by comparing one single depth value from each of them. To handle this properly, you have to take the geometry into account, and use more complex criteria to order them properly.
Then there are cases where there is no valid order:
\ /
\ /
view -----> \/
/\
B A
/ \
Here, there is no sorting order for A and B that would be valid. Parts of B are behind A, and parts of A are behind B. This will be the case whenever you have intersecting triangles. The only way to correctly resolve this is to split triangles.
There are also configuration without any intersecting triangles where there is no valid order. This is an example with 4 triangles, looking from the top this time:
___________
|\ \ |
__|_\________\ |___
| \ |__/
| ______\ |
|________/ \ |
| \ \_____|__
| \_______/ |
__| \ |
/__| \______________|
| \ \ |
|__________\ \|
These difficulties are an important reason why order independent transparency rendering methods are so attractive, beyond just avoiding the overhead of sorting.