The upper left 3×3 defines the base vectors of a coordinate system. It's not just rotation but also scaling, shearing and similar stuff. Imagine three rubber legs which you can stretch and angle into any direction. This is what the upper left 3×3 does. If you want to rotate a coordinate system, you rotate the tree legs labeled X, Y, Z each into the direction you want to denote by that moniker in the other coordinate system. If you want to scale things, you stretch each leg. If you make them nonperpendicular things get sheared. That's really all there is to it.
However because everything (except translation) is intermingled in that upper left 3×3 part it's not trivial to "extract" the individual transformations from it. The rough steps to follow are:
- Major axis transform by the Principal axis theorem to find the major axis of any anisotropic scaling that may have been applied.
- decomposition into shearing and rotation parts.
Translation happens to sit in the rightmost column. The lowermost row must be [0 0 0 1] so that the stuff happening in the rows above it work in the desired way.