Let rectangles center is coordinate origin (and center of rotation).
Large rect dimensions are 2W
, 2H
, small rect 2w
, 2h
.
First vertex (A
) of small rect has initial coordinates (w,h)
, they become (k*w,k*h)
after fitting, second one (B
) : (-k*w,k*h)
, k
is unknown yet coefficient (we need only two vertices for calculations))

For rotation angle f
in range 0..Pi/2
(and Pi..3*Pi/2
) side of large rect which can contain A
, has starting point (F
)
px = W*cos(f)-H*sin(f)
py = W*sin(f)+H*cos(f)
and direction vector
dx = sin(f)
dy = -cos(f)
and this side has parametric equation
x = px + t * dx
y = py + t * dy
We can write equation that side contains A
k*w = px + t * dx
k*h = py + t * dy
and solve it for unknowns k, t
k1 = (px * dy - py * dx) / (w * dy - h * dx)
Similarly side that might contain B
px = W*cos(f)-H*sin(f) //same point
py = W*sin(f)+H*cos(f)
and direction vector
d2x = -cos(f)
d2y = -sin(f)
-k*w = px + t * d2x
k*h = py + t * d2y
k2 = (px * d2y - py * d2x) / (-w * d2y - h * d2x)
We need to choose smaller valid value from k1
and k2
(perhaps some results are just invalid - negative etc)
Similar calculations for angle f
in range Pi/2..Pi
(and 3*Pi/2..2*Pi
) with corresponding sides.
Implemented in Delphi to visually control correctness. Formulas depend on rotation angle quadrant: quad = 0,1,2,3 for rotation angle f in range 0..Pi/2, Pi/2..Pi and so on. (Really we can diminish ranges to 0..Pi due to the symmetry)
procedure TForm1.RectInRotRect(hw, hh, hWW, hHH: Integer; f: Double);
var
px, py, dx, dy, k, kmin: Double;
pts: array [0 .. 3] of TPoint;
quad: Integer;
begin
kmin := 999999;
quad := Floor(2 * f / Pi);
if Odd(quad) then
begin
px := hWW * cos(f) + hHH * sin(f);
py := hWW * sin(f) - hHH * cos(f);
dx := -cos(f);
dy := -sin(f);
if (hw * dy - hh * dx) <> 0 then
begin
k := (px * dy - py * dx) / (hw * dy - hh * dx);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end;
if (-hw * dx + hh * dy) <> 0 then
begin
k := (px * dx + py * dy) / (-hw * dx + hh * dy);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end
end
else
begin
px := hWW * cos(f) - hHH * sin(f);
py := hWW * sin(f) + hHH * cos(f);
dx := sin(f);
dy := -cos(f);
if (hw * dy - hh * dx) <> 0 then
begin
k := (px * dy - py * dx) / (hw * dy - hh * dx);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end;
if (-hw * dx + hh * dy) <> 0 then
begin
k := (px * dx + py * dy) / (-hw * dx + hh * dy);
if quad >= 2 then
k := -k;
if k > 0 then
kmin := Min(kmin, k);
end;
end;
kmin
is used to scale inner rectangle
Result of work:
