Rotating calipers are your best friends.
The tightest bounding rectangle is such that one of its sides coincides with a side of the polygon. So,
starting from an arbitrary side, rotate the polygon to make this side horizontal, at the bottom;
find the three vertices that maximize Y, -X and X respectively (on the figure, 4, 6, and 3). This way, you obtain a bounding box;
then move to the next side, clockwise, and adjust the rotation;
find the next three vertices by updating the maxima (follow the outline and stop before Y decreases; same with -X and X; on the figure, 4->5, 6->6 and 3->3);
for every position, compute the area and keep the best. Stop after a full turn.

It is important to realize that you don't apply the rotations to all vertices, but only when needed. To obtain the initial bounding box, you process O(N) vertices, but the N-1 updates only take O(N) additional work (amortized), and you don't have to compute both coordinates.