I need to prevent dragging and scaling of an object which can be rotated

Question

I have a canvas with images that are placed with a rect trough which the image is shown. Outside the rect the background is shown instead of the image (so it's cropping the image). I'm trying to prevent the image not covering the cropping rect, the image should always be bigger and totally covering the rect. I got this working on straight images/rects but I can't get it to work when a rotation is involved.

I've used the code in this question to get it working on straight images: Prevent Fabric js Objects from scaling out of the canvas boundary

canvas.on('object:moving', (event: any) => {
        // this eventlistener makes sure that a moving image won't leave it's placeholder area uncovered
        //
        const activeObject = event.target;

        // the cropping Rect
        const containerRect = retrieveObjectFromCanvas(this.state.canvas, `${this.selectedImage.value.id}-container`);

        // apply new coords of current move action
        activeObject.setCoords();
        const newCoords = activeObject.getCoords();
        const containerCoords = containerRect.getCoords();

        // left
        if (newCoords[0].x >= containerCoords[0].x) {
          activeObject.left = (containerRect.left - (containerRect.width / 2) + ((activeObject.width * activeObject.scaleX) / 2));
        }

        // top
        if (newCoords[0].y >= containerCoords[0].y) {
          activeObject.top = (containerRect.top - (containerRect.height / 2) + ((activeObject.height * activeObject.scaleY) / 2));
        }

        // right
        if (newCoords[2].x <= containerCoords[2].x) {
          activeObject.left = (containerRect.left + (containerRect.width / 2) - ((activeObject.width * activeObject.scaleX) / 2));
        }

        // bottom
        if (newCoords[2].y <= containerCoords[2].y) {
          activeObject.top = (containerRect.top + (containerRect.height / 2) - ((activeObject.height * activeObject.scaleY) / 2));
        }

        activeObject.setCoords();

      });

      canvas.on('object:scaling', (event:any) => {
        // this eventlistener makes sure that a scaling image won't leave it's placeholder area uncovered
        //
        const activeObject = event.target;
        activeObject.set({ lockScalingFlip: true });
        const boundingRect = activeObject.getBoundingRect();

        // the cropping Rect
        const containerRect = retrieveObjectFromCanvas(this.state.canvas, `${this.selectedImage.value.id}-container`);

        // apply new coords of current move action
        activeObject.setCoords();
        const newCoords = activeObject.getCoords();
        const containerCoords = containerRect.getCoords();

        // left
        let corners = ['tl', 'ml', 'bl'];
        if ((includes(corners, event.transform.corner)) && newCoords[0].x >= containerCoords[0].x) {
          const newScale = (boundingRect.width) / activeObject.width;
          activeObject.scaleX = newScale;
          activeObject.left = (containerRect.left - (containerRect.width / 2) + ((activeObject.width * activeObject.scaleX) / 2));
        }

        // top
        corners = ['tl', 'mt', 'tr'];
        if (includes(corners, event.transform.corner) && newCoords[0].y >= containerCoords[0].y) {
          const newScale = (boundingRect.height) / activeObject.height;
          activeObject.scaleY = newScale;
          activeObject.top = (containerRect.top - (containerRect.height / 2) + ((activeObject.height * activeObject.scaleY) / 2));
        }

        // right
        corners = ['tr', 'mr', 'br'];
        if (includes(corners, event.transform.corner) && newCoords[2].x <= containerCoords[2].x) {
          const newScale = (boundingRect.width) / activeObject.width;
          activeObject.scaleX = newScale;
          activeObject.left = (containerRect.left + (containerRect.width / 2) - ((activeObject.width * activeObject.scaleX) / 2));
        }

        // bottom
        corners = ['bl', 'mb', 'br'];
        if (includes(corners, event.transform.corner) && newCoords[2].y <= containerCoords[2].y) {
          const newScale = (boundingRect.height) / activeObject.height;
          activeObject.scaleY = newScale;
          activeObject.top = (containerRect.top + (containerRect.height / 2) - ((activeObject.height * activeObject.scaleY) / 2));
        }
        activeObject.setCoords();
      });
    }

There are no error messages, but the image will not stop at the borders of the rect, but scale and move unpredictable. I'm not sure if it's related but the Coords calculate from the center of the object/rect to the top-left of the canvas.

Edit: A clarification. Checking if the rect is contained within the picture can be done with .isContainedWithinObject(). The hard part is calculating the values for the image so it's just outside the boundary. In the code above I calculate the top, left, and scale but those calculations only work on images with no skew or rotation.