Android canvas scale ondraw - imageview not scaled

Question

The Question: How to resize the custom imageview as canvas is resized through canvas.scale() within onDraw() method?

I have a problem implementing pinch zoom for custom ImageView. I am using the example from Android Developer site:

private ScaleGestureDetector mScaleDetector;
private float mScaleFactor = 1.f;

public MyCustomView(Context mContext){
    ...
    // View code goes here
    ...
    mScaleDetector = new ScaleGestureDetector(context, new ScaleListener());
}

@Override
public boolean onTouchEvent(MotionEvent ev) {
    // Let the ScaleGestureDetector inspect all events.
    mScaleDetector.onTouchEvent(ev);
    return true;
}

@Override
public void onDraw(Canvas canvas) {
    super.onDraw(canvas);

    canvas.save();
    canvas.scale(mScaleFactor, mScaleFactor);
    ...
    // onDraw() code goes here
    ...
    canvas.restore();
}

private class ScaleListener
        extends ScaleGestureDetector.SimpleOnScaleGestureListener {
    @Override
    public boolean onScale(ScaleGestureDetector detector) {
        mScaleFactor *= detector.getScaleFactor();

        // Don't let the object get too small or too large.
        mScaleFactor = Math.max(0.1f, Math.min(mScaleFactor, 5.0f));

        invalidate();
        return true;
    }
}

The problem here is: canvas.scale(mScaleFactor, mScaleFactor) resizes the canvas, but containing Imageview is not resized accordingly. For example, if the image is enlarged, then it looks clipped. As the image within image is properly zoomed in and out, but the imageview is not.

I tried to a replace the cavnas.scale() with the following code,

setScaleX (mScaleFactor);
setScaleY (mScaleFactor);

This code does zoom the whole view. But the problem with this method is, it is not smooth and flickers.

I have looked at existing libraries, don't like them as they are too complex and just for adding a zoom feature, one needs to include whole bunch of classes and interfaces. I would like to have an imageview that only supports two features: pan and zoom and I would to start with this simple method, if possible, instead of using those libraries.

I have looked at many StackOverflow posts regarding the implementation of zoom feature with imageviews, or custom views in general. No solution worked properly.