Extra Space in Recycled View from ListView

Question

I've got a ListView that makes the text vertical, based off the VerticalTextView code here (2nd example). I thought it was working fine, but then I started to scroll around and noticed that there were these big gaps appearing. I believe it's because of the view recycling the ListView does, but I'm not sure how to fix it. My theory is that it uses an older view and updates the text, but doesn't update the size of the entire view. Any thoughts?

On the left you can see the intial view when things are loaded, looks good! But then if I scroll down a bunch and back up, this happens.

Posting the VerticalTextView code here for convience

public class VerticalTextView extends TextView{
   final boolean topDown;

   public VerticalTextView(Context context, AttributeSet attrs){
      super(context, attrs);
      final int gravity = getGravity();
      if(Gravity.isVertical(gravity) && (gravity&Gravity.VERTICAL_GRAVITY_MASK) == Gravity.BOTTOM) {
         setGravity((gravity&Gravity.HORIZONTAL_GRAVITY_MASK) | Gravity.TOP);
         topDown = false;
      }else
         topDown = true;
   }

   @Override
   protected void onMeasure(int widthMeasureSpec, int heightMeasureSpec){
      super.onMeasure(heightMeasureSpec, widthMeasureSpec);
      setMeasuredDimension(getMeasuredHeight(), getMeasuredWidth());
   }

   @Override
   protected void onDraw(Canvas canvas){
      TextPaint textPaint = getPaint(); 
      textPaint.setColor(getCurrentTextColor());
      textPaint.drawableState = getDrawableState();

      canvas.save();

      if(topDown){
         canvas.translate(getWidth(), 0);
         canvas.rotate(90);
      }else {
         canvas.translate(0, getHeight());
         canvas.rotate(-90);
      }


      canvas.translate(getCompoundPaddingLeft(), getExtendedPaddingTop());

      getLayout().draw(canvas);
      canvas.restore();
  }
}

EDIT: Looks like something to do with the MATCH_PARENT option. If I change it to WRAP_CONTENT is is fine.