You could certainly use a view stub as the comment suggests as these are reasonably inexpensive view elements.
However, you may still find you're not doing it the most efficient way if those 10 views of the different layout are quite expensive themselves.
I'm guessing you're considering manually inflating as you really wanted the benefits of a listview for these 10 elements- all those good things like view recycling, and the opportunity to use efficient patterns like ViewHolder BUT you rightly don't want to use a listview in a scrollview because that's a bad idea
Well, you might still be able to use a listview. Don't use the outer scroll view. Split the layouts into two groups- those below and above the contentPanel. Create a list view for the 10 different views and then the two groups to to that list view as a headerView and afooterView
Perhaps this will help.