Inflating layouts can be expensive, especially on older devices. Think of ViewStub as more of a lazy-loading mechanism for view subtrees that may or may not ever get expanded rather than a way to swap out sections of a view hierarchy at runtime. Once inflated, there's nothing wrong with swapping between VISIBLE/GONE. GONE will make the framework skip that subtree during expensive traversals.
This is true as long as you don't need the memory used by the inflated view. Setting visibility to GONE doesn't release any memory whatsoever; it only marks the view as gone for layout and drawing purposes. To release the memory of the viewstub's inflated view, you would have actually to remove the inflated view and replace it again with a new viewstub for future use.