It's a common N+1 problem, What is SELECT N+1?, multiplied. As of performance, hard to tell: most probably, you'll lose performance. But maybe not.
The problem is, if you try and iterate over an extra-lazy collection, you'll get N+1 query to the database: one for the whole collection, one per entity. It might or might not be good, depending on caching and the nature of your data.
Update: Common sense suggests coding the simplest thing that works (i.e. default settings), then profiling under real-usage scenarios, and deciding on the actual approach after you have data.
There's no real sense in sticking to one particular solution upfront. If you only need one element from collection, extra-lazy retrieval will win. Perhaps. If you need the whole collection, extra-lazy retrieval will impact the performance. That is, if you don't have the whole domain cached, in which case you don't care. Usually.
As of Criteria API and explicit declaration, it sort of undermines the goal of Hibernate: you'd like your business layer not to care (as long as reasonably practicable) about mapping details. So as long as you can keep mapping settings in mappings themselves, it's better to do so. As always, YMMV.