Thinking about this question, I don't think it would be bad since object references only take up 4 bytes of memory (in a 32-bit JVM), but intuitively I feel like I'm doing something wrong when I have many (+100) references to the same object.
From a performance/resource utilization standpoint, references are waaaaaaaaaaaaaaaay more efficient than creating and destroying objects. Lots of ittybitty objects can fragment the heap and tax the memory manager/garbage collector. This is why it's usually worth the effort to make immutable objects singletons. Construction of even small objects in Java is more expensive than using references.
Most programs won't notice any significant difference, but some will.
This usually happens when I create a certain class +100 times and each need to hold the reference.
If every instance of a class references that object, use a static rather than instance variable to store the reference. You can use a static initializer to allocate it, or create a factory method to instantiate objects of the class and have the factory method allocate the referenced object the first time it is invoked.