Generally if I'm using strings as keys in a hashmap, I don't want to have to worry about whether the string I'm using as a key for a lookup is exactly the same reference as what I may have inserted previously, that would complicate things immensely. I want to be able to create a string as a key and know that, if the map has a string with the same value as what I'm using, that the map will find it. So comparing references is not what I want, I want to compare by value.
For objects like Strings or numbers (java.lang.Integer, java.math.BigInteger, java.math.BigDecimal), typically comparing references is not useful; all people are interested in is the value. These are called value objects, and equality and hashCode are based strictly on the object's value, not on comparing references.
From a definition posted by Martin Fowler:
In P of EAA I described Value Object as a small object such as a Money or date range object. Their key property is that they follow value semantics rather than reference semantics.
You can usually tell them because their notion of equality isn't based on identity, instead two value objects are equal if all their fields are equal. Although all fields are equal, you don't need to compare all fields if a subset is unique - for example currency codes for currency objects are enough to test equality.
A general heuristic is that value objects should be entirely immutable. If you want to change a value object you should replace the object with a new one and not be allowed to update the values of the value object itself - updatable value objects lead to aliasing problems.