This isn't a direct answer to your question, but some IDEs (at least IntelliJ) have an inspection to flag suspicious uses of collections, and this is definitely caught by it:
Map<String, Integer> map = new HashMap<String, Integer>();
map.put("hello", 5);
//In IntelliJ, this line gives the warning:
//Map<String, Integer> may not contain objects of type StringBuilder
System.out.println(map.get(new StringBuilder("hello")));
You mentioned you were trying to catch these problems at compile time so I thought it was worth mentioning. You could pair this with a static analysis tool for your build server, such as findbugs.
In IntelliJ the inspection is called "Suspicious collections method calls".
Edit (2011/10/18)
In FindBugs it appears that the bug GC_UNRELATED_TYPES
should catch this (though I haven't attempted to test this):
GC: No relationship between generic parameter and method argument
This call to a generic collection method contains an argument with an incompatible class from that of the collection's parameter (i.e., the type of the argument is neither a supertype nor a subtype of the corresponding generic type argument). Therefore, it is unlikely that the collection contains any objects that are equal to the method argument used here. Most likely, the wrong value is being passed to the method. (...)