Java generics with maps

Question

In the following code, line 14 fails to compile with error:

Type mismatch: cannot convert from Map<String,Set<Integer>> to Map<String,Set<? extends Number>>

  1 import java.util.HashMap;
  2 import java.util.HashSet;
  3 import java.util.Map;
  4 import java.util.Set;
  5
  6 public class GenericTest
  7 {
  8     public static void main(String[] args)
  9     {
 10         Set<? extends Number> set = new HashSet<Integer>();
 11
 12         Map<String, Set<Integer>> intMap = new HashMap<>();
 13         Map<String, Set<? extends Number>> numberMap = intMap;
 14     }
 15 }

Is there any way to declare the numberMap so this compiles?

I know I can use Map<String, Object>, but I want to be able to retrieve values from the map as a set of numbers without having to cast.