One of the biggest benefits of references over pointers is greater simplicity and readability. As always when you simplify something you make it easier to use but at the cost of flexibility and control you get with the low-level stuff (as other people have mentioned).
Pointers are often criticized for being 'ugly'.
class* myClass = new class();
Now everytime you use it you have to dereference it first either by
myClass->Method() or (*myClass).Method()
Despite losing some readability and adding complexity, people still needed to use pointers often as parameters so you could modify the actual object (instead of passing by value) and for the performance gain of not having to copy huge objects.
To me this is why references were 'born' in the first place to provide the same benefit as pointers but without all that pointer syntax. Now you can pass the actual object (not just its value) AND you have a more readable, normal way of interacting with the object.
MyMethod(&type parameter)
{
parameter.DoThis()
parameter.DoThat()
}
C++ references differed from C# / Java references in that once you assign a value to it that was it, you couldn't re-assign it (and it has to be assigned when it was declared). This was the same as using a const pointer (a pointer that could not be re-pointed to another object).
Java and C# are very high level, modern languages which cleaned up a lot of the messes that had accumulated in C / C++ over the years and pointers was definitely one of those things that needed to be 'cleaned up'.
As far as your comment about knowing pointers makes you a stronger programmer, this is true in most cases. If you know 'how' something works as opposed to just using it without knowing I would say this can often give you an edge. How much of an edge will always vary. After all, using something without knowing how it is implemented is one of the many beauties of OOP and Interfaces.
In this specific example, what would knowing about pointers help you with references? Understanding that a C# reference is NOT the object itself but points to the object is a very important concept.
#1: You are NOT passing by value
Well for starters when you use a pointer you know that the pointer holds just an address, that's it. The variable itself is almost empty and that's why it's so nice to pass as arguments. In addition to the performance gain, you are working with the actual object so any changes you make are not temporary
#2: Polymorphism / Interfaces
When you have a reference that is an interface type and it points to an object, you can only call methods of that interface even though the object may have many more abilities. The objects may also implement the same methods differently.
If you understand these concepts well then I don't think you are missing too much from not having used pointers. C++ is often used as a language for learning programming because it is good to get your hands dirty sometimes. Also, working with lower-level aspects makes you appreciate the comforts of a modern language. I started with C++ and am now a C# programmer and I do feel like working with raw pointers have helped me have a better understanding on what goes on under the hood.
I don't think it is necessary for everyone to start with pointers, but what is important is that they understand why references are used instead of value-types and the best way to understand that is to look at its ancestor, the pointer.