There is no simple rule.
Instead, you must learn what pointers do and what they mean in C++, plus what they mean in your current code base.
Programming is about communicating with the compiler, and with other programmers, and with yourself in the future. The use of a pointer or a reference has different implications in all of these situations.
A pointer is a piece of paper with a street address on it. If I pass you a piece of paper with my house on it, you can use it to send a construction crew to destroy, change or rebuild my house.
A value is an actual thing. If I pass you a copy of my actual house, nothing you can do to that house will impact my original house.
A reference is an alias to a value.
The piece of paper with the house's address written on it is itself a value. I can give you another piece of paper referring to that, and using the 2nd piece of paper you can get a scribe to rewrite the first piece of paper.
With a reference, there is no value, there is just a reference to another value.
You can think of a reference as a pointer that prevents anyone from changing it and you cannot get a pointer to the pointer.
In addition, pointers, unlike references, are "nullable" -- they can be in a state where they refer to nothing. Pointers, being values themselves, can be changed; by convention, one might pass the pointer to the first element of an array of elements, plus the length, and expect the user to copy and/or modify that pointer to refer to all of the elements.
Pointers, being values, have operations you can perform on them (as opposed to what they point to). If you have an address to a house on a street, you can "get the next house on the street" (increment the pointer). Woe betide you if you ask for the house after the last house, however.
All of this is what they are. Conventionally, programmers basically never use references to hold free store allocated objects ("heap" objects), while pointers sometimes do this. In some code bases, only smart pointers "own" such references, while others only observe.
In some code bases, using a pointer in order to denote a nullable reference is valid; in others, it is anathema.