I'd like to add another reason to not do this. If you are using pointers in this way, and if you happen to have a bug that depends on the ordering of the elements of a container, then it will be very difficult to find. Even if your program seems to be completely deterministic, it will not be. The order of the elements in a container depends on the algorithm the memory allocator uses, which is completely out of your control. If you run the same example muliple times without restarting your program, some may fail and others succeed.
This is the voice of bitter experience. I did this with a debugger project once, where I had containers filled with C++ symbols. When I needed to sort the symbols, I ended up with symbols which are different, but which have the same name (think overloaded functions) and which were identical in all other respect. So, in this case I compared them as a last resort by the address of the symbol object. I ran into several bugs which were apparently non-deterministic, where the non-determinism was caused by just this phenomenon. Sometimes it took more than 10 or 15 attempts to reproduce the problems. I eventually took the time to eliminate sorting by addresses, and that saved me a lot of trouble over the longer term.
With that said, I won't say I haven't done this recently. But every time I do it I feel like it's a mistake.