There are several different issues at play here. First, the term "constant pool" refers to a very specific part of class files for string and numerical literals, or to the data structures generated from this part of class files that reside in the JVM. Password won't be stored here unless they're part of class files.
However, some String objects are indeed stored and shared throughout the program through String internment. Any string literal is automatically interned, as are any strings that you invoke the intern() method on. To the best of my knowledge, though, no other strings are stored this way, so unless you automatically intern the strings holding passwords yourself I don't think you need to worry about this.
One other issue to be aware of is that if you don't want the passwords residing in memory, you may need to be careful about garbage collection since a String that is no longer referenced could still be in memory. Similarly, if you use certain string methods like substring that share backing representations between strings, you may keep around the full password string after you're done using it.
If what you're worried about is other Java code being able to see old passwords that have been interned or that still live in memory, though, you don't need to worry. There is no way to iterate or look at the elements of the interned string pool, or to crack open a String to see its backing array.