The most common way of exposing sensitive information is to give the user (client) of the program a stack trace.
A stack trace is useful for programmers debugging a problem, and not to anyone else. So logging code should not output stack traces as a matter of course. It should output them only when an exception indicates a bug in the program. And it should output them where they can be made available to a programmer, but to as few others as possible.
If a program has a log file invisible to normal users of the program but visible to administrators (as is the case with servers), that is an appropriate place to log stack traces.
Similarly arguments apply about other sensitive information.
Although your question is entirely about security concerns, this can be considered too as a user experience (user interface) issue: the messages you give to the various users of the program should be appropriate for those users, and should provide them with information that is useful to them, without extraneous information that could confuse them. In particular, the message text of an exception should not be reported to users (but should be include as part of any stack trace).
For a client-server program, the clients have no interest in the details of why the server failed to process a request sent by the client. They need to know that the request did fail. If the request failed because of a problem with the server, rather than a faulty request by the client, they need to know that is the case, so they can contact the administrators to fix the server. If the request failed because the client sent a faulty request, the client should be told that, with a description of what was faulty about the request, so the client can send a corrected request.
Also, beware that not all exceptions indicate a problem that some user must be told about. If the program automatically handles the condition signalled by an exception, in many cases there is no need to tell the user at all about the condition signalled by the exception.