No you should use delegate access in that instance and the Mail.ReadWrite.Shared grant which will give access the Mailbox of the Owner and any of the Mailboxes Shared with that users (eg they have been delegated access to).
I agree what your talking about would be a nice feature being able to opt out of Application permissions for certain mailboxes/users etc is a requirement that comes up but the ability to do so doesn't exist. One way to mitigate that is with logic on the server side eg force the application to check a group before accessing Mailboxes etc and make sure auditing is switch on so you can catch any malicious use of the tokens and credentials.