I'm no expert, but I think the answer to your question depends on the circumstances - a yes or no wouldn't apply to every scenario out there.
What I see a lot of people do is make a .bash_profile function for easily changing their JAVA_HOME or other environmental variables with a simple one-word command in terminal.
I use Java8 95% of the time, so haven't done this myself. Development-wise, for the vast majority of dependency managers, build tools, IDE's, etc. there is some way to specify the Java version for a specific project, workspace, or context, depending on where you need it. If you use Maven, for example, simply add:
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
to you POM.xml
The equivalent for Gradle would be:
apply plugin: 'java'
sourceCompatibility = 1.8
targetCompatibility = 1.8
credit: How do I tell Gradle to use specific JDK version?
I believe there is a way to specify the JDK in META-INF/manifest.mf as well.
Sorry if that didn't answer your question. I have only needed to specify a specific JDK version for a handful of cases, and settings properties like these is typically how I do it.