User-selectable runtime Java dependency in IntelliJ plugin

Question

Task

We would like to create an IntelliJ IDEA plugin for a custom language. There is already a large (IntelliJ-independent) project, let's call it MyLang, written in Scala, providing parsing and typechecking for the custom language.

A major problem is that we would like the plugin to be decoupled from a specific source of the MyLang dependency, but all attempts and ideas we had or we have been told have drawbacks.

tl; dr: Is there a way to have user-selectable runtime Java dependencies in an IntelliJ plugin fulfilling the goals below?

Goals

1. Decouple the plugin from the actual used MyLang dependency source (and thus also from the version, at least non-major ones)
2. Ability to let the MyLang dependency point to a local JAR and/or compiled sources directory.
   
   The rationale is to let developers of MyLang test their latest changes with the plugin. For example, if one developer improved the typechecking in the MyLang language, then he might want to immediately test it within IntelliJ for visual feedback.
3. Non-critical: Ability to configure the MyLang dependency per IntelliJ project.
   
   There's a custom project type which the plugin is providing. A pluginwide setting would not be perfect, but is acceptable.
4. Ability to use datastructures (i.e. classes and functions) from MyLang in the plugin.

Attempts / Ideas

1. Deploy plugin with a MyLang dependency

   • ✅ 1, 2, 3
   • ❌ 4
   • ❌ Plugin gets very large in size.

2. Deploy plugin without MyLang, let user select a MyLang build JAR and use reflection.

   • ✅ 1, 2, 3
   • ❌ 4, especially ugly and non-typesafe code in the plugin.
   • ❌ One can only pass basic datastructures from Java's or Scala's standard API through the reflection channel.

3. Compile plugin against (stable) MyLang, deploy without and use classloader magic

   1. Option 1: Get access to [IntelliJ's plugin class loader](https://github.com/JetBrains/intellij-community/blob/b3ce5e06d44058eb119b53ed0d4b7e2ac3e57722/platform/core-impl/src/com/intellij/ide/plugins/cl/PluginClassLoader.java ) and call addUrl with the URL to the (user-provided) JAR
      • ✅ 1, 2, 4
      • ❌ 3 (which was non-critical, though)
      • ❌ Other drawback: addUrl is marked as deprecated, maybe it will be removed in the future.
   2. Option 2: Create a custom parent-last-child-first classloader
      • ✅ 1, 2, 3, 4 (all!)
      • ❌ Having IntelliJ's classloader and the new one in place is confusing for the plugin develeopers and very prone to errors! One has to be very cautious about passing instances from outside that newly created classloader to the inside of it. E.g. instances from datastructures provided by IntelliJ (editor components, PSIFile instances, ...) are only valid within the old ordinary IntelliJ plugin classloader.
        Accessing IntelliJ datastructures thus becomes near impossible

There's one (unresolved) discussion in IntelliJ's forums where the OP requires exactly the same setup: https://intellij-support.jetbrains.com/hc/en-us/community/posts/206121299-classloader-in-plugin