Scala script engine not initialized when starting java -jar file. javax.script.ScriptException: Failed to compile ctx

Question

i am trying to compile scala code in runtime in java programm. I am using jsr232 api and my code looks like:

ScriptEngineManager manager=new ScriptEngineManager();
Scripted engine = (Scripted) manager.getEngineByName("scala");
engine.compile(sourceCode);

My pom looks like:

<properties>
   <scala.version>2.13.10</scala.version>
</properties>

<dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-reflect</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
       </dependency>
</dependencies>

When i am running such code in IDE all is ok and i have no problems (all dependencies jars are in -classpath option) But after i try to package to jar with copy dependencies (so i have small jar with fat manifest and put all dependencies to lib folder).

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <outputDirectory>${project.build.directory}/lib</outputDirectory>
                    <overWriteReleases>false</overWriteReleases>
                    <overWriteSnapshots>false</overWriteSnapshots>
                    <overWriteIfNewer>true</overWriteIfNewer>
                </configuration>
</plugin>
<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.2.2</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <classpathPrefix>lib/</classpathPrefix>
                            <mainClass>$$$MYCLASS$$$</mainClass>
                        </manifest>
                        <manifestEntries>
                            <Class-Path>.</Class-Path>
                        </manifestEntries>
                    </archive>
                </configuration>
            </plugin>

and run my application i got null on (Scripted) manager.getEngineByName("scala");

After some research i found that javax.script.ScriptEngineManager contains scala engine factory; But 'javax.script.ScriptEngineManager#getEngineByName(String)' returns null if something go wrong during method execution(without printstacktrace). And my problem is on ScriptEngine engine = spi.getScriptEngine();

I manually try to create Scala script engine the same way as manager and Scala.Factory(Engine factory) do this with some debug settings:

if(engine==null) {
    log.warn("No scala script engine exists. Try load again");
    try {
        Settings settings = new Settings();
        settings.usejavacp().value_$eq(true);//same as scala code 
        settings.usemanifestcp().value_$eq(true); //same as scala code
                settings.Yreplclassbased().value_$eq(true); //same as scala code
                settings.verbose().value_$eq(true); //for more logs
                settings.debug().value_$eq(true); //for more logs
        Scripted.Factory fact = new Scripted.Factory();
        engine= Scripted.apply(fact,settings, ReplReporterImpl.defaultOut());
    }catch (Exception e){
        log.error("Scala compiler wasn't initialized",e);
        return;
    }
}

I got next errors. Top level error is:

javax.script.ScriptException: Failed to compile ctx
        at scala.tools.nsc.interpreter.shell.Scripted.<init>(Scripted.scala:89) ~[scala-compiler-2.13.10.jar:?]
        at scala.tools.nsc.interpreter.shell.Scripted$.apply(Scripted.scala:278) ~[scala-compiler-2.13.10.jar:?]
        at scala.tools.nsc.interpreter.shell.Scripted.apply(Scripted.scala) ~[scala-compiler-2.13.10.jar:?]

and some other errors from compiler:

java.lang.NullPointerException
        at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
        at scala.reflect.io.ManifestResources$$anon$3.close(ZipArchive.scala:434)
        at scala.tools.nsc.symtab.classfile.ReusableDataReader.reset(ReusableDataReader.scala:85)
......
java.io.IOException: class file 'file:..../target/lib/scala-library-2.13.10.jar(scala/Predef.class)' is broken
(class java.lang.NullPointerException/null)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.scala$tools$nsc$symtab$classfile$ClassfileParser$$handleError(ClassfileParser.scala:126)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:134)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:132)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
......
java.lang.NullPointerException
        at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
        at scala.reflect.io.ManifestResources$$anon$3.close(ZipArchive.scala:434)
        at scala.tools.nsc.symtab.classfile.ReusableDataReader.reset(ReusableDataReader.scala:85)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2(ClassfileParser.scala:161)
......
java.io.IOException: class file 'file:..../target/lib/scala-library-2.13.10.jar(scala/Unit.class)' is broken
(class java.lang.NullPointerException/null)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.scala$tools$nsc$symtab$classfile$ClassfileParser$$handleError(ClassfileParser.scala:126)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:134)
        at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$parseErrorHandler$1.applyOrElse(ClassfileParser.scala:132)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
        at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:144)

It looks like scala compiler for some reason can not read jar source files.

Java version: openjdk version "11" 2018-09-25 OpenJDK Runtime Environment 18.9 (build 11+28) OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

Updated after Dmytro answer

It helps. Thanks alot.

But still got 1 issue with runtime classloading. I want user be able to add new dependency jar runtime.

On first version i do it like: engine.intp().addUrlsToClassPath(new ArraySeq.ofRef<>(url)). And it works well(i mean well when run inside IDEA)

Now i try do it like:

private final ScalaCompilerClassLoader urlClassLoader = new ScalaCompilerClassLoader(new URL[]{},ClassLoader.getSystemClassLoader());

private static class ScalaCompilerClassLoader extends URLClassLoader {
    public ScalaCompilerClassLoader(URL[] urls, ClassLoader parent) {
           super(urls, parent);
    }

    public void add(URL url){
        super.addURL(url);
    }
}

and i use your code with my classLoader.

JavaUniverse.JavaMirror mirror = universe.runtimeMirror(urlClassLoader);

The issue is that i can add new jar to my urlClassLoader before first call to toolBox.eval(toolBox.parse(sourceCode)). After first toolBox.eval call newly injected dependencies are not recognized by compiler.

And now i have to create new toolBox after each new jar dependency injecting.

Dmytro Mitin · Answer 1 · 2023-02-22T10:29:31.447

Could you try to replace

ScriptEngineManager manager=new ScriptEngineManager();
Scripted engine = (Scripted) manager.getEngineByName("scala");
engine.compile(sourceCode);

with

import scala.reflect.api.JavaUniverse;
import scala.tools.reflect.ToolBox;

// ...

scala.reflect.runtime.package$ runtime = scala.reflect.runtime.package$.MODULE$;
JavaUniverse universe = runtime.universe();
JavaUniverse.JavaMirror mirror = universe.runtimeMirror(this.getClass().getClassLoader());
scala.tools.reflect.package$ toolsReflect = scala.tools.reflect.package$.MODULE$;
ToolBox<?> toolBox = toolsReflect.ToolBox(mirror).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
int res = (int) toolBox.eval(toolBox.parse("1 + 1")); // 2

?

Does this change anything for you?

Using scala macro from java

How can I run generated code during script runtime?

How to compile and execute scala code at run-time in Scala3?

"eval" in Scala

Regarding your original code with JSR232 scripting. Could you check whether anything changes if you add fork := true (sbt syntax) to the build file? I guess in Maven this should be <fork>true</fork>.

Regarding your question with classloading. One option is to use protected method addURL

ClassLoader classLoader = this.getClass().getClassLoader();
URLClassLoader urlClassLoader = (URLClassLoader) classLoader;
Method addUrlMethod = URLClassLoader.class.getDeclaredMethod("addURL", URL.class);
addUrlMethod.setAccessible(true);
String url = "file:///home/dmitin/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_2.13/2.3.10/shapeless_2.13-2.3.10.jar";
addUrlMethod.invoke(urlClassLoader, new URL(url));
Object res = toolBox.eval(toolBox.parse("import shapeless._; 1 :: \"a\" :: HNil"));
// 1 :: a :: HNil

Another is to create a child class loader, new runtime mirror and toolbox

ClassLoader classLoader1 = new URLClassLoader(new URL[]{new URL(url)}, classLoader);
JavaUniverse.JavaMirror mirror1 = universe.runtimeMirror(classLoader1);
ToolBox<?> toolBox1 = toolsReflect.ToolBox(mirror1).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
Object res = toolBox1.eval(toolBox1.parse("import shapeless._; 1 :: \"a\" :: HNil"));
// 1 :: a :: HNil

Every toolbox has two mirrors (and two class loaders). The first is the one it was created with, i.e. the mirror of toolbox dependencies. The second is its own mirror toolbox.mirror, where new definitions live (the class loader is mirror.classloader).

See the following example, where I create a new class, instantiate it, add new JAR, and use previous definition

import scala.reflect.api.JavaUniverse;
import scala.reflect.api.Symbols;
import scala.reflect.api.Trees;
import scala.tools.reflect.ToolBox;
import java.net.URL;
import java.net.URLClassLoader;

public class App {
    public static void main(String[] args) throws Exception {
        new App().run();
    }

    private static scala.reflect.runtime.package$ runtime = scala.reflect.runtime.package$.MODULE$;
    private static JavaUniverse universe = runtime.universe();
    private static ClassLoader classLoader = App.class.getClassLoader();
    private static JavaUniverse.JavaMirror mirror = universe.runtimeMirror(classLoader);
    private static scala.tools.reflect.package$ toolsReflect = scala.tools.reflect.package$.MODULE$;
    private static ToolBox<?> toolBox = toolsReflect.ToolBox(mirror).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
    public static Symbols.SymbolApi symbol = toolBox.define((Trees.ClassDefApi) toolBox.parse("case class A(i: Int, s: String)"));

    public static Object a = toolBox.eval(toolBox.parse(
            "import scala.reflect.runtime.universe._;" +
            " q\"\"\"new ${App.symbol.asInstanceOf[Symbol]}(1, \"a\")\"\"\" "
    ));

    public void run() throws Exception {
        JavaUniverse.JavaMirror mirror1 = (JavaUniverse.JavaMirror) toolBox.mirror();
        ClassLoader classLoader1 = mirror1.classLoader();

        String url = "file:///home/dmitin/.cache/coursier/v1/https/repo1.maven.org/maven2/com/chuusai/shapeless_2.13/2.3.10/shapeless_2.13-2.3.10.jar";
        ClassLoader classLoader2 = new URLClassLoader(new URL[]{new URL(url)}, classLoader1);
        JavaUniverse.JavaMirror mirror2 = universe.runtimeMirror(classLoader2);
        ToolBox<?> toolBox2 = toolsReflect.ToolBox(mirror2).mkToolBox(toolsReflect.mkSilentFrontEnd(), "");
        Object res = toolBox2.eval(toolBox2.parse("import shapeless._; 1 :: \"a\" :: App.a :: HNil"));
        System.out.println(res); // 1 :: a :: A(1,a) :: HNil
    }
}

It helps. Thanks you a lot. But i still have had 1 issue. Updated my question with that issue. — evgrot, Feb 15 '23 at 09:19
@evgrot See the update regarding: 1) your original JSR232 approach 2) your classloading question. — Dmytro Mitin, Feb 16 '23 at 16:57
First option will not work at java11 and later, because system class loader is not URLClassLoader. I have already used second option. As i said i can add url before first eval call. But if i do eval then add new dependency, and do eval again, new dependencies not recognized. My workaroud is to recreate toolbox after each new dependency injected(addUrl call), because it happens were rarely. About i will try, as soon as i return to this project. — evgrot, Feb 21 '23 at 20:28
@evgrot *"system class loader is not URLClassLoader."* Why do you need system class loader? Why not to create a child class loader, which is URLClassLoader, so you'll be able to mutate its urls? *"new dependencies not recognized"* Every toolbox has two mirrors (and two class loaders). The first is the one it was created with, i.e. the mirror of toolbox dependencies. The second is its own mirror, where new definitions live. So when you're adding a JAR, use the 2nd mirror/class loader, not the 1st. See example in update. Also use fully qualified references to previous definitions (or imports). — Dmytro Mitin, Feb 21 '23 at 23:20
@evgrot When you created a new class/object with `val symbol = toolbox.define(...)` normally in Scala you use it further like `q"new $symbol(...)"`. The problem is that `q"..."` is a macro so to make it work in Java you need to run toolbox once again https://stackoverflow.com/questions/25078538/using-scala-macro-from-java Things could be easier if you could run all script calculations in Scala and then just call the result from Java. If you still can't access previous definitions please create MCVE (maybe in a new question) and we'll see how to fix it. — Dmytro Mitin, Feb 21 '23 at 23:26

Scala script engine not initialized when starting java -jar file. javax.script.ScriptException: Failed to compile ctx

1 Answers1