4

I am trying to build a demo project in java 9 with maven that uses the dependency:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.10</artifactId>
    <version>2.2.0</version>
</dependency>

However when I run the jar tool to determine the automatic module name to use in my project's module-info.java I get the following error:

$ jar --file=spark-mllib_2.10/2.2.0/spark-mllib_2.10-2.2.0.jar --describe-module
Unable to derive module descriptor for: spark-mllib_2.10/2.2.0/spark-mllib_2.10-2.2.0.jar
spark.mllib.2.10: Invalid module name: '2' is not a Java identifier

It appears that the automatic module algorithm can't come up with a name that is valid java for this jar. Without adding the proper requires I get compile errors that the packages in spark mllib are missing such as:

package org.apache.spark.mllib.linalg does not exist

Is there anyway I can use this dependency in my project before it adds an official reserved module name or its own module-info?

Naman
  • 27,789
  • 26
  • 218
  • 353
Andy Schmitt
  • 70
  • 1
  • 5
  • 1
    From the look of it seems like the [suggestion made here](https://stackoverflow.com/a/46501811/1746118) might help you as well. There isn't much progress visible from the owners on this one though. – Naman Oct 11 '17 at 08:41
  • 2
    A module name can be derived from most JAR files but there are some outliers with strange names that cannot be mapped to useful names. In this case, it would be good to submit a bug to the Apache Spark project asking them to choose a module name and add it to their JAR files with the "Automatic-Module-Name" attribute. Better still would be to publish explicit modules but they may not be ready to do that yet. – Alan Bateman Oct 11 '17 at 08:55

1 Answers1

5

For a proper, long-term solution the Apache Spark project must either add module declarations (module.info.java) to their JARs or set the Automatic-Module-Name entry in their JAR's manifest.

If you can't wait for that, you can do the latter yourself quite easily:

  1. create a file manifest.txt with the following content:

     Automatic-Module-Name: org.apache.spark.mlib2

  2. Append that entry to the JAR's manifest:

     jar --update --file spark-mllib_2.10.jar --manifest=manifest.txt

  3. Make sure it worked:

     jar --describe-module --file spark-mllib_2.10.jar

If you're planning to use this JAR in a real project, you would make it available to your colleagues, for which I would create a new version (maybe 2.10.patched-auto-name?), upload it to the company's Nexus and then edit the POMs to replace the original dependency with that one.

All in all, this is likely not worth the effort and you just might want to wait.

From the comments:

One thing to note is that the file you make the update with must end in a new line or the update silently fails.

Nicolai Parlog
  • 47,972
  • 24
  • 125
  • 255
  • shouldn't that be `manifest.mf`? – Naman Oct 11 '17 at 10:21
  • 4
    No, the name of the file that you use for the `manifest` option doesn't matter. It could be `foobar` for all `jar` cares - it simply pulls the content out the file and adds it in the right place. – Nicolai Parlog Oct 11 '17 at 10:58
  • 4
    This worked perfectly. The update file can be of any name since you are just appending the entries from it to the existing MANFEST.MF in the jar. One thing to note is that the file you make the update with must end in a new line or the update silently fails. I'll be adding the updated jar to a local repo as described here: https://stackoverflow.com/q/2229757/8757510 – Andy Schmitt Oct 11 '17 at 17:53
  • 1
    "One thing to note is that the file you make the update with must end in a new line or the update silently fails." as @AndySchmitt mentioned is important, I also gave up because of this silently failing – soulcoder Apr 26 '22 at 19:34
  • Thanks @soulcoder and AndySchmitt for that note. A mere 4.5 years later, I added it to the answer. – Nicolai Parlog Apr 27 '22 at 08:34