dependencies of dependency are not included

Question

I have a java application that has some dependencies, among them another project with its own dependencies. Whenever I run it from Netbeans or with mvn exec is fine but when I try to package it, even though it's successfully built, at run time I get the following error:

Exception in thread "main" org.openrdf.rio.UnsupportedRDFormatException: No parser factory available for RDF format Turtle (mimeTypes=text/turtle, application/x-turtle; ext=ttl)
    at org.openrdf.rio.Rio.createParser(Rio.java:198)
    at org.openrdf.rio.Rio.createParser(Rio.java:213)
    at org.openrdf.repository.util.RDFLoader.loadInputStreamOrReader(RDFLoader.java:318)
    at org.openrdf.repository.util.RDFLoader.load(RDFLoader.java:222)
    at org.openrdf.repository.util.RDFLoader.load(RDFLoader.java:104)
    at org.openrdf.repository.base.RepositoryConnectionBase.add(RepositoryConnectionBase.java:217)
    at net.antidot.semantic.rdf.model.impl.sesame.SesameDataSet.loadDataFromFile(SesameDataSet.java:160)

so the problem is that the sesame rio parsers seem not to be included even though I can see them in my maven repository. I include all the dependencies at the included project. I also tried to exclude them and explicitly add them to my main project and it again failed.

Any suggestions?

my dependencies tree:

[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ RMLMapper ---
[INFO] be.ugent.mmlab:RMLMapper:jar:0.1
[INFO] +- junit:junit:jar:4.11:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- xom:xom:jar:1.2.5:compile
[INFO] |  +- xml-apis:xml-apis:jar:1.3.03:compile
[INFO] |  +- xerces:xercesImpl:jar:2.8.0:compile
[INFO] |  \- xalan:xalan:jar:2.7.0:compile
[INFO] +- jlibs:jlibs-xmldog:jar:1.0:compile
[INFO] |  +- jlibs:jlibs-xml:jar:1.0:compile
[INFO] |  |  +- jlibs:jlibs-core:jar:1.0:compile
[INFO] |  |  \- jlibs:jlibs-nbp:jar:1.0:compile
[INFO] |  \- jaxen:jaxen:jar:1.1.1:compile
[INFO] +- com.jayway.jsonpath:json-path:jar:0.8.1:compile
[INFO] |  +- net.minidev:json-smart:jar:1.1.1:compile
[INFO] |  \- commons-lang:commons-lang:jar:2.6:compile
[INFO] +- com.jayway.jsonpath:json-path-assert:jar:0.9.1:test
[INFO] |  \- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] +- net.antidot:db2triples:jar:1.0.2:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.1.2:compile
[INFO] |  +- mysql:mysql-connector-java:jar:5.1.25:compile
[INFO] |  +- postgresql:postgresql:jar:9.1-901.jdbc4:compile
[INFO] |  \- org.slf4j:slf4j-simple:jar:1.6.1:runtime
[INFO] +- org.openrdf.sesame:sesame-runtime:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-model:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-repository-api:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-repository-manager:jar:2.7.12:compile
[INFO] |  |  +- org.openrdf.sesame:sesame-repository-event:jar:2.7.12:compile
[INFO] |  |  +- org.openrdf.sesame:sesame-sail-memory:jar:2.7.12:compile
[INFO] |  |  +- org.openrdf.sesame:sesame-http-client:jar:2.7.12:compile
[INFO] |  |  |  \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  |  \- org.openrdf.sesame:sesame-http-protocol:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-repository-http:jar:2.7.12:compile
[INFO] |  |  +- org.openrdf.sesame:sesame-queryparser-api:jar:2.7.12:compile
[INFO] |  |  \- org.openrdf.sesame:sesame-queryresultio-api:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-repository-sparql:jar:2.7.12:compile
[INFO] |  |  +- org.openrdf.sesame:sesame-queryparser-sparql:jar:2.7.12:compile
[INFO] |  |  \- org.openrdf.sesame:sesame-queryresultio-sparqlxml:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-repository-contextaware:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-repository-sail:jar:2.7.12:compile
[INFO] |  |  \- org.openrdf.sesame:sesame-rio-trig:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-sail-api:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-sail-federation:jar:2.7.12:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.6.1:compile
[INFO] |  +- org.openrdf.sesame:sesame-queryparser-serql:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-queryresultio-binary:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-queryresultio-sparqljson:jar:2.7.12:runtime
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.2.1:runtime
[INFO] |  +- org.openrdf.sesame:sesame-queryresultio-text:jar:2.7.12:runtime
[INFO] |  |  \- net.sf.opencsv:opencsv:jar:2.0:runtime
[INFO] |  +- org.openrdf.sesame:sesame-repository-dataset:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-rio-languages:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-rio-binary:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-rio-nquads:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-rio-n3:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-rio-rdfjson:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-rio-rdfxml:jar:2.7.12:runtime
[INFO] |  +- org.openrdf.sesame:sesame-rio-trix:jar:2.7.12:runtime
[INFO] |  \- org.openrdf.sesame:sesame-sail-rdbms:jar:2.7.12:runtime
[INFO] |     \- commons-dbcp:commons-dbcp:jar:1.3:runtime
[INFO] |        \- commons-pool:commons-pool:jar:1.5.4:runtime
[INFO] +- org.openrdf.sesame:sesame-sail-nativerdf:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-sail-inferencer:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-queryalgebra-evaluation:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-queryalgebra-model:jar:2.7.12:compile
[INFO] |  +- org.openrdf.sesame:sesame-query:jar:2.7.12:compile
[INFO] |  \- org.openrdf.sesame:sesame-util:jar:2.7.12:compile
[INFO] +- com.sun.org.apache:jaxp-ri:jar:1.4:compile
[INFO] |  \- javax.xml.parsers:jaxp-api:jar:1.4:compile
[INFO] +- net.sourceforge.javacsv:javacsv:jar:2.0:compile
[INFO] +- commons-io:commons-io:jar:1.3.2:compile
[INFO] +- net.sf.saxon:Saxon-HE:jar:compressed:9.5.1-4:compile
[INFO] +- org.jodd:jodd-lagarto:jar:3.5:compile
[INFO] |  +- org.jodd:jodd-log:jar:3.5:compile
[INFO] |  \- org.jodd:jodd-core:jar:3.5:compile
[INFO] +- org.jsoup:jsoup:jar:1.7.3:compile
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.2.2:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.2.2:compile
[INFO] |  \- commons-codec:commons-codec:jar:1.6:compile
[INFO] +- com.github.fge:uri-template:jar:0.9:compile
[INFO] |  +- com.github.fge:msg-simple:jar:1.1:compile
[INFO] |  |  \- com.github.fge:btf:jar:1.2:compile
[INFO] |  +- com.google.guava:guava:jar:16.0.1:compile
[INFO] |  \- com.google.code.findbugs:jsr305:jar:2.0.1:compile
[INFO] +- org.openrdf.sesame:sesame-rio-api:jar:2.7.12:compile
[INFO] +- org.openrdf.sesame:sesame-rio-datatypes:jar:2.7.12:compile
[INFO] +- org.openrdf.sesame:sesame-rio-turtle:jar:2.7.12:compile
[INFO] +- org.openrdf.sesame:sesame-rio-ntriples:jar:2.7.12:compile
[INFO] +- org.openrdf.sesame:sesame-rio-n3:pom:2.7.12:compile
[INFO] +- org.openrdf.sesame:sesame-rio:pom:2.7.12:compile
[INFO] \- commons-cli:commons-cli:jar:1.2:compile

maybe that missing factory is an optional dependency and found in one of the dependencies that have scope runtime: org.openrdf.sesame:sesame-queryparser-serql:jar:2.7.12:runtime - usually IDE's dont differentiate that much which jars to add to the classpath - they just add all of them to be safe. In which jar is that factory? And is that jar in the deployment? — wemu, Jun 30 '14 at 10:25
How do you package? Do you leave the original jars intact? Reason I ask is that Sesame uses SPI and ServiceLoaders for its parsers, which require certain metadata to be present in the jar. — Jeen Broekstra, Jun 30 '14 at 10:53
I'm using the maven-assembly plugin with jar-with-dependencies if that answers your question Wemu. I don't do anything in particular to change the jars. I just include the plugins in my pom file. I see the following in my jar file so I assume that the dependency is (well) included: org/openrdf/rio/turtle/ org/openrdf/rio/turtle/TurtleParser.class org/openrdf/rio/turtle/TurtleParserFactory.class .... INF/maven/org.openrdf.sesame/sesame-rio-turtle/ INF/maven/org.openrdf.sesame/sesame-rio-turtle/pom.xml INF/maven/org.openrdf.sesame/sesame-rio-turtle/pom.properties — user3732870, Jun 30 '14 at 12:01
Have you tried contacting the developers of this particular library? Perhaps they can help you with the structure of the library and how to include it. — Ruben Verborgh, Jun 30 '14 at 14:59

score 10 · Accepted Answer · answered Dec 05 '14 at 13:33

starting from your considerations I found a solution for building a fat-jar of my application including Sesame. I had to use another maven plugin, which is shade and the configuration for my build is as follows:

<build>
<plugins>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <executions>
      <execution>
        <phase>package</phase>
        <goals>
          <goal>shade</goal>
        </goals>
        <configuration>
          <transformers>
            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
              <mainClass>my.package.main.MyMain</mainClass>
            </transformer>
            <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
          </transformers>
          <filters>
            <filter>
              <artifact>*:*</artifact>
              <excludes>
                <exclude>META-INF/*.SF</exclude>
                <exclude>META-INF/*.DSA</exclude>
                <exclude>META-INF/*.RSA</exclude>
              </excludes>
            </filter>
          </filters>
        </configuration>
      </execution>
    </executions>
  </plugin>
</plugins>
</build>

The most interesting parts are:

mainClass element: to specify the main
ServicesResourceTransformer: the core of our problem with sesame, which is the META-INF/services/ dir. This transformer helps in merging the dependancies services META-INF (as better explained in http://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer )
filters: to sign the jar (further details on: http://zhentao-li.blogspot.it/2012/06/maven-shade-plugin-invalid-signature.html )

Hope it helps. Cheers, Riccardo

I just noticed that you had found this solution, I updated my own answer before I was aware. +1 on yours for getting there first (and showing the actual config needed). — Jeen Broekstra, Dec 17 '14 at 02:34

Jeen Broekstra · Answer 2 · 2014-12-17T02:32:34.330

3

The problem is that you are repackaging the Sesame jars incorrectly. Maven's jar-with-dependencies target unpacks third party jars to put the contents into a single big project jar. That won't work with the Sesame libraries - you need the original jars intact with the META-INF/services/ dirs in place.

Update Instead of producing a onejar with 'jar-with-dependencies', you can use the Maven Shade plugin, which has specific config options to deal with merging the META-INF/services directories correctly.

edited Dec 17 '14 at 02:32

answered Jun 30 '14 at 13:09

Jeen Broekstra

21,642
4
51
73

How come this doesn't work with Sesame, i.e., in what regard is it different? – Ruben Verborgh Jun 30 '14 at 15:00
Also, would you perhaps know how to do this without repackaging? Is it even possible then to create a single jar for a project that uses Sesame? Should one manually add the `META-INF/services/` folder? – Ruben Verborgh Jun 30 '14 at 15:27
2

This is not specific to Sesame - jar repackaging does not work properly for _any_ java library that uses SPI / ServiceLoader functionality. The problem is that SPI service registration happens via a text file in the META-INF/services folder, and those text files get lost when repackaging. Since different jars contain files with the exact same name, manual fixing is possible but not easy, as you can not just copy the files. Instead, you need to merge their contents. – Jeen Broekstra Jun 30 '14 at 18:48
1

In the Sesame git repository, there is an ant script for doing this merge when creating the Sesame onejar artifact. I imagine you could use that as inspiration to create a similar script for your own project. – Jeen Broekstra Jun 30 '14 at 18:52
Hi, I have the same problem, have you found any solution? – riccardo.tasso Dec 04 '14 at 15:32

dependencies of dependency are not included

2 Answers2