NoClassDefFoundError when creating a JavaSparkContext

Question

I have a Spring Boot application to which I have added the Spark Core dependency as I would like to use a JavaSparkContext in it.

Unfortunately I get this NoClassDefFoundException when I try to initialize the application:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]
... 44 common frames omitted
Caused by: java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer
at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:187) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:70) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:76) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:195) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:146) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:473) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59) ~[spark-core_2.10-1.6.0.jar:1.6.0]
at org.vferrer.sparkker.config.SparkConfig.javaSparkContext(SparkConfig.java:28) ~[classes/:na]
at org.vferrer.sparkker.config.SparkConfig$$EnhancerBySpringCGLIB$$b19a91e3.CGLIB$javaSparkContext$0(<generated>) ~[classes/:na]
at org.vferrer.sparkker.config.SparkConfig$$EnhancerBySpringCGLIB$$b19a91e3$$FastClassBySpringCGLIB$$a68aab86.invoke(<generated>) ~[classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:318) ~[spring-context-4.2.3.RELEASE.jar:4.2.3.RELEASE]
at org.vferrer.sparkker.config.SparkConfig$$EnhancerBySpringCGLIB$$b19a91e3.javaSparkContext(<generated>) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]

This is the relevant bits of my pom.xml:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.3.1.RELEASE</version>
    <relativePath /> <!-- lookup parent from repository -->
</parent>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <java.version>1.8</java.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-eureka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-feign</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-rest</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-hateoas</artifactId>
    </dependency>

    <!-- Spark -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.6.0</version>
        <exclusions>
            <!-- Introduced to fix a problem with log4j and slfj -->
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-parent</artifactId>
            <version>Brixton.M3</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

I have tried manually to add the missing Jersey jar (both core and server) to no avail. Any ideas? Thanks!

UPDATE As suggested by @Davide, I added this new Jersey Server dependency:

<dependency>
    <groupId>com.sun.jersey</groupId>
    <artifactId>jersey-server</artifactId>
    <version>1.2</version>
</dependency>

Unfortunately, there are more errors along the way:

INFO 3300 --- [           main] o.s.c.n.eureka.InstanceInfoFactory       : Setting initial instance status as: STARTING
INFO 3300 --- [           main] c.n.d.provider.DiscoveryJerseyProvider   : Using encoding codec LegacyJacksonJson
INFO 3300 --- [           main] c.n.d.provider.DiscoveryJerseyProvider   : Using decoding codec LegacyJacksonJson
ERROR 3300 --- [           main] com.sun.jersey.spi.inject.Errors         : The following errors and warnings have been detected with resource and/or provider classes:
SEVERE: Missing dependency for field: javax.ws.rs.core.UriInfo com.sun.jersey.server.impl.template.ViewableMessageBodyWriter.ui
SEVERE: Missing dependency for field: com.sun.jersey.spi.template.TemplateContext com.sun.jersey.server.impl.template.ViewableMessageBodyWriter.tc

score 2 · Answer 1 · answered Jan 30 '16 at 18:00

2

It seems that You need to add a dependency to jersey server

<dependency>
    <groupId>com.sun.jersey</groupId>
    <artifactId>jersey-server</artifactId>
    <version>1.2</version>
</dependency>

answered Jan 30 '16 at 18:00

Davide Lorenzo MARINO

26,420
4
39
56

Thanks Davide, I already tried that but using the version 1.19 as suggested by Maven. When I used 1.2.0 I progressed a bit, but It seems that there are more dependencies missing. Let me update the question with the new error. – Victor Jan 30 '16 at 18:23

score 1 · Answer 2 · answered Feb 02 '16 at 22:31

1

Use a prebuilt jar of Spark, which already includes all the dependencies.

You can find it here.

answered Feb 02 '16 at 22:31

imriqwe

1,455
11
15

score 1 · Answer 3 · answered Jan 31 '17 at 05:42

for my case, I had a provided scope in my module's maven dependency.

when trying to run that module locally, provided is suggesting maven not to load the class using project's class path but instead use a provided version.

so comment that provided dependency out solves the issue

score 0 · Answer 4 · edited May 23 '17 at 12:33

According to the answer from here stackoverflow.com/questions/18086218 you need to use Jersey 1 which requires replacing @Davide Lorenzo MARINO dependency with something like

<dependency>
     <groupId>com.sun.jersey</groupId>
     <artifactId>jersey-servlet</artifactId>
     <version>1.19</version>
</dependency>

or use version 1.17.1 as stated in that answer.

You may require more dependencies from this list of jersey repository

score 0 · Answer 5 · answered Feb 03 '16 at 08:54

Finally, I got it running by downgrading to Spark 1.3.1. I tried all the suggestions given here to no avail.

    <!-- Spark -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.3.1</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Still, I don´t really like this solution.

NoClassDefFoundError when creating a JavaSparkContext

5 Answers5