how to package spark scala application

Question

I have developed a standalone spark scala application which uses SparkSQL and SparkStreaming.

This works fine in Eclipse which is configured for spark.

I am newbie in maven.

To package this application using maven, I have followed the below tutorial

http://ryancompton.net/2014/05/19/sample-pomxml-to-build-scala--jar-with-dependenciesjar/

But ended up with the following error.

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source (scala-compile-first) on project TweetAnalyzer: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30: Failed to read artifact descriptor for org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source (scala-compile-first) on project TweetAnalyzer: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:104)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
    ... 19 more
Caused by: org.apache.maven.plugin.PluginResolutionException: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:204)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.createPluginRealm(DefaultMavenPluginManager.java:359)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.setupPluginRealm(DefaultMavenPluginManager.java:327)
    at org.apache.maven.plugin.DefaultBuildPluginManager.getPluginRealm(DefaultBuildPluginManager.java:229)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:100)
    ... 20 more
Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
    at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:292)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:317)
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:192)
    ... 24 more
Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:384)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:231)
    at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461)
    at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
    at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
    at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261)
    ... 26 more
Caused by: org.apache.maven.model.resolution.UnresolvableModelException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
    at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:170)
    at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:219)
    at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:898)
    at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:750)
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:309)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:375)
    ... 31 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239)
    at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:166)
    ... 36 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
    at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$6.wrap(WagonRepositoryConnector.java:1016)
    at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$6.wrap(WagonRepositoryConnector.java:1004)
    at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:725)
    at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.maven.wagon.TransferFailedException: Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
    at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:935)
    at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
    at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
    at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
    at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:660)
    ... 4 more
Caused by: org.apache.maven.wagon.providers.http.httpclient.conn.HttpHostConnectException: Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
    at org.apache.maven.wagon.providers.http.httpclient.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:138)
    at org.apache.maven.wagon.providers.http.httpclient.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
    at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:365)
    at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
    at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
    at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec.execute(RetryExec.java:85)
    at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
    at org.apache.maven.wagon.providers.http.httpclient.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
    at org.apache.maven.wagon.providers.http.httpclient.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:756)
    at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:854)
    ... 8 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at org.apache.maven.wagon.providers.http.httpclient.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
    at org.apache.maven.wagon.providers.http.httpclient.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
    ... 18 more
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

Please provide a solution on how to package my application and run in cluster.

You might rephrase it a bit. We provide help, not solutions here :) — Niels Bech Nielsen, Nov 24 '14 at 11:34
Do you have a Maven repository proxy configured in your settings.xml? (you'll find it at `~/.m2/settings.xml`). It looks like Maven is trying to fetch dependencies from a proxy instead from Maven Central repository. See the exception: `HttpHostConnectException: Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused`. — smola, Nov 24 '14 at 13:56
Thanks for the reply. After setting the proxy settings also error exists. I have used the xml file in the above link. It shows error at the line scala-compile-first process-resources add-source compile — Vijay Innamuri, Nov 27 '14 at 06:13
I couldn't find any reference pom.xml file for packaging the spark scala code. — Vijay Innamuri, Nov 27 '14 at 06:15

score 2 · Accepted Answer · answered Feb 04 '15 at 00:51

I use sbt as my package and compiling tool. With sbt, I use "assembly" plugin to package dependencies in the final jar file. I recommend it.

If you maven, I believe "shade" plugin do the similar job. See the usage page of shade and you will be inspired. http://maven.apache.org/plugins/maven-shade-plugin/usage.html

Basically, you need to change the "build" part in your pom.xml file as this:

<project>
    ...
    <build>
       <plugins>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-shade-plugin</artifactId>
           <version>2.3</version>
           <configuration>
             <!-- put your configurations here -->
           </configuration>
           <executions>
             <execution>
               <phase>package</phase>
               <goals>
                 <goal>shade</goal>
               </goals>
             </execution>
           </executions>
         </plugin>
       </plugins>
     </build>
     ...
</project>

how to package spark scala application

1 Answers1

Linked