0

I am trying to use JavaConverters for a SparkMLlib (machine learning) SparkSQL simple Select Statement. The original worked until it came to creating a docker at which point it returned an error around scala.collections.JavaConversions. I learnt that JavaConversions had been deprecated for JavaConverters.

I have tried different versions of jar files and also of my dependencies and plugins in my pom.xml

List<String> filterColumns

So I changed my Select statement after seeing the following link:

Stackoverflow example of conversion from JavaConversions to JavaConverters

I changed my code as follows but it is not accepting the change!

Select("labels", (Seq<String>) JavaConverters.asScalaIteratorConverter(filterColumns.iterator()).asScala().toSeq());

The JavaConversions method worked until it came to building a docker container. This was originally:

.select("label", JavaConversions.asScalaBuffer(filterColumns).seq());

I then tried to save to a MySQL database as follows:

String dbConnectionUrl = "jdbc:mysql://localhost/lr_data";

        Properties prop = new Properties();

        prop.setProperty("driver", "com.mysql.cj.dbc.Driver");

        prop.setProperty("user", "root");
        prop.setProperty("password", "root");

MySql Dependency in my pom.xml file. 

<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
      <version>5.1.39</version>
 <!--    <version>8.0.12</version> -->

This Breaks with the following error: I have already tried different mysql jar files and executed an update on the mysql installer. Does anyone know what I am doing wrong?

Exception in thread "JavaFX Application Thread" java.security.PrivilegedActionException: java.lang.ClassNotFoundException: com.mysql.cj.dbc.Driver
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:432)
    at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:410)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:431)
    at com.sun.glass.ui.View.handleMouseEvent(View.java:555)
    at com.sun.glass.ui.View.notifyMouse(View.java:937)
    at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
    at com.sun.glass.ui.win.WinApplication.lambda$null$3(WinApplication.java:177)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.mysql.cj.dbc.Driver
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:45)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$5.apply(JDBCOptions.scala:99)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$5.apply(JDBCOptions.scala:99)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:99)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:197)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcOptionsInWrite.<init>(JDBCOptions.scala:201)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:45)
    at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
    at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
    at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:515)
    at com.jobreadyprogrammer.spark.Main.logisticRegression(Main.java:537)
    at com.jobreadyprogrammer.spark.Main.access$1(Main.java:411)
    at com.jobreadyprogrammer.spark.Main$1$1.handle(Main.java:269)
    at com.jobreadyprogrammer.spark.Main$1$1.handle(Main.java:1)
    at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
    at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
    at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)
    at javafx.event.Event.fireEvent(Event.java:198)
    at javafx.scene.Node.fireEvent(Node.java:8411)
    at javafx.scene.control.Button.fire(Button.java:185)
    at com.sun.javafx.scene.control.behavior.ButtonBehavior.mouseReleased(ButtonBehavior.java:182)
    at com.sun.javafx.scene.control.skin.BehaviorSkinBase$1.handle(BehaviorSkinBase.java:96)
    at com.sun.javafx.scene.control.skin.BehaviorSkinBase$1.handle(BehaviorSkinBase.java:89)
    at com.sun.javafx.event.CompositeEventHandler$NormalEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:218)
    at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)
    at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
    at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
    at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
    at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
    at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)
    at javafx.event.Event.fireEvent(Event.java:198)
    at javafx.scene.Scene$MouseHandler.process(Scene.java:3757)
    at javafx.scene.Scene$MouseHandler.access$1500(Scene.java:3485)
    at javafx.scene.Scene.impl_processMouseEvent(Scene.java:1762)
    at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2494)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:394)
    at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:295)
    ... 9 more
Riz Kler
  • 11
  • The config as you wrote will not work for sure, because in my sql 5.x the driver name is `com.mysql.jdbc.Driver` not `com.mysql.cj.dbc.Driver`. If there is an exception wit 8.x, I am pretty sure it is a different one. If so, post that one instead. Anyway, the conversion seems to be fine. – Nabil A. Nov 14 '19 at 17:24
  • Hi Nabil, thanks for correcting my versions mistake and it all ran locally! I was at this stage when using JavaConversions... but I am now gettting the same error I had before when trying to run the docker image! C:\Spark> docker run 26b46e548e46 Error: Could not find or load main class com.jobreadyprogrammer.spark.Main PS C:\Users\rizwa\Documents\Spark Course\workspace\sparkwithjava-master\project9> l – Riz Kler Nov 15 '19 at 11:01
  • This has no relation to the conversion code. Seems more like a packaging problem. Make sure the class/lib is really there. – Nabil A. Nov 15 '19 at 11:57
  • I agree that the type was not the issue. What do you mean "Make sure the class/lib is really there"? I have ran the POM file, maven clean: install dependency:resolve. From both eclipse and the command line. The application runs perfectly locally and I can create a docker image but I can't run the JAR file or run the docker image. Do you have any idea what am I doing wrong? – Riz Kler Nov 18 '19 at 10:22
  • Also is it necessasry to use spark-master and spark-worker environment for a Spark App? This is my 1st Spark App, my first dockerfile, dockerimage and my first spark-master, worker setup. The rest of the code works fine but this end part is delaying the release... – Riz Kler Nov 18 '19 at 10:52
  • I think this is going too far for SO comments. Take simpler examples of the different components. If you have problems with a component search for the solution or ask again. In general, things that work locally will maybe not worked when packed. The packing of the JAR must not be the same as a local run. Therefore check the JAR if everything is inside. Or maybe there is a problem with the java of your docker image. – Nabil A. Nov 18 '19 at 14:14

0 Answers0