7

I am trying to write my dataframe to a mysql table. I am getting No suitable driver found for jdbc:mysql://dbhost when I try write.

As part of the preprocessing I read from other tables in the same DB and have no issues doing that. I can do the full run and save the rows to a parquet file so it is definitely reading from the mysql DB.

I am submitting using:

spark-submit --conf spark.executor.extraClassPath=/home/user/Downloads/mysql-connector-java-5.1.35-bin.jar --driver-class-path /home/user/Downloads/mysql-connector-java-5.1.35-bin.jar --jars /home/user/Downloads/mysql-connector-java-5.1.35-bin.jar main.py

And I am writing using:

df.write.jdbc(url="jdbc:mysql://dbhost/dbname", table="tablename", mode="append", properties={"user":"dbuser", "password": "s3cret"})

Matthew Jones
  • 944
  • 9
  • 17

2 Answers2

2

This is a bug related the the classloader. This is the ticket for it: https://issues.apache.org/jira/browse/SPARK-8463 and this is the pull request for it: https://github.com/apache/spark/pull/6900.

A workaround is to copy mysql-connector-java-5.1.35-bin.jar to every machine at the same location as it is on the driver.

Matthew Jones
  • 944
  • 9
  • 17
  • "to every machine at the same location"? I've just one machine and the driver is at a single location. What do you mean by that? – user3072843 May 26 '20 at 20:54
1

It seems that you may have triggered a bug in Spark SQL. There seems to be a fix, the commit is e991255e7203a0f7080efbd71f57574f46076711 (see https://mail-archives.apache.org/mod_mbox/spark-commits/201505.mbox/%3C6ec5793fb810447388384f3ac03ca670@git.apache.org%3E ) and it describes the problem as "The problem is in java.sql.DriverManager class that can't access drivers loaded by Spark ClassLoader." Probably the simplest solution is to try the latest version from master, or failing that cherry-pick the commit into your branch.

Holden
  • 7,392
  • 1
  • 27
  • 33