Using Scala to Write a DF to a SQL Server Table

Question

I'm trying to use the code below.

import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._

// Aquire a DataFrame collection (val collection)

val config = Config(Map(
  "url"          -> "mysqlserver.database.windows.net",
  "databaseName" -> "MyDatabase",
  "dbTable"      -> "dbo.Clients"
  "user"         -> "username",
  "password"     -> "*********"
))

import org.apache.spark.sql.SaveMode
collection.write.mode(SaveMode.Append).sqlDB(config)

The script is from this link.

https://github.com/Azure/azure-sqldb-spark

I'm running this in a databricks environment. I'm getting these errors:

command-836397363127942:5: error: object sqlDB is not a member of package com.microsoft.azure
import com.microsoft.azure.sqlDB.spark.connect._
                           ^
command-836397363127942:4: error: object sqlDB is not a member of package com.microsoft.azure
import com.microsoft.azure.sqlDB.spark.config.Config
                           ^
command-836397363127942:7: error: not found: value Config
val bulkCopyConfig = Config(Map(
                     ^
command-836397363127942:18: error: value sqlDB is not a member of org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row]
df.write.mode(SaveMode.Append).sqlDB(bulkCopyConfig)

I'm guessing that some kind of library is not installed correctly. I Googled for an answer, but didn't find anything useful. Any idea how to make this work? Thanks.

Have you included the connector dependency `azure-sqldb-spark` ? It seems it's missing — blackbishop, Dec 19 '19 at 21:09
add this and try com.microsoft.azure azure-sqldb-spark 1.0.2 — Rajasekhar, Dec 20 '19 at 05:29

score 1 · Answer 1 · answered Dec 20 '19 at 05:34

I think you have missed the library

If your using Maven Build Add the following library in pom.xml

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-sqldb-spark</artifactId>
    <version>1.0.2</version>
</dependency>

If your using SBT Build Add the following library in build.sbt

libraryDependencies += "com.microsoft.azure" % "azure-sqldb-spark" % "1.0.2"

score 1 · Accepted Answer · answered Apr 05 '22 at 03:59

If you are getting the sqldb error means all other support libraries already imported to your notebook and only the latest JAR with dependencies are missing.

As per the repro, I got the same error message as shown above:

After bit of research, I had found that you will experience this error due to missing JAR with dependencies.

To resolve this issue, you need to download the JAR file from here: https://search.maven.org/artifact/com.microsoft.azure/azure-sqldb-spark/1.0.2/jar

After downloaded the jar file, upload the JAR library into the cluster and install it.

Note: After installing both the libraries, make sure to restart the cluster.

Now, you will be able to run the command successfully.

score 0 · Answer 3 · answered Feb 24 '21 at 13:22

Have you imported and installed the library in DataBricks?

I found it easiest to import the library using Maven. See this answer: How to install a library on a databricks cluster using some command in the notebook?

Note: You need to install the Library on your cluster and then restart the cluster before you can use it.

Using Scala to Write a DF to a SQL Server Table

3 Answers3