Unable to connect cosmos table api from databricks throws errror

Question

Loaded proper library at cluster level. com.microsoft.azure:azure-cosmosdb-spark_2.4.0_2.11:3.7.0

Gave proper connection strings from cosmos table api

    cosmosConfig = {
  "Endpoint" : "https://cosmos-account-name.table.cosmos.azure.com:443/",
  "Masterkey" : "PrimaryKey",
  "Database" : "TablesDB",
  "Collection" : "Deals_Metadata"
}

Started reading this using spark api.

cosmosdbConnection = spark.read.format("com.microsoft.azure.cosmosdb.spark").options(**cosmosConfig).load()

when i execute this throws below error.

java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

score 1 · Accepted Answer · answered Jul 20 '22 at 10:59

I tried to reproduce same in my environment I got same error.

To resolve this error, try to check com.azure.cosmos.spark jar properly installed or not and also follow below code.

Endpoint = "https://xxxx.documents.azure.com:443/"
MasterKey = "cosmos_db_key"
DatabaseName = "<dbname>"
ContainerName = "container"

spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog")
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", Endpoint)
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", MasterKey)

spark.sql("CREATE DATABASE IF NOT EXISTS cosmosCatalog.{};".format(DatabaseName))
spark.sql("CREATE TABLE IF NOT EXISTS cosmosCatalog.{}.{} using cosmos.oltp TBLPROPERTIES(partitionKeyPath = '/id', manualThroughput = '1100')".format(DatabaseName, ContainerName))

Reading the data into spark Dataframe

Cfg1 = {
  "spark.cosmos.accountEndpoint": Endpoint,
  "spark.cosmos.accountKey": MasterKey,
  "spark.cosmos.database": DatabaseName,
  "spark.cosmos.container": ContainerName,
  "spark.cosmos.read.inferSchema.enabled" : "false"
}

df = spark.read.format("cosmos.oltp").options(**Cfg1).load()
print(df.count())

Reference :

Manage data with Azure Cosmos DB Spark 3 OLTP Connector for SQL API | Microsoft

Thanks your kind efforts...Its working fine...com.azure.cosmos.spar jar to be installed...I did n't find this any other doc that this jar to be installed.. — Jaya Prakash, Jul 21 '22 at 09:57
Looks need minimum 15 reputation to mark it as answer.. Currently i have only 9.. — Jaya Prakash, Jul 22 '22 at 04:21
Please mark this question as userfull then wil get points so that i can mark it as answer.. — Jaya Prakash, Jul 22 '22 at 04:27

Unable to connect cosmos table api from databricks throws errror

1 Answers1