1

I am trying to connect to Azure cache for redis from databricks .

I have installed this package com.redislabs:spark-redis:2.3.0 from maven package in databricks. I have created a spark session with below code

SparkSession\
.builder\
.appName("myApp")\ 
.config("spark.redis.host", "my host")\ 
.config("spark.redis.port", "6379")\
.config("spark.redis.auth", "passwd")\ 
.getOrCreate()

But when I ran df.write.format("org.apache.spark.sql.redis").option("table", "people").option("key.column", "name").save()

I am getting below error.

*Py4JJavaError: An error occurred while calling o390.save.
: java.lang.ClassNotFoundException: 
Failed to find data source: org.apache.spark.sql.redis. Please find packages at
http://spark.apache.org/third-party-projects.html*

Could you please let me know the detailed steps to install all necessary libraries/jars to access redis in databricks.

I have seen below code in spark-redis python doc but I don't know how to run it in databricks.

$ ./bin/pyspark --jars <path-to>/spark-redis-<version>-jar-with-dependencies.jar

And also please let me know what is the latest spark-redis version.

Sai
  • 31
  • 2
  • I'm not completely sure, but you can have a look at `init scripts`. Databricks natively supports external JAR files using them. – Dipanjan Mallick Mar 11 '22 at 01:49
  • From the exception, it looks like the spark-redis jar is not found among cluster dependencies. How did you install the spark-redis? Do you use an interactive cluster or job? – fe2s Jun 12 '22 at 22:20

1 Answers1

-1

Redis has a Spark Package that you can download and attach to your cluster

The following notebook shows how to use Redis with Apache Spark in Azure Databricks.

For more details, refer to Azure Databricks - Redis.

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42
  • This link and notebook are in scala. But I would need python as mentioned in the question . Could you please let me know how to install the above mentioned jar with version. – Sai Mar 10 '22 at 16:15
  • -1: copy-pasted answer from here, with same caveats: https://learn.microsoft.com/en-us/answers/questions/766408/how-to-install-jars-libraries-related-to-spark-red.html – 123 May 09 '22 at 18:42
  • @123 It's same answer provided by me on MS Q&A platform. If the person is same how would you say it is copy-pasted from here :( :( :( – CHEEKATLAPRADEEP May 10 '22 at 03:55