1

I started the spark-shell on master node with

spark-shell --jars minimal-json-0.9.4.jar spark-avro_2.11-3.1.0.jar spark-redshift_2.10-0.6.1-SNAPSHOT.jar

query the redshift table with

val DF = sqlContext.read.format("com.databricks.spark.redshift").option("url", jdbcURL).option("tempdir", s3TempDir).option("query", myQuery).option("temporary_aws_access_key_id", awsAccessKey).option("temporary_aws_secret_access_key", awsSecretKey).option("temporary_aws_session_token", token).load()

get the error:

    java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.redshift. Please find packages at https://cwiki.apache.org/confluence/display/SPARK/Third+Party+Projects
  at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:148)
  at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:79)
  at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:79)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:122)
  ... 54 elided
Caused by: java.lang.ClassNotFoundException: com.databricks.spark.redshift.DefaultSource
  at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5$$anonfun$apply$1.apply(DataSource.scala:132)
  at scala.util.Try$.apply(Try.scala:192)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$5.apply(DataSource.scala:132)
  at scala.util.Try.orElse(Try.scala:84)
  at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:132)
  ... 59 more
Hello lad
  • 17,344
  • 46
  • 127
  • 200

1 Answers1

3

You have forgotten commas :

spark-shell --jars minimal-json-0.9.4.jar,spark-avro_2.11-3.1.0.jar,spark-redshift_2.10-0.6.1-SNAPSHOT.jar
eliasah
  • 39,588
  • 11
  • 124
  • 154