1

I have to cross join 2 dataframe in Spark 2.0 I am encountering below error:

User class threw exception: 

org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively expensive and are disabled by default. To explicitly enable them, please set spark.sql.crossJoin.enabled = true;

Please help me where to set this configuration, I am coding in eclipse.

ZygD
  • 22,092
  • 39
  • 79
  • 102
sks
  • 43
  • 1
  • 2
  • 10

1 Answers1

10

As the error message clearly states you need to set spark.sql.crossJoin.enabled = true to your spark configuration

You can set the same something like below:

val sparkConf = new SparkConf().setAppName("Test")
sparkConf.set("spark.sql.crossJoin.enabled", "true")

Then get or create SparkSession by passing this SparkConf

val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

Then do your join...

Prasad Khode
  • 6,602
  • 11
  • 44
  • 59