2

I'm running into an issue when trying to do repeat queries with Delta Lake tables in Zeppelin. This code snippet runs without any problems the first time through:

import io.delta.tables._
val deltaTable = DeltaTable.forPath("s3://bucket/path")
deltaTable.toDF.show()

But when I try to run it a second time, it fails with this error:

java.lang.IllegalArgumentException: Could not find active SparkSession
  at io.delta.tables.DeltaTable$$anonfun$1.apply(DeltaTable.scala:620)
  at io.delta.tables.DeltaTable$$anonfun$1.apply(DeltaTable.scala:620)
  at scala.Option.getOrElse(Option.scala:121)
  at io.delta.tables.DeltaTable$.forPath(DeltaTable.scala:619)
  ... 51 elided

I can restart the Spark interpreter and run the query again, but this is a huge impediment to development. Does anyone know why this is happening and whether there is a workaround that doesn't involve restarting the interpreter every time I want to run a new query?

kellanburket
  • 12,250
  • 3
  • 46
  • 73

1 Answers1

0

I got the same exception

Exception in thread "main" java.lang.IllegalArgumentException: Could not find active SparkSession
at io.delta.tables.DeltaTable$.$anonfun$forPath$1(DeltaTable.scala:1142)
at scala.Option.getOrElse(Option.scala:189)

here people offer to use the latest version
here people offer to use sparkSession.catalog.clearCache() (works in my case)
also, there is a code example of creating a Spark session

Mike
  • 61
  • 1
  • 4