How to fix "origin location must be absolute" error in sbt project (with Spark 2.4.5 and DeltaLake 0.6.1)?

Question

I am trying to setup a SBT project for Spark 2.4.5 with DeltaLake 0.6.1 . My build file is as follows.

However seems this configuration cannot resolve some dependencies.

[info] Reapplying settings...
[info] Set current project to red-basket-pipelnes (in build file:/Users/ashika.umagiliya/git-repo/redbasket-pipelines/red-basket-pipelnes/)
[info] Updating ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.antlr#antlr4;4.7: org.antlr#antlr4;4.7!antlr4.pom(pom.original) origin location must be absolute: file:/Users/ashika.umagiliya/.m2/repository/org/antlr/antlr4/4.7/antlr4-4.7.pom
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Unresolved dependencies path:
[warn]      org.antlr:antlr4:4.7
[warn]        +- io.delta:delta-core_2.11:0.6.1 (/Users/ashika.umagiliya/git-repo/redbasket-pipelines/red-basket-pipelnes/build.sbt#L13-26)
[warn]        +- com.mycompany.dpd.solutions:deltalake-pipelnes_2.11:1.0
[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.antlr#antlr4;4.7: org.antlr#antlr4;4.7!antlr4.pom(pom.original) origin location must be absolute: file:/Users/ashika.umagiliya/.m2/repository/org/antlr/antlr4/4.7/antlr4-4.7.pom
[error]     at sbt.internal.librarymanagement.IvyActions$.resolveAndRetrieve(IvyActions.scala:332)

build.sbt

name := "deltalake-pipelnes"
version := "1.0"
organization := "com.mycompany.dpd.solutions"

// The compatible Scala version for Spark 2.4.1 is 2.11
scalaVersion := "2.11.12"

val sparkVersion = "2.4.5"
val scalatestVersion = "3.0.5"
val deltaLakeCore = "0.6.1"
val sparkTestingBaseVersion = s"${sparkVersion}_0.14.0"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-hive" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-avro" % sparkVersion % "provided",

  "io.delta" %% "delta-core" % deltaLakeCore,

  "org.scalatest" %% "scalatest" % scalatestVersion % "test",
  "com.holdenkarau" %% "spark-testing-base" % sparkTestingBaseVersion % "test"
)

assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

assemblyMergeStrategy in assembly := {
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("changelog.txt") => MergeStrategy.last
  case PathList(ps @ _*) if ps.last contains "spring" => MergeStrategy.last

  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

resolvers ++= Seq(
  "SPDB Maven Repository" at "https://artifactory.mycompany-it.com/spdb-mvn/",
  Resolver.mavenLocal)

publishMavenStyle := true
publishTo := {
  val repoBaseUrl = "https://artifactory.mycompany-it.com/"
  if (isSnapshot.value)
    Some("snapshots" at repoBaseUrl + "spdb-mvn-snapshot/")
  else
    Some("releases"  at repoBaseUrl + "spdb-mvn-release/")
}
publishConfiguration := publishConfiguration.value.withOverwrite(true)
publishLocalConfiguration := publishLocalConfiguration.value.withOverwrite(true)
credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

artifact in (Compile, assembly) := {
  val art = (artifact in (Compile, assembly)).value
  art.withClassifier(Some("assembly"))
}

addArtifact(artifact in (Compile, assembly), assembly)

parallelExecution in Test := false

Any tips on how to fix this?

Jacek Laskowski · Accepted Answer · 2021-04-04T12:09:41.987

I haven't managed to figure it out myself when and why it happens, but I did experience similar resolution-related errors earlier.

Whenever I run into issues like yours I usually delete the affected directory (e.g. /Users/ashika.umagiliya/.m2/repository/org/antlr) and start over. It usually helps. If not, I delete ~./ivy2 too and start over. That's kind of sledge hammer and does the job (aka if all you have is a hammer, everything looks like a nail).

I always make sure to use the latest and greatest sbt. You seem to be on macOS so use sdk update early and often.

I'd also recommend using the latest and greatest versions of the libraries, and more specifically, of Spark it'd be 2.4.7 (in the 2.4.x line) while Delta Lake should be 0.8.0.

How to fix "origin location must be absolute" error in sbt project (with Spark 2.4.5 and DeltaLake 0.6.1)?

1 Answers1

Linked