0

Trying to create a table in Apache Spark gives the following error.

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

I have tried following the same steps as described here Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z, but with no luck.

I have downloaded Apache Spark 3.3.1 (spark-3.3.1-bin-hadoop3.tgz version) and the hadoop.dll file as well as the winutils.exe file from the 3.3.1/bin from https://github.com/kontext-tech/winutils.

I have added the environment variables. I have added the files to the system32 folder.

I have also tried it with Apache Spark 3.3.0.

But I still get the same error message. Not sure what else to do. Can there be some security setting that is blocking the files from working? I'm using a work computer after all.

EDIT: Some clarification

  1. I downloaded spark 3.3.1 (spark-3.3.1-bin-hadoop3.tgz) from the spark website. https://spark.apache.org/downloads.html
  2. I extracted the files into my C:\Spark folder.
  3. I downloaded the hadoop.dll file as well as the winutils.exe file from the 3.3.1/bin from https://github.com/kontext-tech/winutils I put these files into my C:\Hadoop\bin folder as well as the C:\Windows\system32 folder.
  4. I added the environmental variables HADOOP_HOME = C:\Hadoop and SPARK_HOME = C:\Spark as described in STEP 7 in https://phoenixnap.com/kb/install-spark-on-windows-10. I also added these variables to the path, i.e %SPARK_HOME%\bin and %HADOOP_HOME%\bin
  5. Restarted my computer.
'name := "Simple Project"

version := "1.0"

scalaVersion := "2.12.15"

libraryDependencies ++= Seq("org.apache.spark" %% "spark-sql" % "3.3.1", "io.delta" %% "delta-core" % "2.1.1")
Rossario
  • 15
  • 1
  • 5
  • Welcome to Stackoverflow! Can you be more specific with what you've done, in a step-by-step fashion? Don't forget to add details like your path etc. If you say "I have added the environment variables" we don't know what you really did. – Koedlt Dec 05 '22 at 10:57
  • Alright, I added some additional information. I hope it makes it clearer :) – Rossario Dec 05 '22 at 11:39
  • is it a maven or sbt app ? can you share your pom.xml or equivalent ? – Abdennacer Lachiheb Dec 05 '22 at 15:21

1 Answers1

1

Looks like you're doing the correct things! I expect there is something small wrong somewhere, so these are things you can look at:

  • Verify whether your JVM is a 32 or 64 bit JVM. You can do this as explained here. From the link you added it seems like these were pre-compiled for a 64 bit JVM. Maybe you are on a 32 bit one?
  • If it's not the previous point, I expect the error to be somewhere in your environment variables. Carefully study them, looking for unwated space characters, or wrong delimiters, or something else like that.

Hope this helps!

Koedlt
  • 4,286
  • 8
  • 15
  • 33
  • I had the wrong JVM version. Installing the 64 bit JVM made it work. Thank you so much! – Rossario Dec 05 '22 at 16:07
  • Ahhh great! Don't forget to mark the question as answered (helps in the flagging of posts so reviewers don't look up answered questions :) ) – Koedlt Dec 05 '22 at 16:24