how to integrate spark 2.2 with hadoop 3.1 manually?

Asked Apr 19 '18 at 11:29

Active Jun 12 '19 at 23:24

Viewed 843 times

I want to use Spark version 2.2 and Hadoop latest version 3.1. Can I integrate Spark and Hadoop manually?

I have already installed Spark 2.2 with Hadoop 2.6 or later but I want to update Hadoop. Is it possible to find Hadoop directory in Spark with Hadoop folder?

I have downloaded Spark 2.2 without hadoop and hadoop 3.1.0. Both are running but when I configure spark.env.sh.template file with hadoop_home it's not working. Can anyone share proper configuration?

Thanks

edited Apr 19 '18 at 14:22

Panos

asked Apr 19 '18 at 11:29

griez007

Did you rename spark.env.sh.template to spark.env.sh ? – Panos Apr 19 '18 at 13:36
no I will change now thanks @Panos – griez007 Apr 19 '18 at 15:41
what was the outcome? – Ravi OpenSource May 06 '19 at 09:55
1

I believe you need to wait until https://issues.apache.org/jira/browse/SPARK-23534 is implemented. – attila_s Jun 12 '19 at 23:24

how to integrate spark 2.2 with hadoop 3.1 manually?

0 Answers0