spark read contents of zip file in HDFS

Question

I Am trying to read data from zip file

can read whole text file as below

val f = sc.wholeTextFiles("hdfs://")

but don`t know, how to read text data inside zip file

Is there any possible way to do it, if yes please let me know.

Possible duplicate of [How to open/stream .zip files through Spark?](https://stackoverflow.com/questions/28569788/how-to-open-stream-zip-files-through-spark) — Chandan Ray, Aug 23 '18 at 21:23
if it is a gzip file it will be simple val rdd= sc.textFile("path") — madhu, Aug 23 '18 at 21:51

score 3 · Accepted Answer · answered Aug 24 '18 at 04:37

You can create an RDD from the zipFile with the newAPIHadoopFile command.

import com.cotdp.hadoop.ZipFileInputFormat
import org.apache.hadoop.io.BytesWritable
import org.apache.hadoop.io.Text
import org.apache.hadoop.mapreduce.Job

val zipFileRDD = sc.newAPIHadoopFile(
        "hdfs://tmp/sample_zip/LoanStats3a.csv.zip",
        classOf[ZipFileInputFormat],
        classOf[Text],
        classOf[BytesWritable],
        new Job().getConfiguration())
println("The file contents are: " + zipFileRDD.map(s => new String(s._2.getBytes())).first())

Thanks so much, this is what I was exactly looking into – sande Aug 24 '18 at 19:00 — sande, Aug 24 '18 at 19:00

spark read contents of zip file in HDFS

1 Answers1