How to read contents of a CSV file inside zip file using spark (python)

Question

I have multiple zip files containing two types of files(A.csv & B.csv)

/data/jan.zip --> contains A.csv & B.csv
/data/feb.zip --> contains A.csv & B.csv

I want to read the contents of all the A.csv files inside all the zip files using pyspark.

 textFile = sc.textFile("hdfs://<HDFS loc>/data/*.zip")

Can someone tell me how to get the contents of A.csv files into an RDD?

This might help you http://stackoverflow.com/questions/32080475/how-to-read-a-zip-containing-multiple-files-in-apache-spark — Himaprasoon, May 06 '16 at 06:52
[Read whole text files from a compression in Spark](http://stackoverflow.com/q/36604145/1560062) — zero323, May 06 '16 at 11:27
This link might help you https://docs.python.org/2/library/zipfile.html — Ankit Kumar Namdeo, Feb 21 '17 at 14:57

score -1 · Answer 1 · answered Jul 06 '17 at 20:23

-1

Here you want to read all csv files inside the zip files recursively.

val files = sc.CSVFiles("file://path/to/files/*.zip")
files.flatMap({case (name, content) =>
  unzip(content)
})

def unzip(content: String): List[String] = {
  ...
}

answered Jul 06 '17 at 20:23

Ramineni Ravi Teja

3,568
26
37

How to read contents of a CSV file inside zip file using spark (python)

1 Answers1