Read csv from zipfile using pyspark

Question

I am trying to read csv data from a zip file, i know that .gz files are supported naturally in spark.read.csv(), but this is a zip file

How to open/stream .zip files through Spark? I check the above question and tried using it, but not sure how parse the RDD (a whole file of csv data represented as a ROW of text) into to a CSV dataframe

This is the code section used to extract data to RDD

import zipfile
import io

def zip_extract(x):
  file_path, content = row
  z_file = zipfile.ZipFile(io.BytesIO(content), "r")
  files = [i for i in z_file.namelist()]
  return z_file.open(files[0]).read()


zips = sc.binaryFiles("/path/to/some/zipfiles.zip")
data_rdd = zips.map(zip_extract)

Passing the rdd to spark.read.csv() is not giving the desired outcome

nope i ended up adding a preprocessing step to extract them before passing over to spark — Geethanadh, Dec 20 '21 at 20:29
Seems like we can do it this way https://stackoverflow.com/questions/28569788/how-to-open-stream-zip-files-through-spark — kensai, Dec 22 '21 at 05:41
I will post solution when implemented in next days... as we need it — kensai, Dec 23 '21 at 06:47

score 1 · Answer 1 · answered Jul 22 '19 at 21:38

1

Not sure I understand it correctly, if you already have a RDD, isn't it a simple call of data_rdd.toDF() to convert it to a DataFrame?

df=data_rdd.toDF()

answered Jul 22 '19 at 21:38

niuer

1,589
2
11
14

that would just convert the text data from RDD to DF, but the text data is csv, i want that to be parsed into a DF – Geethanadh Jul 22 '19 at 21:39
I don't believe that kind of thing is supported, for now, at least. If you don't want a DF, what are you actually trying to do? – ASH Aug 11 '19 at 12:08
i don't think you understood clearly, i DO want and DF, but i want the csv parsed into the DF, not just as a text data but as CSV columns (that works with spark.read.csv() for .gz files but not zip files) – Geethanadh Aug 16 '19 at 08:30

Read csv from zipfile using pyspark

1 Answers1