How to read gz compressed file by pyspark

Question

I have line data in .gz compressed format. I have to read it in pyspark Following is the code snippet

rdd = sc.textFile("data/label.gz").map(func)

But I could not read the above file successfully. How do I read gz compressed file. I have found a similar question here but my current version of spark is different that the version in that question. I expect there should be some built in function as in hadoop.

Possible duplicate of [Why are my \`binaryFiles\` empty when I collect them in pyspark?](http://stackoverflow.com/questions/38256631/why-are-my-binaryfiles-empty-when-i-collect-them-in-pyspark) — T. Gawęda, Mar 13 '17 at 11:03
It's not one to one duplicate, but you have necessary code snippet there :) — T. Gawęda, Mar 13 '17 at 11:04

score 18 · Accepted Answer · answered Mar 13 '17 at 11:05

Spark document clearly specify that you can read gz file automatically:

All of Spark’s file-based input methods, including textFile, support running on directories, compressed files, and wildcards as well. For example, you can use textFile("/my/directory"), textFile("/my/directory/.txt"), and textFile("/my/directory/.gz").

I'd suggest running the following command, and see the result:

rdd = sc.textFile("data/label.gz")

print rdd.take(10)

Assuming that spark finds the the file data/label.gz, it will print the 10 rows from the file.

Note, that the default location for a file like data/label.gz will be in the hdfs folder of the spark-user. Is it there?

score 8 · Answer 2 · edited May 29 '19 at 13:26

8

You can load compressed files directly into dataframes through the spark instance, you just need to specify the compression in the path:

df = spark.read.csv("filepath/part-000.csv.gz")

You can also optionally specify if a header present or if schema needs applying too

df = spark.read.csv("filepath/part-000.csv.gz", header=True, schema=schema).

edited May 29 '19 at 13:26

jpeg

2,372
4
18
31

answered May 29 '19 at 12:49

AJR

177
2
7

score 2 · Answer 3 · edited May 23 '17 at 12:32

2

You didn't write the error message you got, but it's probably not going well for you because gzipped files are not splittable. You need to use a splittable compression codec, like bzip2.

edited May 23 '17 at 12:32

Community

1
1

answered Mar 13 '17 at 11:04

Tim

3,675
12
25

if you run `sc.textFile("data/label.gz").count()` it shows 0? – Tim Mar 13 '17 at 11:16

How to read gz compressed file by pyspark

3 Answers3

Linked