Is spark able to read only column values satisfying some condition from parquet file?

Question

I have a code

val count = spark.read.parquet("data.parquet").select("foo").where("foo > 3").count

I'm interested if spark is able to push down filter somehow and read from parquet file only values satisfying where condition. Can we avoid full scan in this case?

Please refer https://stackoverflow.com/questions/50129411/why-is-predicate-pushdown-not-used-in-typed-dataset-api-vs-untyped-dataframe-ap — dassum, Aug 26 '19 at 17:22

score 2 · Accepted Answer · answered Aug 26 '19 at 19:45

2

Short answer is yes, in this case, but not all cases.

You can try .explain and see for yourself.

This is an excellent reference document freely available on the Internet that I learnt a few things from in the past: https://db-blog.web.cern.ch/blog/luca-canali/2017-06-diving-spark-and-parquet-workloads-example

answered Aug 26 '19 at 19:45

thebluephantom

16,458
8
40
83

@thebluephantom..Thanks for sharing such a wonderful link. I am new to Parquet and it's helping me a lot – vikrant rana Aug 27 '19 at 05:10
1

I wish if I could, I am not the owner of this question. hence just upvoted :-) – vikrant rana Aug 27 '19 at 05:51
Yes indeed, it's well thought out and set out. – thebluephantom Aug 27 '19 at 11:57

Is spark able to read only column values satisfying some condition from parquet file?

1 Answers1