How to read multiple .parquet files from multiple directories into single pandas dataframe?

Question

I need to read parquet files from multiple directories.

for example,

 Dir---
          |
           ----dir1---
                      |
                       .parquet
                       .parquet
          |
           ----dir2---
                      |
                       .parquet
                       .parquet
                       .parquet

Is there a way to read these file to single pandas data frame?

note: All of parquet files was generated using pyspark.

jezrael · Accepted Answer · 2020-10-06T05:06:38.257

7

Use read_parquet in list comprehension and concat with all files generated by glob with ** (python 3.5+):

import pandas as pd
import glob

files = glob.glob('Dir/**/*.parquet')
df = pd.concat([pd.read_parquet(fp) for fp in files])

edited Oct 06 '20 at 05:06

answered Jan 15 '20 at 04:58

jezrael

822,522
95
1,334
1,252

I got this error ```RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED']``` – Ahmad Senousi Jan 15 '20 at 05:07
1

@AhmadSuliman - Check [this](https://stackoverflow.com/questions/50800748/decompression-snappy-not-available-with-fastparquet) – jezrael Jan 15 '20 at 05:12
Will it work if each directly has over 30 parquet files of about 75MB each? – Avantika Banerjee Sep 21 '22 at 15:46

How to read multiple .parquet files from multiple directories into single pandas dataframe?

1 Answers1