0

I wrote an AWS lambda function which that reads a parquet file which has 2 columns whose type is boolean. When I exclude those two boolean columns named: "iscritical" and "iscyclic" from the input columns list the wr.s3.read_parquet() operation success.

code snippet:

valid_cols = [col for col in list(parquet_file_cols_metadata.keys()) if col != "iscritical" and col != "iscyclic"]
stage_file_full_data_df = wr.s3.read _parquet(
            path=stage_file,
            ignore_empty=True,
            use_threads=True,
            columns=valid_cols)

When I am trying to read the entire data (inlcude the boolean types columns) the wr.s3.read_parquet() operation fails with exception: "Unknown encoding type"

code snippet:


stage_file_full_data_df = wr.s3.read _parquet(
            path=stage_file,
            ignore_empty=True,
            use_threads=True)

My aim is to read the entire data with the boolean columns.

What I am asking is if someone encountered OR know why wr.s3.read_parquet() cannot handle boolean columns? Thanks.

0x26res
  • 11,925
  • 11
  • 54
  • 108

0 Answers0