0

I am relatively new to python. I need a fast IO format with modest disk requirements. feather lacks compression, so parquet it is. I tried to install 

sudo pip3 install parquet

on macos 10.14.6, Python 3.7.4, pip 19.1.1.

the first set of errors relate to snappy. they can be fixed with brew install snappy and sudo pip3 install snappy. (Q: could pip3 not automatically figure out and satisfy the dependency, or at least instruct how to fix this?)

but then, I am out of luck:

...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c thriftpy/transport/cybase.c -o build/temp.macosx-10.14-x86_64-3.7/thriftpy/transport/cybase.o
thriftpy/transport/cybase.c:3127:24: error: no member named 'exc_type' in 'struct _ts'
    tmp_type = tstate->exc_type;
               ~~~~~~  ^
thriftpy/transport/cybase.c:3128:25: error: no member named 'exc_value' in 'struct _ts'; did you mean 'curexc_value'?
    tmp_value = tstate->exc_value;
                        ^~~~~~~~~
                        curexc_value
...

Quo Vadis?

ivo Welch
  • 2,427
  • 2
  • 23
  • 31

1 Answers1

2

The parquet package is unmaintained and outdated. To read and write Parquet files in Python, you should install pyarrow and use the pyarrow.parquet module.

When you have pyarrow and pandas installed you can also use pandas's Parquet functionality with pandas.read_parquet and pandas.DataFrame.to_parquet for convenience.

Uwe L. Korn
  • 8,080
  • 1
  • 30
  • 42
  • can it be removed? I am probably not the first to stumble over parquet and fastparquet and then wonder why nothing works... – ivo Welch Aug 29 '19 at 22:42
  • `fastparquet` should also work. Deleting packages from PyPI is though something highly discussable and probably better discussed on http://discuss.python.org – Uwe L. Korn Aug 30 '19 at 15:50
  • fastparquet won't work if you need 'snappy' because 'python-snappy' fails to build in MacOS – MikeB2019x Feb 26 '20 at 16:34