I'd like to use local feather-files as a sources in Intake. Is the plugin for feather/arrow not yet existing or am I missing something?
Asked
Active
Viewed 144 times
1 Answers
2
You are right, there is not currently a feather or arrow (i.e., framed buffers) driver for Intake, as far as I know. Neither is supported by Dask either.
Given that pandas does explicitly support feather, it would be easy to build a driver for Intake, supporting multiple remote files, and these could even be loaded in parallel with Dask, without adding code to dask.
However, first I'd like to ask: why not parquet? That seems to be the standard format, at least partly because its reach goes well beyond python/arrow. That format is supported by Intake and any other tabular data engine.
mdurant
- 27,272
- 5
- 45
- 74
-
Main reason is that pandas supports feather/arrow and so we use feather-files as our storage-format. – bowlby Mar 03 '21 at 19:39
-
Pandas supports parquet too – mdurant Mar 03 '21 at 21:58