Stack Overflow
Stack Overflow

Questions tagged [feather]

Feather is a file format for storing data frames. It allows fast data exchange between Python and R.

147 questions
196
votes
2 answers

What are the differences between feather and parquet?

Both are columnar (disk-)storage formats for use in data analysis systems. Both are integrated within Apache Arrow (pyarrow package for python) and are designed to correspond with Arrow as a columnar in-memory analytics layer. How do both formats…
Darkonaut
  • 20,186
  • 7
  • 54
  • 65
22
votes
2 answers

R cannot read Python Pandas dataframe saved in feather format

I have a pandas dataframe dfwin. And I save it to feather format hoping I can read it in R. But R always throws the error "Error in openFeather(path): Invalid: Not a feather…
L.Yang
  • 553
  • 1
  • 6
  • 12
21
votes
1 answer

Feather format for long term storage since the release of apache arrow 1.0.1

As I'm given to understand due to the search of issues in the Feather Github, as well as questions in stackoverflow such as What are the differences between feather and parquet?, the Feather format was not recommended as long term storage due to…
Serelia
  • 213
  • 2
  • 6
19
votes
2 answers

CSV to Feather in Pandas with slicing Rows

I am processing a huge dataset (50 million rows) in CSV. I am trying to slice it and save it as Feather Format in order to save some memory while loading the feather format later. As a workaround, I loaded the data in chunks as CSV file and later…
MKJ
  • 499
  • 1
  • 7
  • 20
17
votes
2 answers

OverflowError while saving large Pandas df to hdf

I have a large Pandas dataframe (~15GB, 83m rows) that I am interested in saving as an h5 (or feather) file. One column contains long ID strings of numbers, which should have string/object type. But even when I ensure that pandas parses all columns…
Josh Friedlander
  • 10,870
  • 5
  • 35
  • 75
17
votes
2 answers

Can you append to a .feather format?

Is there a way to append to a .feather format file using pd.to_feather? I am also curious if anyone knows some of the limitations in terms of max file size, and whether it is possible to query for some specific data when you read a .feather file…
trench
  • 5,075
  • 12
  • 50
  • 80
13
votes
4 answers

How to read feather/arrow file natively?

I have feather format file sales.feather that I am using for exchanging data between python and R. In R I use the following command: df = arrow::read_feather("sales.feather", as_data_frame=TRUE) In python I used that: df =…
jangorecki
  • 16,384
  • 4
  • 79
  • 160
12
votes
1 answer

Arrow IPC vs Feather

What is the difference between Arrow IPC and Feather? The official Arrow documentation says: Version 2 (V2), the default version, which is exactly represented as the Arrow IPC file format on disk. V2 files support storing all Arrow data types as…
tsorn
  • 3,365
  • 1
  • 29
  • 48
11
votes
2 answers

pandas.read_feather got an unexpected argument nthreads

I tried saving a dataframe to feather format but while loading back I got the error os.makedirs('tmp', exist_ok=True) df_hist.to_feather('tmp/historical-raw') Here's the loading back into the dataset df_hist=…
kramer
  • 849
  • 2
  • 10
  • 19
9
votes
1 answer

How to save file in Feather format\storage from Spark?

Is it possible to export data-frame from Apache Spark to feather (https://github.com/wesm/feather) file?
Dmitry Petrov
  • 1,490
  • 1
  • 19
  • 34
8
votes
1 answer

If the feather file format still relevant or is the community leaning towards other file formats for large file storage?

I'm exploring file storage format options for Python and stumbled on feather. I noticed the last release was back in 2017 and was concerned about its long term existence. Web searches are pulling back posts that all seem to stop around 2017.
cauthon
  • 161
  • 1
  • 10
8
votes
2 answers

What format to export pandas dataframe while retaining data types? Not CSV; Sqlite? Parquet?

My workflow typically involves loading some data, typically from CSV files, into a pandas dataframe, cleansing it, defining what the right data type for each column is, then exporting it to a SQL server. For those situations when a SQL server is not…
Pythonista anonymous
  • 8,140
  • 20
  • 70
  • 112
8
votes
3 answers

Error when trying to write DataFrame to feather. Does feather support list columns?

I'm working with both R and Python and I want to write one of my pandas DataFrames as a feather so I can work with it more easily in R. However, when I try to write it as a feather, I get the following error: ArrowInvalid: trying to convert NumPy…
Ben G
  • 4,148
  • 2
  • 22
  • 42
8
votes
4 answers

ArrowNotImplementedError: halffloat error on applying pandas.to_feather on a dataframe

I have a dataframe with columns of different datatypes including dates. No after doing some modifications, i want to save it a feather file so as to access it later. But i am getting the error on the following…
kramer
  • 849
  • 2
  • 10
  • 19
8
votes
1 answer

feather data storage library for python 'module' object has no attribute 'write_dataframe' error

I am getting the following error after attempting to read in an xlsx file, write it to a dataframe using feather, then read in that same dataframe using feather and display the results using df.head() import pandas as pd import feather v =…
yoshiserry
  • 20,175
  • 35
  • 77
  • 104
1
2 3
…
9 10