Write pandas DataFrame to HDF in memory buffer

Question

I want to get a dataframe as hdf in memory. The code below results in "AttributeError: '_io.BytesIO' object has no attribute 'put'". I am using python 3.5 and pandas 0.17

import pandas as pd
import numpy as np
import io

df = pd.DataFrame(np.arange(8).reshape(-1, 2), columns=['a', 'b'])
buf = io.BytesIO()
df.to_hdf(buf, 'some_key')

Update: As UpSampler pointed out "path_or_buf" cannot be an io stream (which I find confusing since buf usually can be an io stream, see to_csv). Other than writing to disk and reading it back in, can I get a dataframe as hdf in memory?

Out of curiosity - why would you want to do that? – MaxU - stand with Ukraine Jan 06 '17 at 15:15 — MaxU - stand with Ukraine, Jan 06 '17 at 15:15
I just came to the same point, did you manage to solve it? – jlandercy Sep 25 '17 at 15:14 — jlandercy, Sep 25 '17 at 15:14

score 2 · Answer 1 · answered Jan 06 '17 at 14:22

2

Your first argument to df.to_hdf() has to be a "path (string) or HDFStore object" not an io stream. Documentation: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.to_hdf.html

answered Jan 06 '17 at 14:22

UpSampler

399
2
6

Ahh, didn't see that. to_csv has a similar argument path_or_buf where buf can be an io stream, which is why I got confused. – user2133814 Jan 06 '17 at 14:25
CSV is a a bulit-in module and HDF is external. I don't know what pandas is using for HDF file access, could be pytables or pyhdf... – UpSampler Jan 07 '17 at 01:58

score -1 · Answer 2 · answered Jan 06 '17 at 14:15

-1

just try this

df = pd.DataFrame(np.arange(8).reshape(-1, 2), columns=['a', 'b'])
df.to_hdf(path_or_buf='path\to\your\file')

refer pandas.DataFrame.to_hdf

answered Jan 06 '17 at 14:15

Shijo

9,313
3
19
31

1

I want it in memory, not on disk, and would prefer not writing to disk and then reading it back in – user2133814 Jan 06 '17 at 14:21

Write pandas DataFrame to HDF in memory buffer

2 Answers2