pandas loading compressed csv from html response directly

Question

Currently I have a way to save the file from the html response:

_header={}
_header['Prefer']='respond-async'
_header['Content-Type']='application/json; odata.metadata=minimal'
_header['Accept-Charset']='UTF-8'
_header['Authorization']='Token'+token

resp=get(_getResultURL,headers=_header,stream=True)
 with open(outputfilepath, 'wb') as f:
     f.write(resp.raw.read())

This writes a compressed csv to outputfilepath adn can be loaded like this:

df=pd.read_csv(outputfilepath,compression='gzip')

I tried loading it directly in two ways, both unsuccessful:

pd.read_csv(resp.raw.read(),compression='gzip')
pd.read_csv(io.BytesIO(resp.raw.read()),compression='gzip')

How can this be done in one go?

Have you tried this ? https://stackoverflow.com/a/32400969/19603419 — Timeless, Sep 06 '22 at 16:29
@L'Artiste, thanks for the directions. It was not exactly the same use case, but I managed to get the data now using the directions in the link — EddyG, Sep 07 '22 at 11:35

score 0 · Accepted Answer · answered Sep 07 '22 at 11:37

0

The key was to get the content of the response instead of reading the raw output

df = pd.read_csv(io.BytesIO(resp.content))

answered Sep 07 '22 at 11:37

EddyG

675
4
13

pandas loading compressed csv from html response directly

1 Answers1