Getting MemoryError converting bytes to str

Question

I am downloading data in CSV format via a third party API, then saving locally. The code I had was working fine, until the data increased from 80MB to 280MB. I am getting a MemoryError message converting the API response content from bytes to str.

Along with the MemoryError, I feel I'm doing too many steps, and hoping it can be simplified.

Code to get the bytes data via API

response = session.get('https://example.eu/api/2.0/get/', headers=headers, params=params)
file_data = response.content

The response.content is bytes type and has 4 rows of label text. The actual CSV data starts on row 5. I'm using pandas to read the contents & skip rows, then writing to a file.

file_str = file_data.decode('utf-8')
file_io = StringIO(file_str)
df=pd.read_csv(file_io,skiprows=4)
df.to_csv(filename, index=False,date_format='yyyy-mm-dd hh:mm:ss')

I also tried the below, also getting MemoryError

file_str = str(file_data,'utf-8')

Check this out: https://stackoverflow.com/questions/35572348/how-to-write-an-huge-bytearray-to-file-progressively-without-hitting-memoryerror — Mayank Porwal, May 26 '20 at 09:12
This might also be relevant: https://stackoverflow.com/a/25962187/8805293 — Hampus Larsson, May 26 '20 at 09:24
I can't see how chunking will help, unless I write the bytes data to file, then read back again, which I want to avoid. — BenOnline, May 26 '20 at 09:49

Getting MemoryError converting bytes to str

0 Answers0