0

I am converting a dictionary to pandas object with to_csv. I have both way of doing this

1 - by writing file in disk(with open statement)

2 - by writing in memory (StringIO,BytesIO)

I have used it in both way creating file in disk and using StringIO to convert to pandas object. I tried to read comparisons between these three, but bit confused which one is faster so i can use it in production to process tons of data.

Chidananda Nayak
  • 1,161
  • 1
  • 13
  • 45

2 Answers2

2

Writing and reading from memory is fast. But keep in mind that you have tons of data. So storing all that in-memory might take up all your memory and might make the system slow or might throw errors due to Out of Memory. So, analyze and understand which all data to be put in memory and which all to be written to files.

Nandu Raj
  • 2,072
  • 9
  • 20
  • i think it would better to use file object in disk than memory. Can i use tempfile module for that? – Chidananda Nayak Apr 09 '20 at 04:55
  • Yea that's the usual practice when we have lots of data. We would store it in files, but we make use of caches. The data which frequently access will be stored in memory (specifically cache), and the others will be read from disk or DB on request. I am not sure on the tempfile module. – Nandu Raj Apr 09 '20 at 04:58
2

In general - writing to RAM (memory) will be faster.

However, you might want to use Iterators (saving memory using iterators) if you have too much data, because your machine might will run out-of-memory, or just will write a lot to your SWAP file (in short - that's an "extension" of your RAM in your hard drive, you can read about it here), which will hurt your performance, a lot.

For benchmarking, if your code is pretty simple - I would recommenced using timeit, but there are even better resources for that, such as this one, from scipy

user3467955
  • 561
  • 5
  • 11