0
I have a DataFrame having more than 50 columns whose memory usage is 17.1 mb after reading the csv by using below code I am able to Load

Data but when I am loading my data again i am getting out of memory error even after deleting few columns from DateFrame I am getting out of memory error ..I have tried several methods but nothing seemed to be working

code which I have used for reading csv file is :

df = pd.read_csv('ip.csv',encoding='cp1252')

i have also used low_memory='False' but nothing seemed to working
  • 1
    Do you mean `low_memory=False` ? Shouldn't it be boolan and not a string? – kaliiiiiiiii Jan 17 '23 at 14:20
  • yes, i have used low_memory=False but it doesn't seem to be working i am unable to load my dataframe again ? –  Jan 17 '23 at 14:26
  • A DataFrame is a rather complex container. It may use far more memory than you would expect. For example, a small integer value in a csv file could use 2 or 3 bytes, when the same integer value in a int64 column will use 8. And temporary data also use memory. When you hit a out of memory error, you have only 2 ways: buy and install more memory, or do not load everything in memory at the same time and only process *chunks*. That second way is the most efficient, unfortunately it is not straightforword and requires a careful analysis of what you are doing... – Serge Ballesta Jan 17 '23 at 14:32
  • @SergeBallesta my dataframe size is only 17 mb is this issue due to more than 50 columns in single dataframe ? –  Jan 17 '23 at 14:37
  • I have already used dataframes with more than 50 columns. What matters is what it contains and how you use it. What I meant in my previous comment, it that you are right if you want to understand why you have hitten an out of memory error. But it is unfortunately a hard task and to be able to help you, I would need to read your application source code. And it is not something that can be done on SO... – Serge Ballesta Jan 17 '23 at 16:20
  • ... I can only give the best practice advices: identify the data that is loaded in memory, how it is stored (format and datatypes) and evaluate the involved memory. Then identify if multiple or temporary copies are involved in any processing step. If necessary, [this other SO question](https://stackoverflow.com/q/938733/3545273) gives ways to track the memory used by the Python process (the accepted answer uses the psutils module and really looks nice). Good luck on your quest... – Serge Ballesta Jan 17 '23 at 16:28
  • @SergeBallesta thanks for taking your precious time out and giving precise reply to my query ..much appreciated –  Jan 17 '23 at 18:50

0 Answers0