0

Below code is used to read csv file and write the output to a csv file. This code worked perfectly fine. But when the csv file size (number of rows) got increased this gives an error. I tried changing the Xms to 512m, Xmx to 2024m and XX:ReservedCodeCacheSize to 480m. But still getting the memory error.

Traceback (most recent call last):
File "/root/PycharmProjects/AppAct/statfile.py", line 5, in <module>
   df = df.astype(float)
File "pandas/core/generic.py", line 5691, in astype
   **kwargs)
File "pandas/core/internals/managers.py", line 531, in astype
   return self.apply('astype', dtype=dtype, **kwargs)
File "pandas/core/internals/managers.py", line 402, in apply
   bm._consolidate_inplace()
File "pandas/core/internals/managers.py", line 929, in _consolidate_inplace
   self.blocks = tuple(_consolidate(self.blocks))
File "pandas/core/internals/managers.py", line 1899, in _consolidate
   _can_consolidate=_can_consolidate)
File "pandas/core/internals/blocks.py", line 3149, in _merge_blocks
   new_values = new_values[argsort]
MemoryError

import pandas as pd

all_df = pd.read_csv("/root/Desktop/Time-20ms/AllDataNew20ms.csv")
df = all_df.loc[:, all_df.columns != "activity"]
df = df.astype(float)
mask = (df != 0).any(axis=1)
df = df[mask]
recover_lines_of_activity_column = all_df["activity"][mask]
final_df = pd.concat([recover_lines_of_activity_column, df], axis=1)
final_df.to_csv("/root/Desktop/Dataset.csv", index=False)
AKX
  • 152,115
  • 15
  • 115
  • 172
user3535695
  • 75
  • 2
  • 12

1 Answers1

0

Changing your PyCharm memory limits (as -Xms and other JVM settings do) will have absolutely no effect on the Python interpreter that's actually running your Python code.

Plain and simple, you're running out of system memory when you're converting your entire dataframe to floats (df = df.astype(float)).

Beyond changing your code to do things more efficiently, you could add physical memory or enable swap.

(Also, surely you're using a 64-bit Python?)

One easy optimization would be to do less work copying and converting the data – pass in dtype=... directly to pd.read_csv(). See this answer, for example.

AKX
  • 152,115
  • 15
  • 115
  • 172
  • my work was based on this https://stackoverflow.com/questions/57037715/pandas-delete-rows-with-different-encoding-of-0s-in-python So if I dont covert dataframe to floats, how can I delete the rows having only 0 and 0.0 values? Please help. Thanks – user3535695 Oct 08 '19 at 12:38
  • @user3535695 Augmented my answer with one possible optimization. – AKX Oct 08 '19 at 12:57