7

Got this error on an ubuntu server (100GB RAM) in anacondas py3.6 installation after it opened multiple files this way in a for loop (7 files * 4GB):

temp_df = pd.read_csv(datafolder + str(file), encoding="ISO-8859-1", delimiter=';',low_memory=False)

this is the error, it appeared after i set

low_memory=False

Doesnt happen when low_memory = True

* Error in `python': free(): invalid pointer: 0x00007fc3c90dc98e *

Anyone ideas? Thanks

snakecharmerb
  • 47,570
  • 11
  • 100
  • 153
Mario L
  • 507
  • 1
  • 6
  • 15
  • 1
    According to this: https://github.com/ContinuumIO/topik/issues/71 it seems to be a problem of anaconda. I have however no idea how to fix it. – DerWeh Sep 12 '18 at 09:14
  • I am facing this issue as well when I was prompted to set low_memory=False. The csv file i am reading into pandas is 16 Gb though. – griffinleow Jan 21 '20 at 09:26
  • is your `num_workers=0`? (mine is and I am getting the error, where else could it be trying to open multiple files?) – Charlie Parker Jan 07 '21 at 22:05
  • related link: https://discuss.pytorch.org/t/segmentation-fault/23489/17 – Charlie Parker Jan 07 '21 at 22:10

1 Answers1

5

This issue is caused due to multiple threads performed parallelly when opening a file , this causes sometimes segmentation fault as well. As you said your file consist of 4 gb each so multiple threads which are executed parallelly to read up your file is causing the issue.

ScreenSHot

Invalid Pointer

Shivam Pandey
  • 86
  • 2
  • 8