0

I am programming on Ubuntu, with Python 2.7.3.

I am using CFFI to populate a Python list with values coming from some C code.
That list is quite big : around 71 000 characters long when printed.

The C code is using many libraries. Hence, the following code is only here for a better understanding of what is happening.

datas_list = []
for i in range( 0, x ):
    c_pDataStructure = ffi.new( "c_DataStructure[]", 1 )    // Create a pointer to the data structure
    c.SomeCFunction( c_pDataStructure )    // Populate the data structure
    datas_list.append( c.GetSomeInfo( c_pDataStructure ) )    // Get some info from the data structure
    c.FreeDataStructure( c_pDataStructure )    // Release dynamically allocated memory

The program runs well using Wingware IDE but ends with a glibc error (*** glibc detected *** python: free(): invalid next size (fast): 0x0000000003b0b080 ***) when started from the command line, right before: 

c_pDataStructure = ffi.new( "c_Datastructure[]", 1)

After reading wim's answer, I checked if both the IDE and the command line were running the code using the same interpreter — they are (/usr/bin/python).

EDIT (valgrind report):

==5089== Process terminating with default action of signal 11 (SIGSEGV)  
==5089==  General Protection Fault  
==5089==    at 0x54FBB0: PyObject_Malloc (in /usr/bin/python2.7)  
==5089==    by 0x10B30625: allocate_owning_object (_cffi_backend.c:2972)  
==5089==    by 0x10B40EE8: allocate_with_allocator.constprop.84 (_cffi_backend.c:3032)  
==5089==    by 0x10B41010: direct_newp (_cffi_backend.c:3153)  
==5089==    by 0x10B4138C: b_newp (_cffi_backend.c:3177)  
==5089==    by 0x4F95A4: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x5008C1: PyEval_EvalCodeEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9AB7: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)  
==5089==    by 0x4F9D01: PyEval_EvalFrameEx (in /usr/bin/python2.7)

EDIT:
Here are some more information about the C data structure. This is how it looks : 

typedef struct _STRUCT3{
    some int, char*
}STRUCT3, *PSTRUCT3;

typedef struct _STRUCT2{
    some int
    PSTRUCT3 pStruct3;
}STRUCT3, *PSTRUCT3;

typedef struct _STRUCT1{
    some int, char*
    PSTRUCT2 pStruct2;
}STRUCT1, *PSTRUCT1;

I made a little C program to allocate / deallocate a full C structure and valgrind did not find any memory leak.

Questions:

  • What does the above valgrind report exactly mean?
  • What could be the differences between running the program from the IDE and from the command line?
    Note: The IDE uses the Python argument -u (unbuffered) to run the program, but adding it to the command line makes no difference.
  • As I deallocate the structure on my own, is Python's garbage collector acting? Should I use ffi.gc( c_pDataStructure, c.FreeDataStructure ) instead?
Community
  • 1
  • 1
DRz
  • 1,078
  • 1
  • 11
  • 29
  • `FreeDataStructure(p)` should not free the pointer ``p`` itself, but only any dynamically-allocated stuff inside. I guess this is the case? – Armin Rigo Mar 30 '16 at 13:11
  • Yes, FreeDataStructure frees everything the populating function dynamically-allocates – DRz Mar 30 '16 at 13:16
  • The program crashes after a data structure with an optional member is allocated for the 33rd time. Thus, I am suspecting that optional value not to be freed properly. Is Python (or CFFI or C?) shitfting on a "32 memory address range" before overwritting previously used memory address? Then, if Wingware runs the program on a 64 bits Python and the command line on a 32 bits Python, it could explain why the latter ends in a glibc error... I am investigating this – DRz Mar 30 '16 at 14:52
  • Investigation result: Both the command line and Wingware are running the program on a 64 bits Python. – DRz Mar 30 '16 at 15:03
  • The code present here is correct as far as I can tell. The problem must be elsewhere. The best way to debug is if we can reproduce the problem ourselves. – Armin Rigo Mar 30 '16 at 18:56
  • Unfortunately, I cannot put the whole code on stack overflow. The part where the structure is instantiated is itself around 1 000 lines. I have updated the question with more details and more accurate questions. – DRz Mar 30 '16 at 21:09

2 Answers2

0

I found how to fix my issues:

I used ffi.gc(cdata, destructor) to create the structure. My Python code now looks like: 

data_list = []
for i in range( 0, x ):  
    # Create a pointer to the data structure and tell the garbage collector how to destroy it
    gc_c_pDataStructure = ffi.gc( c.CreateDataStructure(), c.FreeDataStructure )
    c.SomeCFunction( gc_c_pDataStructure )  # Populate the data structure
    datas_list.append( c.GetSomeInfo( gc_c_pDataStructure ) ) # Store some data

Here are some links related to ffi.gc():

And here is the C function to create the data structure (according to the structure example from the question): 

 PSTRUCT1 CreateDataStructure()
 {
     PSTRUCT1 pStruct1 = ( PSTRUCT1 ) malloc( sizeof( STRUCT1 ) );
     _SetDummyValues( pStruct1 );

     return pStruct1;
 }

As you can see, I had to create the function void _SetDummyValues( PSTRUCT1 pStruct1 ). That function sets the given structure pointers to NULL.

Community
  • 1
  • 1
DRz
  • 1,078
  • 1
  • 11
  • 29
  • 1
    Note a difference: in the original code, `c.FreeDataStructure(p)` should not call `free(p)` because that is done automatically after `p=ffi.new()`. In the modified code here, `c.FreeDataStructure(p)` must call `free(p)` corresponding to `c.CreateDataStructure()` calling `malloc()`. If it doesn't, you have a leak. – Armin Rigo Apr 29 '16 at 06:34
  • 1
    On the other hand, if it does call `free(p)` and always did, then we just found out the original reason for the crash---double frees! – Armin Rigo Apr 29 '16 at 06:38
  • I remember I had to delete `free(p)` before using `ffi.gc()` because that was indeed causing a double free. And you are right, my program was then leaking because I forgot to add that line again. Thanks! – DRz Apr 29 '16 at 14:38
  • The crash was probably a mix of both. I was making "keepalive" variables for `pStruct2` and `pStruct3`. These keepalive variables were instantiated using `ffi.new()` and freed during the function `c.FreeDataStructure(p)`. Thanks again! :) – DRz Apr 29 '16 at 14:39
0

This may have also been related to this bug which Coverity found in some of our CFFI-generated code:

  x0 = (void *)alloca((size_t)datasize);
  ...
  { 
      free(x0); 
  }

As you can see, it's calling free on stack-allocated memory.

Abhijeet
  • 4,069
  • 1
  • 22
  • 38
Anton
  • 1