How to debug C compiler errors in Python? (malloc error)

Question

I am doing web development using Cherrypy in Python.

I had a working web page that did not have errors, but after I started using Mako for the front-end codes to parametrize it, the following error message pops up.

Python quit unexpectedly while using the libmysqlclient.18.dylib plug-in.

also it's throwing the following error at the console.

127.0.0.1 - - [09/Apr/2014:11:20:00] "GET /submit_data?idx=2 HTTP/1.1" 200 5990 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36"
python(375,0x103980000) malloc: *** error for object 0x7fb5a4061000: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

It seems like an error from C compiler, and maybe due to MySQL, but I cannot figure out what is wrong.

What can I do? I am guessing maybe I am using MySQL in a wrong way, I am attaching the Python code snippet in which I make connections to MySQL databases and use it.

dbdict = {}
for name in params["dashboards"]:
    for chart in params["dashboards"][name]:
        dbdict[chart["dbname"]] = None

def connect(thread_index): 
    for name in dbdict:
        db = params["db"][name]
        dbdict[name] = MySQLdb.connect(db["host"], db["user"], db["password"], db["dbname"])

cherrypy.engine.subscribe('start_thread', connect)

class AjaxApp(object):
    @cherrypy.expose
    @cherrypy.tools.mako(filename="index.html", directories=MEDIA_DIR)
    def index(name=None):
        return {'size': len(params["dashboards"]["first"])}

    @cherrypy.expose 
    def submit_data(self, idx):
        idx = int(idx)
        chart = params["dashboards"]["first"][idx]
        # Sample page that displays the number of records in "table" 
        # Open a cursor, using the DB connection for the current thread 
        c = dbdict[chart["dbname"]].cursor() 
        print 'query being executed......'
        c.execute(chart["command"])
        print 'query being fetched......'
        res = c.fetchall()
        c.close()

        # construct a JSON object from query result
        jsres = []
        jsres.append(chart["selected"])
        q_result = []
        for x in res:
            tmp_arr = []
            for i in x:
                if type(i) is datetime.datetime:
                    tmp_arr.append(str(i))
                else:
                    tmp_arr.append(i)
            q_result.append(tmp_arr)
        jsres.append(q_result)

        return json.dumps(jsres)

Here I am connecting to all dbs that are used, and putting them in a python dictionary, and whenever I am running a query command, I look up the corresponding db object, and make queries using it.

Now my connect function looks like this

def connect(thread_index): 
    for name in dbdict:
        print name
        db = params["db"][name]
        cherrypy.thread_data.db = MySQLdb.connect(db["host"], db["user"], db["password"], db["dbname"])
        dbdict[name] = cherrypy.thread_data.db

I am having the same error.

univerio · Accepted Answer · 2014-04-09T21:54:04.423

2

It looks like you're sharing a single connection between multiple threads (you're creating a single connection for each db in your connect() function). This may cause unexpected synchronization issues, especially if a C library is not written to handle it. Try attaching your connection objects to cherrypy.thread_data as shown here.

Here's how I would write it (untested):

def connect(thread_index):
    cherrypy.thread_data.dbdict = dbdict = {}
    for name in params["dashboards"]:
        for chart in params["dashboards"][name]:
            dbdict[chart["dbname"]] = None
    for name in dbdict
        db = params["db"][name]
        dbdict[name] = MySQLdb.connect(db["host"], db["user"], db["password"], db["dbname"])

And then in submit_data():

@cherrypy.expose
def submit_data(self, idx):
    ...
    c = cherrypy.thread_data.dbdict[chart["dbname"]].cursor()

edited Apr 09 '14 at 21:54

answered Apr 09 '14 at 19:12

univerio

19,548
3
66
68

You're right. I used to do it that way, but for some reason I do not remember, I removed the `cherrypy.thread_data` from all of db objects. – Eric Apr 09 '14 at 21:32
Now I am adding the db objects in the way you suggested, but I am having the same problem. Could you see if I am doing it the right way? – Eric Apr 09 '14 at 21:39
I added my revised code in the original question text – Eric Apr 09 '14 at 21:41
Nope, that's not correct. You need to access your db connection in your `submit_data()` function from `cherrypy.thread_data` as well. `cherrypy.thread_data` is what's called a *thread-local* variable, meaning it holds a different value depending on which thread you access it from. What you're doing is you're correctly attaching a different connection to `cherrypy.thread_data` for each thread, but you're still accessing from `dbdict`, which holds the value of the connection that corresponds to the newest thread cherrypy created, giving you the error. – univerio Apr 09 '14 at 21:49
Thanks, but still the same error... Before I started using a template language called `Mako`, I did not have this issue, and the template language has nothing to do with MySQL query. Could you take a look at my entire code and see if there is anything I am doing wrong? It's in https://www.dropbox.com/sh/ik43di5xlyazuig/VoPUMAh-4s – Eric Apr 09 '14 at 22:34
c = cherrypy.thread_data.dbdict[chart["dbname"]].cursor() AttributeError: '_ThreadData' object has no attribute 'dbdict' – Eric Apr 09 '14 at 22:56
@user2418202 You need to put `cherrypy.thread_data.dbdict = dbdict = {}` etc. inside your `connect()` function. – univerio Apr 09 '14 at 22:57
`dbdict` is a global variable. If I put it inside `connect()` function, I won't be able to access it from `submit_data()` function etc. – Eric Apr 09 '14 at 23:04
@user2418202 That's why you need to access it with `cherrypy.thread_data.dbdict`. Your entire problem is due to `dbdict` being a global. By putting a different dbdict for each thread on `cherrypy.thread_data` you avoid sharing the connection among your threads. – univerio Apr 09 '14 at 23:35
Works like a charm! Thank you. (still not 100% sure how this works) – Eric Apr 10 '14 at 17:21
1

@user2418202 I will attempt to explain again. I think it is very important that you understand *why* it works so you know how to solve similar problems in the future. If you don't understand any particular part, feel free to ask for clarification. First, note that cherrypy is multi-threaded. This means that, when the cherrypy server starts, it spawns n concurrent threads of execution, in order to handle concurrent client requests. What this means for you, the developer, is that whatever you do inside `submit_data()` etc. is not the same thread in which the module loads. (cont.) – univerio Apr 10 '14 at 18:20
1

(cont.) Thus, if you have a global variable `dbdict`, when you access it inside `submit_data()`, you are possibly accessing it under many different threads. This is where you get the weird `malloc` error, since you were attempting to access a single connection from multiple threads without some form of concurrency control (e.g. locks). Now, how do you solve this problem? Simply don't allow concurrent access to the connections. The solution we've chosen here is to attach `dbdict` to `cherrypy.thread_data`. (cont.) – univerio Apr 10 '14 at 18:25
1

(cont.) This forbids you from concurrent access because `cherrpy.thread_data` is a *thread-local* variable, meaning each thread has a different `cherrypy.thread_data` variable, and cannot access each other's `cherrypy.thread_data`. The magic happens when you subscribe to the `start_thread` event with the `connect()` function. By doing this, cherrypy runs `connect()` *under each different thread that it spawns*, so you have a private copy of `dbdict` under `cherrypy.thread_data` for each thread you could possibly run `submit_data()` in. This completely avoids concurrent access of connections. – univerio Apr 10 '14 at 18:30

score 1 · Answer 2 · edited May 23 '17 at 11:57

python(375,0x103980000) malloc: * error for object 0x7fb5a4061000: pointer being freed was not allocated * set a breakpoint in malloc_error_break to debug

Yes, this is C program error which you have mentioned could be due to MYSQL. The error indicates that somewhere in your program, you are passing the address which is not allocated and recieved by malloc/calloc/realloc dynamic call. free() expects the memory which is received by malloc/calloc/realloc. While free() call, memory manager does some sanity check on the input address and while doing so if it does not pass,it throws such error message and abort the program.

In your program, due to some reason incorrect memory is passed to free(). Now it would be bit difficult to understand the possible reason from your PHP/MYSQL code. I would suggest you that you may want to attach your program in dynamic tool and find out which is causing this problem. You may want to refer my previous post on these article.

https://stackoverflow.com/a/22658693/2724703

How to debug C compiler errors in Python? (malloc error)

2 Answers2

Linked