2

I have a multi threaded app in python, wherein I create multiple producer threads and they extract the data from DB. Data is extracted in chunks. So the part where a thread creates sql statement with limit values is kept within lock. And to let threads execute queries simultaneously, query() function is kept outside the lock. Then the result fetching part is again kept under the lock. Below is the code snippet:

with UserAgent.lock:
    sqlGeoTarget = "call sp_ax_ari_select_user_agent_list('0'," + str(self.chunkStart) + "," + str(self.chunkSize) + ",1);"
    self.chunkStart += self.chunkSize

self.dbObj.query(sqlGeoTarget)
print "query executed. Processing data now..."+sqlGeoTarget

with UserAgent.lock:
    result = self.dbObj.fetchAll()
    self.dbObj.dbCursor.close()

But this code generates fatal error segmentation fault (core dumped). Because if I put all the code under lock, it executes fine. I explicitly close the cursor after fetching the data, it is reopened when query() function fired again.

This code is inside a class named UserAgent and it's a shared resource for a class named Producer. Thus, database object is shared. So the problem area 99% must be that as the db object is shared hitting query simultaneously and closing cursor then must be messing up with result set. But then how to solve this problem and achieve concurrent db query execution?

Shades88
  • 7,934
  • 22
  • 88
  • 130

1 Answers1

7

Do not reuse connections across threads. Create a new connection for each thread instead.

From the MySQLdb User Guide:

The MySQL protocol can not handle multiple threads using the same connection at once. Some earlier versions of MySQLdb utilized locking to achieve a threadsafety of 2. While this is not terribly hard to accomplish using the standard Cursor class (which uses mysql_store_result()), it is complicated by SSCursor (which uses mysql_use_result(); with the latter you must ensure all the rows have been read before another query can be executed. It is further complicated by the addition of transactions, since transactions start when a cursor execute a query, but end when COMMIT or ROLLBACK is executed by the Connection object. Two threads simply cannot share a connection while a transaction is in progress, in addition to not being able to share it during query execution. This excessively complicated the code to the point where it just isn't worth it.

The general upshot of this is: Don't share connections between threads. It's really not worth your effort or mine, and in the end, will probably hurt performance, since the MySQL server runs a separate thread for each connection. You can certainly do things like cache connections in a pool, and give those connections to one thread at a time. If you let two threads use a connection simultaneously, the MySQL client library will probably upchuck and die. You have been warned.

Emphasis mine.

Use thread local storage or a dedicated connection pooling library instead.

Community
  • 1
  • 1
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • ok, that's some really helpful info. now I have a design related question. this class UserAgent is supposed to be a self contained class, whose object once instantiated, creates a db connection and when `getData()` is called, it will return the data to caller. Now if I have to let threads access this, I need to have separate db connections for each thread. So how can I achieve that? Shall I create db connection in producer and pass it to `getData()`. that sounds a bit strange, what would you suggest? – Shades88 Nov 28 '12 at 12:12
  • 1
    Use a thread local: [Thread local storage in Python](http://stackoverflow.com/q/1408171) – Martijn Pieters Nov 28 '12 at 12:13
  • Alternatively, use a dedicated [connection pool library](http://code.google.com/p/pysqlpool/). – Martijn Pieters Nov 28 '12 at 12:14