5

We are developing a small c server application. The server application does some data processing and responds back to the client. To keep the data processing part configurable and flexible we decided to go for scripting and based on the availability of various ready modules we decided to go for Python. We are using the Python-C api to send/receive the data between c and python.

The Algorithm works something like this:-

  1. Server receives some data from client, this data is stored in a dictionary created in c. The dictionary is created using the api function PyDict_New(); from c. The input is stored as a key value pair in the dictionary using the api function PyDict_SetItemString();
  2. Next, we execute the python script PyRun_SimpleString(); passing the script as a parameter. This script makes use of the dictionary created in c. Please note, we make the dictionary created in c, accessible to the script using the methods PyImport_AddModule(); and PyModule_AddObject();
  3. We store the result of the data processing in the script as a key value pair in the same dictionary created above. The c code can then simply access the result variable(key-value pair) after the script has executed.

The problem The problem we are facing is in the case of concurrent requests coming in from different clients. When multiple requests come in from different clients we tend to object reference count exceptions. Please note, that for each request which comes in for a user, we create an independent dictionary for that user alone. To overcome this problem we encompassed the call to PyRun_SimpleString(); within PyEval_AcquireLock(); and PyEval_ReleaseLock();, but doing this has resulted in the script execution being a blocking call. So if a script is taking long time to execute, all the other users are also waiting for a response.

Could you please suggest the best possible approach or give pointers to where we are going wrong. Please ping me for more information.

Any help/guidance will be appreciated.

Some programmer dude
  • 400,186
  • 35
  • 402
  • 621
Will
  • 482
  • 1
  • 8
  • 21

3 Answers3

1

Perhaps you are missing one of the calls mentioned in this answer.

Community
  • 1
  • 1
Janne Karila
  • 24,266
  • 6
  • 53
  • 94
  • Thanks for the reference Jane. Incidentally, I had made calls to those functions, but it still will not work. – Will Feb 08 '12 at 10:08
1

You should probably read http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock Your problem is explained in the first paragraph.

When you acquire the GIL, do so around your direct manipulation of Python objects. The call to PyRun_SimpleString will handle the GIL internally and will give it up on long-running operations or just every X instructions. It WILL NOT be truly multi-threaded, however.

Edit:

You need to acquire the lock and you need to ensure that Python knows it's in a different thread state:

// acquire the lock and switch thread state
PyEval_AcquireLock();
PyThreadState_Swap(perThreadState);

// execute some python code
PyEval_SimpleString("print 123");

// clear the thread state and release the lock
PyThreadState_Swap(NULL);
PyEval_ReleaseLock();
Tom Whittock
  • 4,081
  • 19
  • 24
  • Hello Tom, thanks for the answer. Could you please elaborate to the fact that it will "NOT" be truly multi-threaded? Do you mean to say that the scripts can never get executed in parallel? – Will Feb 08 '12 at 10:10
  • There is a global interpreter lock - only one python bytecode instruction will execute at any one time, no matter how many threads there are. Long-running functions such as file open release the lock temporarily while they are executing, and the python interpreter periodically gives up the lock, but ultimately within one process python bytecodes are executed serially. – Tom Whittock Feb 08 '12 at 12:49
  • Thanks for that Tom. Appreciate your time for the responses. Please confirm this: If PyRun_SimpleString is executing in 2 (or maybe n) separate "c" threads, it takes care of the GIL itself and hence I do not need to block or lock before making a call to PyRun_SimpleString from any thread. – Will Feb 08 '12 at 14:39
1

I suggest you investigate the multiprocessing module.

cdarke
  • 42,728
  • 8
  • 80
  • 84