efficient way to wait for job completion : python and drmaa

Question

I wanted to ask about "wait" feature in drmaa API I am using through Python. Does it do constant qstat's ( if we are running it on SGE) to check whether a program has finished execution.

Our admin want us to avoid any constant qstat's as it slows down the performance due to extra load on scheduler.

In general wat would be an efficient way to check for job status through DRMAA without overboarding the scheduler.

Thanks! -Abhi

Answer 1

From an DRMAA API point of view, there is no better way. The API simply reflects what you are able to do in a shell script with the default command-line tools.

The problem lies in the implementation strategy of your particular DRMAA library, since SGE offers better ways than constant polling to get job status updates.You therefore have the following options:

• Fix the DRMAA implementation you are using to rely on some direct communication with the scheduler. One source of information for the wire protocol could be the Open Grid Scheduler project.
• Wait until Univa (or others) deliver a DRMAAv2 implementation for their Grid Engine product. This version of the API supports library callbacks on job status changes (http://ogf.org/documents/GFD.198.pdf), which seems to be exactly what you want.