3

Does anyone have experience running Docker on Grid Engine / Sun Grid Engine / Son of Grid Engine and being able to monitor the resource used by the daemon? The issue is that when I qsub docker run ..., the actual process in the container is run by the docker daemon rather than the docker client which means the process trees are different.

Is there any way for SGE to track the resources of a process in a different tree (I would assume not)?

Another option would be to qsub a script that first launches the docker daemon and then runs the desired docker run command so that all processes are in the same process tree launched by SGE. This would require that sge have permissions to launch the daemon (root access) and could also result in multiple docker daemons running on the host.

Another problem with the alternative process tree is if the job submitted specified daemeon (-d). In this case the docker client would finish, but the docker daemon might still be running the container

Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120

1 Answers1

1

You are correct, when you use 'docker run ....' it talks to the daemon and creates a container from the image. This means that the process tree is 'not manageable by Grid Engine' because the Grid Engine execd (the daemon that launches the job) is not the parent of the tree. With Sun Grid Engine, Son of Grid Engine and Open Grid Scheduler there is currently no way to have Grid Engine inserted into the process tree.

You could modify the Grid Engine Source code and use the Docker API to control the container and to collect resource information from the container - however I don't think anyone has done that work in Sun Grid Engine.

However Univa Grid Engine, the commercial version of Grid Engine does have this functionality (Disclaimer: I work for Univa). Univa Grid Engine uses the Docker API to launch the containers and has a co-shepherd which is a small daemon that runs in the container to collect resource usage and provide job control.

Bll Bryce
  • 21
  • 2