6

Single docker container is working good for less number of parellel processes but when we increase the number of parellel processes to 20-30 the process execution get slows. The processes are getting slow but still the docker is utilizing only 30-40% of cpu.

I have tried following things to make docker utilize proper cpu and don't slow down the processes -

  1. I have explicitly allocated the cpu and ram to the docker container.
  2. I have also increased the number of file descriptors, number of process and stack size using ulimit.

even after doing this two thing still the container is not utilizing cpu properly. I am using docker exec to start multiple processes in single running container. Is there any efficient way to use single docker container for executing multiple processes or to make container use 100% of cpu?

The configuration i am using is

Server - aws ec2 t2.2Xlarge ( 8 core, 32 gb ram)

Docker version - 18.09.7

Os- ubuntu 18.04

shubhamsoni136
  • 81
  • 6
  • Please describe the nature of the 'processes' you are running. It could be that CPU isn't the problem, but file I/O, memory, network, software dependencies not optimized for multiple cores, etc, etc... – Knelis Jul 18 '19 at 07:39
  • The processes are just simple python scripts with "print("hello world")" code in it. As i am using docker exec to exceute so i thunk network is not gonna be an issue and also only 1 gb of memory is used 31 gb is still remaining when i run it on 8 core 32 gb ram instance. – shubhamsoni136 Jul 18 '19 at 07:47
  • "Hello world" is being written to ? – Ankit Deshpande Jul 18 '19 at 08:24
  • If the CPU usage is around 100%, this means that your server is trying to do more work than it has the capacity for. This is usually OK, but it means that programs may slow down a little. Why do you want this 100% use? – NID Jul 18 '19 at 08:28
  • Why aren't you just running 20-30 parallel containers based on the same image? That seems like a better structure than having a single mega-container and using `docker exec`. – David Maze Jul 18 '19 at 08:41
  • @AnkitDeshpande the ouput is written on the docker exec process's stdout. – shubhamsoni136 Jul 18 '19 at 08:44
  • @NID the issue is that the cpu is only 30-40% still the processes are getting slow. – shubhamsoni136 Jul 18 '19 at 08:45
  • Also note that [AWS t2 instances aren't guaranteed 100% CPU availability](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html) and if you're running under load for a long period of time [you'll in effect get 40% CPU](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html). I don't know off hand how the OS sees this. Switching to a c5 or m5 instance would avoid this. – David Maze Jul 18 '19 at 08:47
  • @DavidMaze if we run 20-30 containers parellelly that they all will fight for cpu time and this makes cpu utilization 100% and eventually slow down the process execution. – shubhamsoni136 Jul 18 '19 at 08:51
  • I think you should test this some other way. I mean, to really conclude that Docker isn't using all available CPU you should be running some application designed to use maximum CPU. Right now, all you're saying is it's an application printing `Hello world` to `stdout`. Doesn't sound like a process that should be using much CPU, even when running many instances. I think there are other bottlenecks here, if any. – Knelis Jul 18 '19 at 08:58
  • @DavidMaze i have tried this with m5d.2xlarge but still the situation is same as i am increasing the number of processes, the process execution time is also increasing. – shubhamsoni136 Jul 18 '19 at 09:16
  • @Knelis i am also trying to find that bottlenack but till now i haven't found any solution. As i am running all this processes through docker exec, so is there any limitation on docker exec command? – shubhamsoni136 Jul 18 '19 at 09:19
  • I don't think `docker exec` was especially designed to be used as a high-volume concurrent process driver. I would not be surprised at all to see mutexes in the Docker core that caused this behavior. Basically the only thing your benchmark is testing is `docker exec` performance and Python startup time; I'd build something more realistic. – David Maze Jul 18 '19 at 10:05

1 Answers1

0

When you run something on machine it consumes following resources:1. CPU 2. RAM 3. DISK I/O 4. Network Bandwidth. If your container is exhausting any one resource listed above than it is possible that other resources available. So monitor your system matrices to find the root cause.

Manoj Sahu
  • 2,774
  • 20
  • 18
  • it is using only 30-40% of cpu, ram is not an issue, it is using only 1 gb out of 32 gb available and as i am using docker exec and the script is directly writing to docker exec process's stdout so i think network bandwidth is not an issue. I have checked disk io with iotop anf found that it is around 0.2%, so this is also not an issue. – shubhamsoni136 Jul 18 '19 at 12:26
  • @shubhamsoni136 did you manage to find the solution to this problem? – HelmBurger Jun 06 '22 at 13:49