13

I'm having a problem with sluggish network performance between Docker containers and host's network. I asked this question on the Docker's forum but have received no answers so far.

Problem

Set-up: two Macs on the same local network; the first runs an MQTT broker (mosquitto); the second runs Docker for Mac. Two C++ programs run on the second Mac and exchange data multiple times through the MQTT broker (on the first Mac), using the Paho MQTT C library.

Native run: when I ran the two C++ programs natively, the network performance was excellent as expected. The programs were built with XCode 7.3.

Docker runs: when I ran either of the C++ programs, or both of them, in Docker, the network performance dropped dramatically, roughly 30 times slower than the native run. The Docker image is based on ubuntu:latest, and the programs were built by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.1) 5.4.0 20160609.

I tried to use the host network (--network="host" in Docker run) but it didn't help. I also tried to run the MQTT broker on the second Mac (so that the broker and the containers ran on the same host); the problem persisted. The problem existed on both my work LAN and my home network.

In theory, it could have been that the C++ programs were generally slow in Docker containers. But I doubt this was the case because in my experience, the general performance of C++ code in Docker is about as fast as in the native environment.

Question

What could be the cause of this problem? Are there any settings in Docker that can solve this issue?

Truong
  • 569
  • 2
  • 4
  • 13

2 Answers2

4

Your problem sounds very similar to this open issue on the Docker for Mac repo. Unfortunately, there doesn't seem to be a known solution, but the discussion in there may be useful. My personal guess at the moment is that the bug lives near the hyperkit virtualization being used on Docker for Mac specifically.

In my case, I was oddly able to bypass this issue by using a different physical router, but I have no idea why it worked. Sadly that's not really a 'solution' though.

I hate that this isn't a great answer, but I wanted to at least share the discussion in the open issue. Good luck and keep us posted.

Davis Engeler
  • 119
  • 6
3

I suspect the default allocation of memory and CPU for the containers might not be optimal for the kind of network performance you are trying to achieve.

  1. Investigate the utilization of resources within the containers using standard tools like top, htop, strace etc. Or you can use docker stat command when these instances are in peak operation $ docker stats node1 node2 CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O node1 0.07% 796 KB/64 MB 1.21% 788 B/648 B node2 0.07% 2.746 MB/64 MB 4.29% 1.266 KB/648 B

  2. Then you might want to modify various resource allocation parameters available with docker run. enter image description here

    1. EDIT: Another thing to check would be MTU of the actual system interface and the setting on the docker interfaces. Use --mtu=BYTES to set MTU of your docker values to match your system interface's MTU value
Santanu Dey
  • 2,900
  • 3
  • 24
  • 38
  • Thanks for your answer. I tried what you had suggested. The CPU usage was very low (0.3%), memory usage 4.4Mb of 2Gb limit, net I/O about 500Kb/400Kb total over 35 seconds. The net I/O speed was therefore only 14Kb and 11Kb per second, unusually slow on modern computers and networks. The native run took less than 1 second. Any suggestion? – Truong Jul 25 '16 at 09:05
  • @Truong Can you check the MTU setting as I have added in the EDIT? – Santanu Dey Jul 26 '16 at 07:32
  • The MTU option does not exist on Docker for Mac: https://docs.docker.com/docker-for-mac/networking/. I suppose the MTU will be set to match the host's value by default (1500 in my case). – Truong Jul 26 '16 at 09:59
  • Wow..ok.. I do not know how to go inside the Hyperkit VM to debug or fix the issue. But that VM setting is probably the issue. Would suggest to log a bug with Docker. :( – Santanu Dey Jul 26 '16 at 11:37