Docker service poor network performance

Question

I've got a go service (http.ListenAndServe) that simply echoes 'hello world' (the most basic service in order to not introduce overhead to the benchmark). The question is, if I run the service with go run server.go and run a performance test (using wrk https://github.com/wg/wrk) on my local machine (Macbook pro), I got a performance of

Requests/sec: 59336.07
Transfer/sec: 8.66MiB

but if I just run the service in docker (docker run -p8080:8080 periket2000/goku) and run the performance test again, I got:

Requests/sec: 4743.77
Transfer/sec: 0.69MiB

Does it make sense? It's docker so poor in performance? I've tested it with several services/stacks and got the same results.

Docker for mac is going to be slower than running docker on a Linux machine. I'm not aware of the specifics of the networking setup but it would make sense it's not as fast as a process running directly on your main OS. Would test with docker performance on a Linux server. The overhead should not be as significant as what you're seeing: https://stackoverflow.com/questions/21889053/what-is-the-runtime-performance-cost-of-a-docker-container — maxm, Jan 14 '19 at 16:55
Presumably you could test with `docker run --net=host -p8080:8080 periket2000/goku` but I'm not sure how that works on osx. — maxm, Jan 14 '19 at 16:56
Ok, I'll do it on linux and let you know for having this post complete. — periket2000, Jan 14 '19 at 17:11
If I run wrk in another container in Docker for Mac the performance is closer: ``` wrk_1 | Requests/sec: 39878.21 wrk_1 | Transfer/sec: 4.87MB ``` I get similar network performance to you when running wrk on my host os. Either way, docker for mac is a development tool, it is not expected to have production performance. — maxm, Jan 14 '19 at 17:22
You're right. If I run the server on a remote location (aws for example) and run the tests from a remote machine (linux server ---- internet ---- testing machine), the performance is quite similar. So, don't rely on docker@Macbook for testing performance. — periket2000, Jan 16 '19 at 09:59

score 1 · Answer 1 · answered Jan 15 '19 at 02:01

The service you describe does extremely little, and from looking at its GitHub page it looks like wrk goes out of its way to avoid things like HTTP persistent connections, so your benchmark basically only tests how fast the combined system can set up and tear down TCP connections. Note that the data rate, even in the loopback case, is really low: there's a lot of handshaking and IP and TCP overhead and HTTP headers before you get to say "hello world" and hang up.

In the loopback case it's highly likely that the MacOS kernel can funnel data almost directly from one process to the other.

Process A |--> Local loopback ----> | Process B

In the Docker case, traffic needs to go over a virtual network interface, into a virtual machine, getting received by a Linux-kernel network interface, traversing a NAT layer into Docker space, and then gets received by the process; replies need to go through this stack in reverse. It's not surprising that this has more overhead, and the tiny packets involved make it worse (see the discussion of connection establishment in the Wikipedia page on TCP; there is waiting for each side to complete the SYN-ACK and FIN-RST sequences and those don't carry data IIRC). So, these results aren't surprising.

Process A |--> Virtual interface
               Linux VM eth0
               iptables NAT
               Linux VM docker0
               Container eth0 ----> | Process B

In absolute numbers, in systems I've worked with in the past, getting over 4,000 requests per second definitely gets into the "respectable performance" space. (The loopback-only 8 MB/s data rate doesn't look stellar, because of the nature of the workload.) It might be worth re-running the benchmark with a more realistic workload (mix of GETs and POSTs; non-trivial data responses; some computation or external I/O required to produce results). I wouldn't expect Docker for Mac to hit 100% of the performance of a purely-native no-external-network setup, but I would expect larger data-to-overhead ratios and things like HTTP persistent connections to help.

Docker service poor network performance

1 Answers1