4

I have a web server running inside a docker container in AWS EC2 Ubuntu instance. When I send requests to the web server, I get the response very slowly (20+ seconds most of the times, although the response time varies). It does not time-out though. The web server is a very lightweight. It is only to test, so almost does not do anything.

docker version 17.03.0-ce

docker-compose version 1.12.0-rc1

How I debugged so far

  1. When sending requests to the web server running in the docker container from within the EC2 instance (url = ' http:// localhost:xxxx/api ') it is still very slow. So should not be related to sending requests from outside.

  2. I run another web server inside the EC2 directly (not in a docker container), and it is not slow. It responses very fastly.

  3. I run another web server inside another docker container in EC2, and it is also very slow!

  4. When I send the request from inside the docker container to the web server that is running in it (at its localhost), it is also very slow!

  5. I run the containers with the same command on my mac computer and the get response is not slow!

Here is one of the containers stats:

CPU %: 0.28%

MEM USAGE / LIMIT: 27.49 MiB / 992.5 MiB

MEM %: 2.77%

NET I/O: 53.7 kB / 30.5 kB

BLOCK I/O: 2.24 MB / 0 B

I understand it might be very hard to know the issue. My question is the steps to debug the cause and finally find the solution. I appreciate if you could explain your approach in detail.

Mahshid Zeinaly
  • 3,590
  • 6
  • 25
  • 32
  • First thing that comes to mind is DNS lookups timing out. – BMitch Mar 23 '17 at 21:35
  • @BMitch Could you please explain of how I can debug? What DNS server? Please note that the request is slow within EC2, so for localhost is slow. – Mahshid Zeinaly Mar 23 '17 at 21:41
  • The DNS configuration inside the container isn't necessarily the same as that of the host, especially if the host is using something like loopback for the DNS server (e.g. dnsmasq). If something on the webserver is doing a lookup that's timing out, like logging the connecting hostname, that could cause delays. But this is one of many possibilities, hard to say without a lot more details. – BMitch Mar 23 '17 at 21:58
  • @BMitch How can I get one step closer to understand/fix it? – Mahshid Zeinaly Mar 23 '17 at 22:34
  • Hello please give us the Dockerfile, your docker run command and how you are testing inside the ec2. So we can help – Carlos Rafael Ramirez Mar 24 '17 at 02:13
  • @CarlosRafaelRamirez Nothing fancy, everything is so simple for just testing purpose. I am using docker-compose version2. I only specified image and port forwarding in the docker-compose.yml. Just these two lines! – Mahshid Zeinaly Mar 24 '17 at 03:35
  • @CarlosRafaelRamirez Please note that when I run the same command to run my docker in my mac computer, it works perfectly and it is not slow! – Mahshid Zeinaly Mar 24 '17 at 04:00
  • @CarlosRafaelRamirez And I only specified docker-compose up to run my containers. – Mahshid Zeinaly Mar 24 '17 at 04:31
  • Okey I cannot find why the slowness. One question is it a t2.micro? – Carlos Rafael Ramirez Mar 24 '17 at 17:02
  • yes, again please keep in mind that another web server that is running on the EC2 (not in docker container) is very fast. So it has to be something about dockers. And I understand it is hard to know the reason. All I am asking is steps towards debugging the root of the problem. – Mahshid Zeinaly Mar 24 '17 at 17:06
  • Did you get to solve the problem? I am experiencing the very same issue. Thanks – Ren Jan 13 '18 at 11:32
  • @Renato No, unfortunately, the question still is open. I just thought maybe because the EC2 machines are virtual machines and we are running another virtualization on top of them, that is why it is too slow?!! I have no idea if there is a solution to this or not. If you found one, please post here :) – Mahshid Zeinaly Jan 25 '18 at 19:53

1 Answers1

1

This sounds like a name resolution problem. To debug this, you can do different things.

  • You can first start a simple tcp server with nc -l -p 8000 within the docker container (which is started -p 8000:8000), and the on the host launch: nc 127.0.0.1 8000, type some charachter, see if the TCP communication works, they should appear within the container.

  • Next, you can do the same as before but using "localhost" instead of 127.0.0.1

  • After this, you can do the same HTTP request you did, but using 127.0.0.1 instead of localhost (this will set the request's Host: header to the same value, which the webserver might not check , or might resolve more easily).

You can also have have a look at the generated /etc/hosts and /etc/resolv.conf within the container. Maybe they do not make sense in the network context of your container.

Also, you can precisely measure the time needed for your requests, if they are near a precise second, this sounds once more like a DNS timeout (if it's 5.003, 10.200, 20.030 seconds, it's like a X seconds timeout, plus the real time needed to respond).

smaftoul
  • 2,375
  • 17
  • 14