datadog agent not reachable from inside docker container

Question

I installed dd-agent on Amazon linux ec2. If I run my python script directly on the host machine (I used the SDK named "dogstatsd-python"), all the metrics can be sent to datadog (I logged in to datadoghq.com and saw the metrics there). the script is something like:

from statsd import statsd
statsd.connect('localhost', 8125)
statsd.increment('mymetrics')

However, I launched a docker container and run the same script from inside the container:

from statsd import statsd
statsd.connect('172.14.0.1', 8125)
statsd.increment('my metrics')

'172.14.0.1' is the IP of the host, which was extracted with command

netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'

No metrics were sent to datadog at all.....

I'm guessing that maybe it's due to some configuration issue like "address binding". Maybe the dd-agent I installed on the host can only receive metrics from 'localhost'.

Hope someone could help me. Thank you in advance.

Did you figure out a fix?. I set `non_local_traffic: yes` but it doesn't seem to be working. — nerandell, Aug 29 '17 at 14:39

score 11 · Answer 1 · answered Feb 01 '16 at 14:31

11

You will need to set non_local_traffic: yes in your /etc/dd-agent/datadog.conf file. Otherwise the agent will reject metrics from containers.

After setting, you will need to restart the agent for the change to take effect: sudo /etc/init.d/datadog-agent restart or sudo service datadog-agent restart

The docker-dd-agent image enables non_local_traffic: yes by default.

answered Feb 01 '16 at 14:31

Scott Enriquez

111
2

In the latest version, the file is /etc/datadog-agent/datadog.conf, not dd-agent. – Stian Jensen Aug 02 '18 at 16:49
This is exactly what was needed. My setup: OS Windows server 2016 datadog agent on host. windows containers. leaving the details for others who may need. – Zameer Fouzan Mar 04 '19 at 11:36
I ran this in Docker and for me the easiest way to get this to work was with the environment variable DD_DOGSTATSD_NON_LOCAL_TRAFFIC=True. See also https://docs.datadoghq.com/agent/docker/?tab=standard#dogstatsd-custom-metrics. – Ben Mar 07 '21 at 13:04

score 3 · Answer 2 · answered Jan 31 '16 at 11:39

3

If dd-agent listens on localhost it can receive data only from localhost (127.0.0.1). Try to change the dd-agent host to 0.0.0.0 instead of localhost.

We are using docker-dd-agent and it works OOTB.

answered Jan 31 '16 at 11:39

Aleš Roubíček

5,198
27
29

Thank you! I changed "bind_host" from "localhost" to "0.0.0.0" in /etc/dd-agent/datadog.conf. and restarted dd-agent. But the problem still exists..... is there anything else I need to change? – sheny35 Jan 31 '16 at 20:13
Yes, I know the dd-agent docker image. But for some reason, we try to avoid using it. – sheny35 Jan 31 '16 at 20:14

score 0 · Answer 3 · answered Sep 22 '16 at 12:43

You don't actually want to use the IP of the host in this case. If you're running the docker dd-agent, there are two environment variables you can tap into:

statsd.connect(DOGSTATSD_PORT_8125_UDP_ADDR, DOGSTATSD_PORT_8125_UDP_PORT)

That should do the trick. If not, you should be able to find the relevant info to your problem in this section of the Datadog docs.

Also, I should point out that the only Python library that Datadog shows in their docs is datadogpy.

score 0 · Answer 4 · answered Jun 30 '22 at 10:57

Here are instructions for Datadog version and Ubuntu Linux. The config vars have changed a bit:

Agent 7.35.2 - Commit: edd49be - Serialization version: v5.0.16 - Go version: go1.17.6

Check nano /etc/datadog-agent/datadog.yaml:

# Remember to enable this section
apt_config:
    apm_non_local_traffic: true
    dogstatsd_non_local_traffic: true

Then:

sudo systemctl restart datadog-agent

and:

sudo datadog-agent status

datadog agent not reachable from inside docker container

4 Answers4

Linked