How do you deploy aws-xray-daemon in Docker swarm?

Question

I'm trying to deploy amazon/aws-xray-daemon to my docker swarm.

I didn't do much in terms of configuration because there's not much I can see to configure in the README.md

services:
  xrayd:
    image: amazon/aws-xray-daemon
    deploy:
      restart_policy:
        delay: 2m

I get the following in the logs

2021-02-27T04:50:38Z [Info] Initializing AWS X-Ray daemon 3.2.0
2021-02-27T04:50:38Z [Info] Using buffer memory limit of 78 MB
2021-02-27T04:50:38Z [Info] 1248 segment buffers allocated
2021-02-27T04:50:39Z [Error] Unable to retrieve the region from the EC2 instance EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/dynamic/instance-identity/document: dial tcp 169.254.169.254:80: connect: network is unreachable
2021-02-27T04:50:39Z [Error] Cannot fetch region variable from config file, environment variables and ec2 metadata.

I gave the EC2 instance full xray:* in IAM as well.

score 0 · Answer 1 · answered Mar 01 '21 at 18:24

My understanding would be that X-Ray daemon would not be able to get ec2 metadata - https://github.com/aws/aws-xray-daemon/blob/7494caf05b6f5c1e8c9a59ebefc64b8f822983cd/pkg/conn/conn.go#L152. My recommendation would be to set region explicitly using AWS_REGION environment variable. Also, as a debugging step I would recommend to see if you're able to get ec2 meta data manually. You can follow this post (Find region from within an EC2 instance) to check this for your ec2 instance.

score 0 · Accepted Answer · answered Mar 01 '21 at 18:46

Actually found the problem. docker stack deploy does not update the network.*.internal setting. So even when I changed it to network.*.internal: false it never took the change.

I had to delete and redeploy the stack to get it working.

services:
  xrayd:
    image: amazon/aws-xray-daemon
    # command: --log-level warn
    command: --log-level error
    networks:
      - xray
    logging:
      driver: none
    deploy:
      restart_policy:
        delay: 2m
        max_attempts: 2
networks:
  xray:
    internal: false
    attachable: false

How do you deploy aws-xray-daemon in Docker swarm?

2 Answers2