4

I'm trying to use the loki log driver in a docker container on an AWS ec2 host. The config below works fine on my own ubuntu 20.04 machine but not in an ubuntu 16.04 ec2 host. iptables has a loopback rule and the appropriate docker port for loki, :3100. I even opened up that port in the security group. The docker version is 20.10.2.

I have tried to use localhost:3100, the hostname:3100, also the docker container name. I have a bridge network and don't want to use a host network. I also don't want to use the container IP address.

Nothing in this SO question works for me. How to access host port from docker container

Here is my docker-compose.yaml which works on my local machine but not in ec2.

the tools container is a go html server. The tools and loki containers are on the same host.

I can post a log to loki from an alpine container using curl to URL http://loki:3100/loki/api/v1/push but not from my tools container. I can connect with telnet but curl times out.

version: '3.3'

networks:
  traefik:
    external: true
  loki: {}
  
services:

  tools:
    build: .
    container_name: tools
    restart: always
    networks:
      - traefik
      - loki
    extra_hosts:
      - "host.docker.internal:host-gateway"
    logging:
      driver: loki
      options: 
        loki-url: http://loki:3100/loki/api/v1/push
        loki-external-labels: job=tools
    labels:
    ...

Here is my loki docker-compose.yaml

version: "3.8"

networks:
  traefik:
    external: true
  loki: {}

volumes:
  loki_data:
   
services:
  
  loki:
    container_name: "loki"
    image: grafana/loki:2.1.0
    restart: always
    networks:
      - traefik
      - loki
    ports:
      - 3100:3100
    volumes:
      - type: volume
        source: loki_data
        target: /data
      - type: bind
        source: ./config/s3-loki-bolt-conf.yml
        target: /etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml        
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

the docker plugin is installed

docker plugin ls
ID             NAME          DESCRIPTION           ENABLED
a03c22e8375e   loki:latest   Loki Logging Driver   true

sudo journalctl -u docker.service | grep loki is giving me this error.

08 06:46:06 docker1 dockerd[30842]: time="2021-02-08T06:46:06-08:00" level=info msg="level=warn ts=2021-02-08T14:46:06.586642758Z caller=client.go:322 container_id=5a5fbd8a7077243de9db74f549ab619f783eda978ee234651ad1849263a534fe component=client host=localhost:3100 msg=\"error sending batch, will retry\" status=-1 error=\"Post \\\"http://localhost:3100/loki/api/v1/push\\\": context deadline exceeded\"" plugin=fef8734ec8cc2d252f8c4e73e3e91fe8293d2847c7ce1d6df2fb2172a1c288ce
Jens Baitinger
  • 2,230
  • 14
  • 34
markhorrocks
  • 1,199
  • 19
  • 82
  • 151
  • 1
    Have you checked docker logs for warnings / errors? `journalctl -u docker.service | grep loki` . Make the application to produce some logs before executing the command. The journal can be rotated if you haven't used the application for some time. – anemyte Feb 04 '21 at 20:39
  • 1
    I see this `Feb 04 17:16:00 docker1 dockerd[1582]: time="2021-02-04T17:16:00.528957535-08:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/a03c22e8375edea4500d425c526c3012375e042151725c7c3456efbd9859f1f2/loki.sock/LogDriver.Capabilities: Post http://%2Frun%2Fdocker%2Fplugins%2Fa03c22e8375edea4500d425c526c3012375e042151725c7c3456efbd9859f1f2%2Floki.sock/LogDriver.Capabilities: dial unix /run/docker/plugins/a03c22e8375edea4500d425c526c3012375e042151725c7c3456efbd9859f1f2/loki.sock: connect: no such file or directory, retrying in 1s"` – markhorrocks Feb 05 '21 at 01:20
  • 1
    No socket, huh? The only suggestion about missing socket I found is to reinstall the logging plugin, would you try it? As for the `loki-url` I think it should be `http://localhost:3100/loki/api/v1/push`. I know you have tried using localhost, it's just for the sake of eliminating a secondary possible problem. I have doubts that `loki` hostname could be resolved at the host namespace, but `localhost:3100` must work if the loki container is working (`curl localhost:3100/ready` if you want to check that). – anemyte Feb 05 '21 at 06:22
  • docker ps shows the port as `0.0.0.0:3100->3100/tcp` but `curl localhost:3100/ready` does not work – markhorrocks Feb 08 '21 at 14:08
  • 2
    I think I know where the problem may be. Can you add a simple bridge network to the loki container along with the `traefik` network? After that check if you can reach loki with `curl localhost:3100/ready`. – anemyte Feb 08 '21 at 14:36
  • I removed the plugin and reinstalled it. The socket error went away but still the push timed out. – markhorrocks Feb 08 '21 at 14:40
  • I added a new external network named loki but it made no difference, question edited to show the error. – markhorrocks Feb 08 '21 at 14:45
  • 1
    Why have you made it external? Define it like this: `loki: {}` – anemyte Feb 08 '21 at 14:49
  • I created the network as you suggested and edited my question to show how I created itbut the error persists. – markhorrocks Feb 08 '21 at 14:53
  • Please add it to the `loki` container, not `tools`. You can't reach loki from localhost and that is the problem. – anemyte Feb 08 '21 at 14:55
  • yes, I did. That was an error editing here only. – markhorrocks Feb 08 '21 at 14:57
  • I'm also seeing this error but the IP doesn't exist. `error=\"Post \\\"http://localhost:3100/loki/api/v1/push\\\": dial tcp 52.205.69.18:3100: i/o timeout` – markhorrocks Feb 08 '21 at 15:04
  • Can you add what's in your hosts file? – anemyte Feb 09 '21 at 06:08
  • 127.0.0.1 docker1.mydomain.com docker1 localhost – markhorrocks Feb 09 '21 at 06:10

0 Answers0