Consul Empty reply from server

Question

I'm trying to get a consul server cluster up and running. I have 3 dockerized consul servers running, but I can't access the Web UI, the HTTP API nor the DNS.

$ docker logs net-sci_discovery-service_consul_1
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v0.8.5'
           Node ID: 'ccd38897-6047-f8b6-be1c-2aa0022a1483'
         Node name: 'consul1'
        Datacenter: 'dc1'
            Server: true (bootstrap: false)
       Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 172.20.0.2 (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2017/07/07 23:24:07 [INFO] raft: Initial configuration (index=0): []
    2017/07/07 23:24:07 [INFO] raft: Node at 172.20.0.2:8300 [Follower] entering Follower state (Leader: "")
    2017/07/07 23:24:07 [INFO] serf: EventMemberJoin: consul1 172.20.0.2
    2017/07/07 23:24:07 [INFO] consul: Adding LAN server consul1 (Addr: tcp/172.20.0.2:8300) (DC: dc1)
    2017/07/07 23:24:07 [INFO] serf: EventMemberJoin: consul1.dc1 172.20.0.2
    2017/07/07 23:24:07 [INFO] consul: Handled member-join event for server "consul1.dc1" in area "wan"
    2017/07/07 23:24:07 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2017/07/07 23:24:07 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2017/07/07 23:24:07 [INFO] agent: Started HTTP server on 127.0.0.1:8500
    2017/07/07 23:24:09 [INFO] serf: EventMemberJoin: consul2 172.20.0.3
    2017/07/07 23:24:09 [INFO] consul: Adding LAN server consul2 (Addr: tcp/172.20.0.3:8300) (DC: dc1)
    2017/07/07 23:24:09 [INFO] serf: EventMemberJoin: consul2.dc1 172.20.0.3
    2017/07/07 23:24:09 [INFO] consul: Handled member-join event for server "consul2.dc1" in area "wan"
    2017/07/07 23:24:10 [INFO] serf: EventMemberJoin: consul3 172.20.0.4
    2017/07/07 23:24:10 [INFO] consul: Adding LAN server consul3 (Addr: tcp/172.20.0.4:8300) (DC: dc1)
    2017/07/07 23:24:10 [INFO] consul: Found expected number of peers, attempting bootstrap: 172.20.0.2:8300,172.20.0.3:8300,172.20.0.4:8300
    2017/07/07 23:24:10 [INFO] serf: EventMemberJoin: consul3.dc1 172.20.0.4
    2017/07/07 23:24:10 [INFO] consul: Handled member-join event for server "consul3.dc1" in area "wan"
    2017/07/07 23:24:14 [ERR] agent: failed to sync remote state: No cluster leader
    2017/07/07 23:24:17 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2017/07/07 23:24:17 [INFO] raft: Node at 172.20.0.2:8300 [Candidate] entering Candidate state in term 2
    2017/07/07 23:24:17 [INFO] raft: Election won. Tally: 2
    2017/07/07 23:24:17 [INFO] raft: Node at 172.20.0.2:8300 [Leader] entering Leader state
    2017/07/07 23:24:17 [INFO] raft: Added peer 172.20.0.3:8300, starting replication
    2017/07/07 23:24:17 [INFO] raft: Added peer 172.20.0.4:8300, starting replication
    2017/07/07 23:24:17 [INFO] consul: cluster leadership acquired
    2017/07/07 23:24:17 [INFO] consul: New leader elected: consul1
    2017/07/07 23:24:17 [WARN] raft: AppendEntries to {Voter 172.20.0.3:8300 172.20.0.3:8300} rejected, sending older logs (next: 1)
    2017/07/07 23:24:17 [WARN] raft: AppendEntries to {Voter 172.20.0.4:8300 172.20.0.4:8300} rejected, sending older logs (next: 1)
    2017/07/07 23:24:17 [INFO] raft: pipelining replication to peer {Voter 172.20.0.3:8300 172.20.0.3:8300}
    2017/07/07 23:24:17 [INFO] raft: pipelining replication to peer {Voter 172.20.0.4:8300 172.20.0.4:8300}
    2017/07/07 23:24:18 [INFO] consul: member 'consul1' joined, marking health alive
    2017/07/07 23:24:18 [INFO] consul: member 'consul2' joined, marking health alive
    2017/07/07 23:24:18 [INFO] consul: member 'consul3' joined, marking health alive
    2017/07/07 23:24:20 [INFO] agent: Synced service 'consul'
    2017/07/07 23:24:20 [INFO] agent: Synced service 'messaging-service-kafka'
    2017/07/07 23:24:20 [INFO] agent: Synced service 'messaging-service-zookeeper'


$ curl http://127.0.0.1:8500/v1/catalog/service/consul
curl: (52) Empty reply from server
dig @127.0.0.1 -p 8600 consul.service.consul

; <<>> DiG 9.8.3-P1 <<>> @127.0.0.1 -p 8600 consul.service.consul
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached


$ dig @127.0.0.1 -p 8600 messaging-service-kafka.service.consul

; <<>> DiG 9.8.3-P1 <<>> @127.0.0.1 -p 8600 messaging-service-kafka.service.consul
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

I can't get my services to register via the HTTP API either; those shown above are registered using a config script when the container launches.

Here's my docker-compose.yml:

version: '2'
services:
  consul1:
    image: "consul:latest"
    container_name: "net-sci_discovery-service_consul_1"
    hostname: "consul1"
    ports:
      - "8400:8400"
      - "8500:8500"
      - "8600:8600"
    volumes:
      - ./etc/consul.d:/etc/consul.d
    command: "agent -server -ui -bootstrap-expect 3 -config-dir=/etc/consul.d -bind=0.0.0.0"
  consul2:
    image: "consul:latest"
    container_name: "net-sci_discovery-service_consul_2"
    hostname: "consul2"
    command: "agent -server -join=consul1"
    links:
      - "consul1"
  consul3:
    image: "consul:latest"
    container_name: "net-sci_discovery-service_consul_3"
    hostname: "consul3"
    command: "agent -server -join=consul1"
    links:
      - "consul1"

I'm relatively new to both docker and consul. I've had a look around the web and the above options are my understanding of what is required. Any suggestions on the way forward would be very welcome.

Edit:

Result of docker container ps -all:

$ docker container ps --all
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                                                                            NAMES
e0a1c3bba165        consul:latest            "docker-entrypoint..."   38 seconds ago      Up 36 seconds       8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                                       net-sci_discovery-service_consul_3
7f05555e81e0        consul:latest            "docker-entrypoint..."   38 seconds ago      Up 36 seconds       8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp                                                       net-sci_discovery-service_consul_2
9e2dedaa224b        consul:latest            "docker-entrypoint..."   39 seconds ago      Up 38 seconds       0.0.0.0:8400->8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 8300-8302/tcp, 8600/udp, 0.0.0.0:8600->8600/tcp   net-sci_discovery-service_consul_1
27b34c5dacb7        messagingservice_kafka   "start-kafka.sh"         3 hours ago         Up 3 hours          0.0.0.0:9092->9092/tcp                                                                                           net-sci_messaging-service_kafka
0389797b0b8f        wurstmeister/zookeeper   "/bin/sh -c '/usr/..."   3 hours ago         Up 3 hours          22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp                                                               net-sci_messaging-service_zookeeper

Edit:

Updated docker-compose.yml to include long format for ports:

version: '3.2'
services:
  consul1:
    image: "consul:latest"
    container_name: "net-sci_discovery-service_consul_1"
    hostname: "consul1"
    ports:
      - target: 8400
        published: 8400
        mode: host
      - target: 8500
        published: 8500
        mode: host
      - target: 8600
        published: 8600
        mode: host
    volumes:
      - ./etc/consul.d:/etc/consul.d
    command: "agent -server -ui -bootstrap-expect 3 -config-dir=/etc/consul.d -bind=0.0.0.0 -client=127.0.0.1"
  consul2:
    image: "consul:latest"
    container_name: "net-sci_discovery-service_consul_2"
    hostname: "consul2"
    command: "agent -server -join=consul1"
    links:
      - "consul1"
  consul3:
    image: "consul:latest"
    container_name: "net-sci_discovery-service_consul_3"
    hostname: "consul3"
    command: "agent -server -join=consul1"
    links:
      - "consul1"

`$ docker version Client: Version: 17.06.0-ce API version: 1.30 Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:31:53 2017 OS/Arch: darwin/amd64 Server: Version: 17.06.0-ce API version: 1.30 (minimum version 1.12) Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:51:55 2017 OS/Arch: linux/amd64 Experimental: true`. That's macOS Sierra. — junglie85, Jul 08 '17 at 06:51
Are you using DOcker Toolbox, with VirtualBox, or Docker for Mac (https://docs.docker.com/docker-for-mac/install/) with Xhyve (https://github.com/mist64/xhyve)? — VonC, Jul 08 '17 at 09:34
Docker for Mac. If that's with Xhyve by default then that too. — junglie85, Jul 08 '17 at 09:38
When you do a `docker container ps --all`, do you see those port as published (like `*:80 -> 80`)? — VonC, Jul 08 '17 at 09:39
Did you active the port forwarding as in https://telliott.io/2016/04/23/docker-for-mac-first-impressions.html? `pinata set native/port-forwarding true` — VonC, Jul 08 '17 at 09:48
I've edited the question to include the result of `docker container ps -all`. Re port forwarding, I've not done that (and the update at the bottom saying `localhost` is now the default for port mapping makes me think I don't need to?). If I run a container with consul in dev mode I can access the various endpoints. I've also had no issue with other services, e.g. zookeeper and kafka containers above work as expected. — junglie85, Jul 08 '17 at 10:05

VonC · Answer 1 · 2017-07-08T10:11:14.327

0

From the Consul Web Gui page, make sure you have launched an agent with the -ui parameter.

The UI is available at the /ui path on the same port as the HTTP API.
By default this is http://localhost:8500/ui

I do see 8500 mapped to your host on broadcast (0.0.0.0).

Check also (as in this answer) if the client_addr can help (at least for testing)

edited Jul 08 '17 at 10:11

answered Jul 08 '17 at 10:09

VonC

1,262,500
529
4,410
5,250

@amb85 I see it now :) I have added another option. – VonC Jul 08 '17 at 10:12
@amb85 If this is still not working, that would look like a port forwarding issue between the virtual machine xhyve and the physical host Mac. – VonC Jul 08 '17 at 10:12
@amb85 Chec the output of `pinata -list`, as in https://github.com/moby/moby/issues/23388 – VonC Jul 08 '17 at 10:14
Will check that out. I've been thinking about the `client` and `bind` addresses and to be perfectly honest I'm very confused about the purpose of each and what they actually do. I've not found the documentation overly clear on that front and even the link above has left me scratching my head a bit. – junglie85 Jul 08 '17 at 10:18
Is there a docker-ce equivalent to `pinata -list`? Command isn't found. – junglie85 Jul 08 '17 at 10:20
It seems that [Docker for Mac no longer includes pinata](https://forums.docker.com/t/pinata-missing-in-latest-mac-beta-1-11-2-beta15/15541/34). Struggling to figure out how I execute this from the GUI preferences. – junglie85 Jul 08 '17 at 10:36
@amb OK, so the ports should be forwarded already. Did you check with telnet? `telet localhost 8500` – VonC Jul 08 '17 at 10:38
`$ telnet localhost 8500 Trying ::1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host.` – junglie85 Jul 08 '17 at 10:43
So the port is listening. Is there any log in the consulate container that might show any clue? `docker container logs consul1` (replace consulate by its actual name or is ID) – VonC Jul 08 '17 at 10:47
No, the logs are only showing information about bootstrapping, member joining and leadership election. – junglie85 Jul 08 '17 at 10:53
How about the docker daemon logs (maybe with `docker system logs`, I don't remember) – VonC Jul 08 '17 at 10:59
Can't find anything. The best I could come up with was searching the mac system logs for Docker. – junglie85 Jul 08 '17 at 11:11
I appreciate all your help with this. I've still not found a solution, so am going to stick with using consul in dev mode for the time being. As and when I find a solution to this problem I'll try and remember to come back and post it. – junglie85 Jul 08 '17 at 12:13
@amb85 Sorry I was away for the day. For testing, can you replace your port publishing option in your Dockerfile by the long form? https://docs.docker.com/compose/compose-file/#long-syntax-1 That is: `target: 8500 published: 8080 mode: host` (on multiple lines as shown in the doc) The "`mode host`" part is important. – VonC Jul 08 '17 at 18:50
No worries, we can't be glued to the computer all the time! I appreciate your persistence in trying to help me resolve this problem. I've tried the long format, a copy of `docker-compose.yml` is above. Still getting the same problem. – junglie85 Jul 08 '17 at 23:10
@amb85 OK. Did you check out https://docs.docker.com/docker-for-mac/networking/#i-cannot-ping-my-containers? For instance, it mentions bridge networks are not routed. But I think the netork created is an overlay by default (`docker network ls`) – VonC Jul 09 '17 at 04:32

Consul Empty reply from server

1 Answers1