2

I am trying to install hyperledger fabric and kafka orderer in a single VM by docker-compose. I found it is not work when I use VM in NAT network mode. It's work when I use VM in Bridge network node. Bellow is console log:

orderer0.example.com      | [sarama] 2017/09/01 14:10:49.471276 broker.go:96: Failed to connect to broker kafka3:9092: dial tcp: i/o timeout
orderer0.example.com      | [sarama] 2017/09/01 14:10:49.471339 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: i/o timeout
orderer0.example.com      | [sarama] 2017/09/01 14:10:49.471350 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer0.example.com      | [sarama] 2017/09/01 14:10:49.471363 client.go:601: client/metadata fetching metadata for all topics from broker kafka0:9092
orderer1.example.com      | [sarama] 2017/09/01 14:10:49.581308 broker.go:96: Failed to connect to broker kafka1:9092: dial tcp: i/o timeout
orderer1.example.com      | [sarama] 2017/09/01 14:10:49.581336 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: i/o timeout
orderer1.example.com      | [sarama] 2017/09/01 14:10:49.581344 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer1.example.com      | [sarama] 2017/09/01 14:10:49.581357 client.go:601: client/metadata fetching metadata for all topics from broker kafka3:9092
orderer2.example.com      | [sarama] 2017/09/01 14:10:50.633648 broker.go:96: Failed to connect to broker kafka2:9092: dial tcp: i/o timeout
orderer2.example.com      | [sarama] 2017/09/01 14:10:50.633691 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: i/o timeout
orderer2.example.com      | [sarama] 2017/09/01 14:10:50.633699 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer2.example.com      | [sarama] 2017/09/01 14:10:50.633713 client.go:601: client/metadata fetching metadata for all topics from broker kafka0:9092
orderer0.example.com      | [sarama] 2017/09/01 14:10:59.471960 broker.go:96: Failed to connect to broker kafka0:9092: dial tcp: i/o timeout
orderer0.example.com      | [sarama] 2017/09/01 14:10:59.472069 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: i/o timeout
orderer0.example.com      | [sarama] 2017/09/01 14:10:59.472100 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer0.example.com      | [sarama] 2017/09/01 14:10:59.472150 client.go:601: client/metadata fetching metadata for all topics from broker kafka2:9092
orderer1.example.com      | [sarama] 2017/09/01 14:10:59.581900 broker.go:96: Failed to connect to broker kafka3:9092: dial tcp: i/o timeout
orderer1.example.com      | [sarama] 2017/09/01 14:10:59.581927 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: i/o timeout
orderer1.example.com      | [sarama] 2017/09/01 14:10:59.581935 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer1.example.com      | [sarama] 2017/09/01 14:10:59.581948 client.go:601: client/metadata fetching metadata for all topics from broker kafka0:9092
orderer2.example.com      | [sarama] 2017/09/01 14:11:00.635034 broker.go:96: Failed to connect to broker kafka0:9092: dial tcp: i/o timeout
orderer2.example.com      | [sarama] 2017/09/01 14:11:00.635106 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: i/o timeout
orderer2.example.com      | [sarama] 2017/09/01 14:11:00.635122 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer2.example.com      | [sarama] 2017/09/01 14:11:00.635157 client.go:601: client/metadata fetching metadata for all topics from broker kafka1:9092
orderer0.example.com      | [sarama] 2017/09/01 14:11:09.481998 broker.go:96: Failed to connect to broker kafka2:9092: dial tcp: lookup kafka2 on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout
orderer0.example.com      | [sarama] 2017/09/01 14:11:09.482080 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: lookup kafka2 on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout
orderer0.example.com      | [sarama] 2017/09/01 14:11:09.482172 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer0.example.com      | [sarama] 2017/09/01 14:11:09.482194 client.go:601: client/metadata fetching metadata for all topics from broker kafka1:9092
orderer1.example.com      | [sarama] 2017/09/01 14:11:09.591512 broker.go:96: Failed to connect to broker kafka0:9092: dial tcp: lookup kafka0 on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout
orderer1.example.com      | [sarama] 2017/09/01 14:11:09.591542 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: lookup kafka0 on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout
orderer1.example.com      | [sarama] 2017/09/01 14:11:09.591549 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer1.example.com      | [sarama] 2017/09/01 14:11:09.591562 client.go:601: client/metadata fetching metadata for all topics from broker kafka2:9092
orderer2.example.com      | [sarama] 2017/09/01 14:11:10.635648 broker.go:96: Failed to connect to broker kafka1:9092: dial tcp: i/o timeout
orderer2.example.com      | [sarama] 2017/09/01 14:11:10.635798 client.go:620: client/metadata got error from broker while fetching metadata: dial tcp: i/o timeout
orderer2.example.com      | [sarama] 2017/09/01 14:11:10.635827 config.go:329: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
orderer2.example.com      | [sarama] 2017/09/01 14:11:10.635918 client.go:601: client/metadata fetching metadata for all topics from broker kafka3:9092

I use docker network inspect to see the ip of kafka0 is bellow, it is 172.22.0.11

"11eaffebfa5be61cfcaaa400b6a30c4da1b05df109d7b72f1f841ff7d0a57e06": {
            "Name": "kafka0",
            "EndpointID": "c19bcc0f152498e18c2602855e72da01fefd29fd57c7c3d7ff07f4397e2db6bf",
            "MacAddress": "02:42:ac:16:00:0b",
            "IPv4Address": "172.22.0.11/16",
            "IPv6Address": ""
        },

But why orderer is trying to connect kafka0 on 127.0.0.11? see bellow log,

Failed to connect to broker kafka0:9092: dial tcp: lookup kafka0 on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout

I don't get it why orderer look kafka0 in 127.0.0.11 instead of 172.22.0.11. All of container were in the same docker bridge network. I had double confirm it by docker network inspect. The log is bellow,

[
    {
        "Name": "fabricdockercomposesvt_default",
        "Id": "4e8fac6babf5e119e12bb65cf9816be913a038e9f11bd06106f20af53abaa5dc",
        "Created": "2017-09-01T02:59:52.678519627-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.22.0.0/16",
                    "Gateway": "172.22.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "05f86e2c5f9ab82aa17063d84330a8026c730b21c547567855b81599243c3f0d": {
                "Name": "peer1.org1.example.com",
                "EndpointID": "932343ee2303ee87d056f2089305994ab750efc59d0f2ebb0e961e4e940da0eb",
                "MacAddress": "02:42:ac:16:00:02",
                "IPv4Address": "172.22.0.2/16",
                "IPv6Address": ""
            },
            "0e40b56fafb4a53c2efc23e0121c3447218fda02bd0fc50a7b3cf97777a3e552": {
                "Name": "ca_peerOrg1",
                "EndpointID": "6f794c84ab7585bc979a815fe7b510f811cfefd0a8c87c51c4b3fb5e0d14c23e",
                "MacAddress": "02:42:ac:16:00:08",
                "IPv4Address": "172.22.0.8/16",
                "IPv6Address": ""
            },
            "11eaffebfa5be61cfcaaa400b6a30c4da1b05df109d7b72f1f841ff7d0a57e06": {
                "Name": "kafka0",
                "EndpointID": "c19bcc0f152498e18c2602855e72da01fefd29fd57c7c3d7ff07f4397e2db6bf",
                "MacAddress": "02:42:ac:16:00:0b",
                "IPv4Address": "172.22.0.11/16",
                "IPv6Address": ""
            },
            "239381cb14521601efc9dccc9ae96f21b8a7883cc6c1174679e0e59d9f284771": {
                "Name": "orderer1.example.com",
                "EndpointID": "995349ac618b3ce3099d3b8de5466077a08dc48196651fd7eae2bb8ebe29e013",
                "MacAddress": "02:42:ac:16:00:0f",
                "IPv4Address": "172.22.0.15/16",
                "IPv6Address": ""
            },
            "27b4ef727aa9bf189e9b7262d972c627cc77c60407f1e669412b431d6bd48949": {
                "Name": "kafka2",
                "EndpointID": "294e7d9d79da12e35db42d870a9ed78136d19dad5c67bc3e301da4a52a34efde",
                "MacAddress": "02:42:ac:16:00:0d",
                "IPv4Address": "172.22.0.13/16",
                "IPv6Address": ""
            },
            "3959d50127280b311e358668b33b4718c18540c0c7695a2a7bbdd07bbcaf373a": {
                "Name": "zookeeper2",
                "EndpointID": "4e3d9c0d905190bbe24e5e1c9579d4046cb2b8ff33d4a57b23af4a4cd6eddf87",
                "MacAddress": "02:42:ac:16:00:07",
                "IPv4Address": "172.22.0.7/16",
                "IPv6Address": ""
            },
            "5689c378b255e649c40dd5988810437a6c1d0d2ea1c48340da5eb66a684b3a78": {
                "Name": "peer1.org2.example.com",
                "EndpointID": "1ec40982fbbf10005231e6a73557ad677f32c0f1aeb2a77818ae86627c3b0f93",
                "MacAddress": "02:42:ac:16:00:05",
                "IPv4Address": "172.22.0.5/16",
                "IPv6Address": ""
            },
            "5f386fd1480ea079dfaa9ed15246bf7f42cafd82f6bc1f3eaa1f9ae4a9852655": {
                "Name": "orderer0.example.com",
                "EndpointID": "fb76384614442a12735ee2a844787acbd0970a428682236b9a2aba22d8a62269",
                "MacAddress": "02:42:ac:16:00:11",
                "IPv4Address": "172.22.0.17/16",
                "IPv6Address": ""
            },
            "6e773f10d98ce70b768c9ff13cfa15349fd512376748197168762be4af116b0d": {
                "Name": "zookeeper1",
                "EndpointID": "3d6902f623c8541d4657ef4323c1527a6021e7e140eca5dcb9037154f739c96a",
                "MacAddress": "02:42:ac:16:00:0a",
                "IPv4Address": "172.22.0.10/16",
                "IPv6Address": ""
            },
            "8198dd5c82939c6513dd9fcec8b4a308c631d4b008e4d5774bf0ca0f46bc914e": {
                "Name": "orderer2.example.com",
                "EndpointID": "d073d216e8e7173984b0d64ef77d0b85315c6440b306324e518f18893ab3896c",
                "MacAddress": "02:42:ac:16:00:10",
                "IPv4Address": "172.22.0.16/16",
                "IPv6Address": ""
            },
            "8352c58d066ab776ce3e1c50d43d5105130664ba193ae52bc1e00d723923062f": {
                "Name": "cli",
                "EndpointID": "39ce6d60b4f223adcd9969832d0442e78d9a7b30737ab6892614f2198db58c0c",
                "MacAddress": "02:42:ac:16:00:12",
                "IPv4Address": "172.22.0.18/16",
                "IPv6Address": ""
            },
            "8ec391c08bfcb00dd8c08ddb3c230c47e2c19a340e936451aa25dac875556e3f": {
                "Name": "kafka3",
                "EndpointID": "1ddc30616adc68ddb39e912131b10b8719f887d8c589e2044e453dd9aac11e3f",
                "MacAddress": "02:42:ac:16:00:0c",
                "IPv4Address": "172.22.0.12/16",
                "IPv6Address": ""
            },
            "9962d040aa1532798195a4fd29d60308ed2f14cdd6db9c5de15802fde82c9a2f": {
                "Name": "ca_peerOrg2",
                "EndpointID": "bdb3959d034d2989d0b046d323ee716ab85fa168a27cb717b9c8837fe9fef0e1",
                "MacAddress": "02:42:ac:16:00:09",
                "IPv4Address": "172.22.0.9/16",
                "IPv6Address": ""
            },
            "9f7c182c543ae849db3cdf09c011e109b5d4a5cf5922ba6b6bfc5c80e83b080c": {
                "Name": "peer0.org2.example.com",
                "EndpointID": "db5986d7e0ffe77e4eced6c43950aaeabe64e4949ab976d017d8df2c446f58f0",
                "MacAddress": "02:42:ac:16:00:04",
                "IPv4Address": "172.22.0.4/16",
                "IPv6Address": ""
            },
            "ad17f4de4a0a1a4348515bf061fcec3ed70c36882e7181165eb53060714a137d": {
                "Name": "kafka1",
                "EndpointID": "3897d1c584214be158b1c84f65824dcf2fb6649070128e0589da78cbb383472a",
                "MacAddress": "02:42:ac:16:00:0e",
                "IPv4Address": "172.22.0.14/16",
                "IPv6Address": ""
            },
            "d9923058ffe9697103daa740c68fa10d015788f6bc4e83214d8eff21cf9ada43": {
                "Name": "zookeeper0",
                "EndpointID": "a0b2d2a321be2bbbdbaab931b3789572f50ea0fcaed4c80da2acc906fe01fed4",
                "MacAddress": "02:42:ac:16:00:06",
                "IPv4Address": "172.22.0.6/16",
                "IPv6Address": ""
            },
            "fa6e25647fb0450aadcab51266a94701bf6db49439157372db6e016555c08079": {
                "Name": "peer0.org1.example.com",
                "EndpointID": "d8e1566da5b6ad7ad1a6e180ebb13239477b454cfe7f7ef70b7859c3bd4d0d67",
                "MacAddress": "02:42:ac:16:00:03",
                "IPv4Address": "172.22.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Is there anyone who encounter the same problem like me? Any advise will be appreciated!

ray
  • 5,454
  • 1
  • 18
  • 40
Wang Tony
  • 55
  • 6
  • What VM are you using? VirtualBox, KVM, etc. Also, what version of Docker? You might want to try setting the dns-search field for your orderer container in your Compose file. See https://stackoverflow.com/a/45916717/6160507 (this was for the peer, but you can make the same change for your orderer) – Gari Singh Sep 02 '17 at 20:34
  • Many thanks for your prompt reply. What I used id VMware® Workstation 12 Pro 12.5.2 build-4638234. Docker version 17.03.1-ce. I will try your suggestion, many thanks! – Wang Tony Sep 04 '17 at 01:56
  • Hello Gari Singy......many thanks for your advise. It's works. May I ask the purpose of specified dns_search=. in yaml file? many thanks for your help! – Wang Tony Sep 04 '17 at 16:05
  • I'll post in an answer below – Gari Singh Sep 04 '17 at 22:16

1 Answers1

1

dial tcp: lookup kafka2 on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout orderer0.example.com

The error above indicates that the orderer attempted to do a DNS lookup for host kafka2 and then timed out trying to connect to the address returned from DNS.

My guess is that when running the VM in NAT mode you end up using the DNS settings of the host and likely the host's DNS is set to search for DNS names in some specific DNS domain - for example the search domain might be something like search.com so DNS is trying to resolve kafka2 as kafka2.search.com. Likely the internet provider you are using somehow resolves this to an address and then the orderer tries to connect to it.

In order to override this behavior in Docker, you can set the the dns-search field for your orderer container in your Compose file. See here (this was for the peer, but you can make the same change for your orderer). With this setting, it will not append the DNS suffix and will just try to resolve kafka2 which would then resolve to the container IP via Docker's embedded DNS

Community
  • 1
  • 1
Gari Singh
  • 11,418
  • 2
  • 18
  • 41