0

I have deployed a 4 node Couchbase cluster using Google GKE. The master node exposes ports 8091, 8093 to the Loadbaancer. When connecting to the Loadbalancer IP (external) via a Java app to insert data, I get the timeout error with this stack:

Apr 04, 2017 3:32:15 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][ViewEndpoint]: Socket connect took longer than specified timeout.
Apr 04, 2017 3:32:15 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
Apr 04, 2017 3:32:15 PM com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise notifyListener0
WARNING: An exception was thrown by com.couchbase.client.core.endpoint.AbstractEndpoint$2.operationComplete()
rx.exceptions.OnErrorNotImplementedException: connection timed out: /10.4.0.3:8093
    at rx.Observable$26.onError(Observable.java:7955)
    at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:159)
    at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:120)
    at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)

What's puzzling is that the stack shows 10.4.0.3:8093 which is actually the the IP of the docker container. Appreciate all suggestions.

Prasad
  • 1
  • 1
  • 3

1 Answers1

0

Have you checked the firewall rules for the master node and the workers? You need to allow ingress for the ports you have set up.

See this answer

Community
  • 1
  • 1
Oswin Noetzelmann
  • 9,166
  • 1
  • 33
  • 46
  • gke-auto-complete-a3114b27-all default 10.4.0.0/14 udp,icmp,esp,ah,sctp,tcp gke-auto-complete-a3114b27-ssh default 35.184.136.11/32 tcp:22 gke-auto-complete-a3114b27-node gke-auto-complete-a3114b27-vms default 10.128.0.0/9 tcp:1-65535,udp:1-65535,icmp gke-auto-complete-a3114b27-node k8s-fw-a9b9a61a4198411e780b042010a8000d default 0.0.0.0/0 tcp:8091,tcp:8093 gke-auto-complete-a3114b27-node – Prasad Apr 05 '17 at 18:27
  • Sorry, the above was the firewall-rules listing which was not formatted properly. I have exposed ports 8091 and 8093 to the LB only in the master node (actually, master pod), not the workers. – Prasad Apr 05 '17 at 18:39
  • Update: Increased timeout from 20 seconds to 60 seconds. Bucket openes. However get a lot of warnings thusly: `WARNING: DNS Reverse Lookup of 10.4.1.5 is slow, took 4514ms Apr 05, 2017 1:21:59 PM com.couchbase.client.core.node.CouchbaseNode ` Again timeout when inserting data. Kubedns is up and running. – Prasad Apr 05 '17 at 20:26
  • What is the machine size you are using? Maybe it's a RAM issue? – Oswin Noetzelmann Apr 05 '17 at 21:10
  • Thanks so much for responding, appreciate it greatly! The RAM size per node is 3 GB. The ssystem has 3 nodes and 4 Pods, but all the Pods are running on 2 nodes. The bucket which I'm trying to load is empty. I can open the Couchbase Console from the LB IP, query the travel-sample which I have also, and I see that the travel-sample data is rebalanced properly across the nodes. However, the SELECT query returns very slow. The problem is this reverse DNS look-up, I think. It keeps timing out and takes 5 seconds per node. I think a 1 node system would be very fast without this reverse DNS. – Prasad Apr 05 '17 at 23:33
  • Hm, 3GB should be enough. Can you create a printout of the following commands: `kubectl get events`, `kubectl describe pods`, `kubectl describe nodes` (and `kubectl logs pod-name` for each pod)? – Oswin Noetzelmann Apr 06 '17 at 00:17
  • **Update**: I had Couchbase running on a docker container on my windows box. No Kubernetes. Just 1 node. My App connected to the bucket and inserted data. No problems at all. Created 1 more container. Added to cluster to form 2 node cluster. Rebalanced. Tried exact same app without changes. Now running into the same **timeout** error. – Prasad Apr 06 '17 at 01:17
  • hmm.... Looks like [https://forums.couchbase.com/t/couchbase-issue-with-java-and-docker-setup/10896](https://forums.couchbase.com/t/couchbase-issue-with-java-and-docker-setup/10896) The solution of modifying /etc/hosts may not work with kubernetes because of ephemeral IPs. The problem seems to be only with Java SDK. Incentive to move to Python. – Prasad Apr 06 '17 at 01:27
  • Python is not a bad language :) – Oswin Noetzelmann Apr 06 '17 at 01:58