0

I set up a Cassandra cluster of two DCs, one in East US and one West US. There is not VPN/gateway. Every time I restart the whole cluster, nodetool describecluster reports everything normal, and a long read with consistent level ALL works fine.

However, after a few minutes, the "describeculster" on each node shows a UNREACHABLE: [xxx.xxx, ...], the unreachable set increase gradually until the two DCs are not reachable to each other. and the read request fail on ReadTimeoutException.

But any run of "nodetool status" on each node reports all nodes Up & normal. And ssh to the node and ping the other DC works fine, too.

Increasing read_request_timeout_in_ms, tuning GC does not solve the problem.

Any ideas why this can happen?

casandra.yaml: listen_address: {{private_ip}} rpc_address: 0.0.0.0 broadcast_rpc_address: {{public_ip}} broadcast_address: {{public_ip}}

xiaochuanQ
  • 190
  • 5
  • Any errors in logs? – Chris Lohfink Jun 29 '17 at 01:25
  • No errors. Only warning about jmx port not opening. And one at the starting up saying gc pause greater than jvm setting in Cassandra-env.sh, but not marking node down. Then all logs clean. I think I need to turn on more logs. Can you suggest something I should try? – xiaochuanQ Jun 29 '17 at 02:50
  • Finally this line of system.log lead me to this question https://stackoverflow.com/questions/42477913/unexpected-exception-during-request and my own answer: INFO [SharedPool-Worker-4] 2017-06-30 16:33:27,772 Message.java:536 - Unexpected exception during request; channel = [id: 0x4c8011ee, L:/10.2.1.100:9042 ! R:/167.220.0.104:57169] io.netty.channel.unix.Errors$NativeIoException: readAddress() failed: Connection timed out at io.netty.channel.unix.Errors.newIOException(Errors.java:117) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] – xiaochuanQ Jun 30 '17 at 18:01

1 Answers1

0

After several rounds of searching and tuning. I can mitigate this issue mostly now.

  • GC tuning.
  • Increase the read_request_timetout_in_ms to higher level, but this only hide the problem at best.

  • This one should be the critical one. Your infrastructure provider's firewall settings may sleep the idle connections when traffic is low. so keep the tcp alive longer. Per DataStax,

    sudo sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10

My systems default settings were 75, 9 and 300. http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html

Other tuning such as setup gateways and vnet2vnet vpn connections between DC does not help.

xiaochuanQ
  • 190
  • 5