1

I have a Cassandra cluster with 3 nodes and want to enable PasswordAuthenticator. I have made the following changes in cassandra.yaml.

authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 60000
roles_update_interval_in_ms: 60000
permissions_validity_in_ms: 60000
permissions_update_interval_in_ms: 60000
credentials_validity_in_ms: 60000
credentials_update_interval_in_ms: 60000

I have increased the validity interval to 60 seconds because there won't be frequent changes in roles.

Now, when i restart a cassandra node, the node sometimes connect successfully with client but after a few seconds, it starts giving "Connection timeout" error. Also, the CPU load increases to 100%. I have attached both screenshots.

During this time, if i run nodetool status, all 3 nodes are shown as UN and service cassandra status also shows Active status

Note: I have not enabled PasswordAuthenticator on all nodes. I just tried it on one node and it starts giving timeout error on connection request.

UPDATE: Tried enabling on all nodes but still getting same error.

Timeout error

Server load

Vitul Goyal
  • 611
  • 7
  • 19

1 Answers1

1

You are getting the Connection timeout because it's likely your app has hit a node which doesn't have authentication enabled.

You need to enable authentication on ALL nodes or your test is not going to be valid. You are not using the feature as it is designed so it shouldn't be a surprise that it is not working as expected. Cheers!

Erick Ramirez
  • 13,964
  • 1
  • 18
  • 23
  • Hi. I tried enabling on all nodes but still getting the same error: "Connection timeout" – Vitul Goyal Aug 25 '21 at 04:11
  • Is there a way to enable logs of why timeout is happening. I looked at var/log/cassandra/system.log and debug.log but nothing is being logged there related to this – Vitul Goyal Aug 25 '21 at 05:47
  • What's the replication settings for the `system_auth` keyspace? We recommend a replication factor of 3. Make sure you repair the keyspace as well. My suspicion is that your app can't create a connection because it can't authenticate to the cluster. Cheers! – Erick Ramirez Aug 25 '21 at 06:00
  • Hi. The replication is 3 only. I have also executed: ```nodetool repair system_auth```. From this link: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/configuration/secureConfigNativeAuth.html – Vitul Goyal Aug 25 '21 at 06:23
  • On running: ```SELECT * FROM system_schema.keyspaces```, i am getting ```{"class":"org.apache.cassandra.locator.SimpleStrategy","replication_factor":"3"}``` for system_auth – Vitul Goyal Aug 25 '21 at 06:23
  • Those indicate that you've got a network connectivity issue. Is there a network route from your app server(s) to the C* cluster? From the app server, try to either `telnet` or `nc` to CQL port `9042` of any of the nodes. If that doesn't work, you have a network issue and should speak to your sysadmins. Cheers! – Erick Ramirez Aug 25 '21 at 06:27
  • Update: I tried with incorrect username/password and it showed "Provided username and/or password are incorrect". But on entering correct username/password, it gives timeout error – Vitul Goyal Aug 25 '21 at 07:12
  • I tried telnet and it's getting connected so this is not the issue. Also, after enabling Password auth, i tried to connect without username/password and got this error instead of timeout: "Underlying connection error: Received error response 'java.lang.RuntimeException: com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty' (0x02000000)" – Vitul Goyal Aug 25 '21 at 10:40