Retry connection to Cassandra node upon startup

Question

I want to use Docker to start my application and Cassandra database, and I would like to use Docker Compose for that. Unfortunately, Cassandra starts much slower than my application, and since my application eagerly initializes the Cluster object, I get the following exception:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra/172.18.0.2:9042 (com.datastax.driver.core.exceptions.TransportException: [cassandra/172.18.0.2:9042] Cannot connect))
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:233)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1454)
    at com.datastax.driver.core.Cluster.init(Cluster.java:163)
    at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:334)
    at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:309)
    at com.datastax.driver.core.Cluster.connect(Cluster.java:251)

According to the stacktrace and a little debugging, it seems that Cassandra Java driver does not apply retry policies to the initial startup. This seems kinda weird to me. Is there a way for me to configure the driver so it will continue its attempts to connect to the server until it succeeds?

@SotiriosDelimanolis sure; I don't need more for local development and tests, at least for now. — Vladimir Matveev, Aug 02 '16 at 20:18

score 13 · Accepted Answer · edited Aug 09 '16 at 00:58

13

You should be able to write some try/catch logic on the NoHostAvailableException to retry the connection after a 5-10 second wait. I would recommend only doing this a few times before throwing the exception after a certain time period where you know that it should have started by that point.

Example pseudocode

Connection makeCassandraConnection(int retryCount) {
    Exception lastException = new IllegalStateException();
    while (retryCount > 0) {
        try {
            return doConnectionStuff();
        } catch (NoHostAvailableException e) {
            lastException = e;
            retryCount--;
            Thread.sleep(TimeUnit.SECONDS.toMillis(5));
        }
    }
    throw lastException;
}

edited Aug 09 '16 at 00:58

vsminkov

10,912
2
38
50

answered Aug 05 '16 at 14:36

gsteiner

776
5
20

1

Yep, it seems this is what I'll have to do. It is very strange that the driver does not do this automatically. I will probably create a ticket for this feature in their bug tracker. – Vladimir Matveev Aug 11 '16 at 12:38
What if multiple threads are trying to call this method? All the other threads have to wait and it's a performance hit right? – Radhakrishna Sharma Gorenta Aug 23 '18 at 14:28

score 2 · Answer 2 · answered Aug 09 '16 at 16:41

If you don't want to change your client code, and your client application's docker container stops because of the error you can use the following attribute for the client app in your docker-compose file.

restart: unless-stopped

That will restart your client application container as many times as it fails. Example docker-compose.yml file:

version: '2'
services:
  cassandra:
    image: cassandra:3.5
    ports:
      - "9042:9042"
      - "9160:9160"
    environment:
      CASSANDRA_CLUSTER_NAME: demo
  app:
    image: your-app
    restart: unless-stopped

Yes, this is a possible solution, but it does not work if the application restart is not free (e.g. if it has some initialization to do). At the moment my program does not need to do any complex initialization, but it is highly likely that it will need something like it in the future. — Vladimir Matveev, Aug 11 '16 at 12:45

score 1 · Answer 3 · edited May 23 '17 at 11:54

1

The Datastax driver cannot be configured this way.

If this is only a problem with Docker and you do not wish to change your code, you could consider using something such as wait-for-it which is a simple script which will wait for a TCP port to be listening before starting your application. 9042 is cassandra's native transport port.

Other options are discussed here in the docker documentation, but I personally have only used wait-for-it but found it to be useful when working with cassandra within docker.

edited May 23 '17 at 11:54

Community

1
1

answered Aug 10 '16 at 11:05

Samyel

139
5

Actually, I'm currently using [`goss`](https://github.com/aelsabbahy/goss) exactly for this purpose. Among everything else, it is able to check for the availability of a TCP port. But, naturally, I would want the driver to do that instead. – Vladimir Matveev Aug 11 '16 at 12:41

score 0 · Answer 4 · answered Nov 19 '21 at 06:34

refer: https://stackoverflow.com/a/69612290/10428392

You could modify docker compose file like this with a health check.

version: '3.8'
services:
  applicaion-service:
    image: your-applicaion-service:0.0.1
    depends_on:
      cassandra:
        condition: service_healthy


  cassandra:
    image: cassandra:4.0.1
    ports:
      - "9042:9042"
    healthcheck:
      test: ["CMD", "cqlsh", "-u cassandra", "-p cassandra" ,"-e describe keyspaces"]
      interval: 15s
      timeout: 10s
      retries: 10

score -1 · Answer 5 · answered Aug 10 '16 at 16:52

-1

If you orchestrating many dockers, you should go for a docker compose with depends on tag

version: '2'
services:
  cassandra:
    image: cassandra:3.5
    ports:
      - "9042:9042"
      - "9160:9160"
    environment:
      CASSANDRA_CLUSTER_NAME: demo
  app:
    image: your-app
    restart: unless-stopped
    depends_on:
      - cassandra

answered Aug 10 '16 at 16:52

Leandro Jacomelli

1
1
2

The depends on tag only waits for the container to be initiated, Cassandra itself will still take some extra time to startup. – Samyel Aug 11 '16 at 10:23
Yep, `depends_on` does not actually work, this is the first thing I have tried. – Vladimir Matveev Aug 11 '16 at 12:40
even more, depends_on's condition have been removed in docker 3, so you can no longer wait until Cassandra is healthy – George Marin Dec 03 '19 at 10:40

score -2 · Answer 6 · answered Aug 04 '16 at 18:19

Try increasing the connection timeout, it's the one thing sometimes happens on AWS and the like. I think you're looking at a late stage in the error log, at some point it should tell you it couldn't connect because of a timeout or unreachable network, and then it flags nodes as not available.

Using phantom, code is like below:

val Connector = ContactPoints(Seq(seedHost))
    .withClusterBuilder(_.withSocketOptions(
      new SocketOptions()
      .setReadTimeoutMillis(1500)
      .setConnectTimeoutMillis(20000)
    )).keySpace("bla")

Resource Link:

com.datastax.driver.core.exceptions.NoHostAvailableException #445

Cassandra hasn't started listening for clients in the OP's example. Connection attempts will be rejected as nothing is listening on the corresponding port. A timeout won't change that. — Sotirios Delimanolis, Aug 04 '16 at 19:15

Retry connection to Cassandra node upon startup

6 Answers6

Resource Link: