How to find Cassandra startup failure cause

Question

I'm trying to run Cassandra in Docker. Previously that was fine (unfortunately I used latest image). Now when I run compose with different versions it never succeeds. Sometimes I see in logs that startup compete, sometimes now. Every time it ends up with cassandra exited with code 137. I can find no errors in logs. How can I diagnose the problem?

Here's my compose file. I tried to switch between 3.0.24, 3.11, 4 and 4.0.1 versions with no luck.

version: '3'

services:

  cassandra:
    image: cassandra:3.0.24
    container_name: cassandra
    ports:
      - '7000:7000'
      - '9042:9042'
      - '9142:9142'

    volumes:
      - ./cassandra/cassandra-data:/var/lib/cassandra

    environment:
      - CASSANDRA_SEEDS=cassandra
      - CASSANDRA_PASSWORD_SEEDER=yes
      - CASSANDRA_PASSWORD=cassandra
    
    networks:
      - default-dev-network

networks:
  default-dev-network:
    external: true

UPDATE

Here's a logs example. But it varies from run to run.

INFO  16:01:43 Node /172.18.0.5 state jump to NORMAL

INFO  16:01:43 Waiting for gossip to settle before accepting client requests...

INFO  16:01:51 No gossip backlog; proceeding

INFO  16:01:51 Netty using native Epoll event loop

INFO  16:01:51 Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]

INFO  16:01:51 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...

INFO  16:01:51 Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it

INFO  16:01:51 Startup complete

This might help https://stackoverflow.com/a/36666348/11895568 — ale917k, Oct 23 '21 at 16:11
What do you get by running `docker inspect`? Most importantly, is `OOMKilled` on true? — ale917k, Oct 23 '21 at 16:20
no, it's `false` ``` "State": { "Status": "exited", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 137, "Error": "", "StartedAt": "2021-10-23T16:01:36.434508489Z", "FinishedAt": "2021-10-23T16:01:54.005601692Z" } ``` — Yura, Oct 23 '21 at 16:24

Yura · Answer 1 · 2021-10-23T21:11:54.433

0

The reason was memory or CPU issue. After adding resources it runs, but not every time. Playing with CPUs and memory somehow helps but didn't bring reliable result.

Here's full compose file

version: '3'

services:

  cassandra:
    image: cassandra:3.0.24
    container_name: cassandra
    
    deploy:
      replicas: 1
      resources:
        limits: 
          cpus: '2'
          memory: 2G

    ports:
      - '7000:7000'
      - '9042:9042'
      - '9142:9142'

    volumes:
      - ./cassandra/cassandra-data:/var/lib/cassandra

    environment:
      - CASSANDRA_SEEDS=cassandra
      - CASSANDRA_PASSWORD_SEEDER=yes
      - CASSANDRA_PASSWORD=cassandra
    
    networks:
      - default-dev-network

networks:
  default-dev-network:
    external: true

edited Oct 23 '21 at 21:11

answered Oct 23 '21 at 16:39

Yura

969
14
33

unfortunately this also doesn't work good – Yura Oct 23 '21 at 20:48
1

I would say there's not enough RAM to run cassandra. cassandra-env.sh will attempt to default the heap size to one quarter of available RAM. Cassandra 3 would like to have at least 8G in production, for example – LHWizard Mar 01 '22 at 18:18
It was long time ago, but I think you're rigth – Yura Mar 03 '22 at 12:23

How to find Cassandra startup failure cause

1 Answers1