1

I have lost connection to Cassandra server and I am not able to restore it back.

When I run: sudo service cassandra status I can see service is running: 

● cassandra.service - LSB: distributed storage system for structured data
   Loaded: loaded (/etc/init.d/cassandra; generated; vendor preset: enabled)
   Active: active (exited) since Fri 2018-04-27 13:06:21 UTC; 4min 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 18479 ExecStop=/etc/init.d/cassandra stop (code=exited, status=0/SUCCESS)
  Process: 18539 ExecStart=/etc/init.d/cassandra start (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/cassandra.service

Apr 27 13:06:21 serveur-1 systemd[1]: Starting LSB: distributed storage system for structured data...
Apr 27 13:06:21 serveur-1 systemd[1]: Started LSB: distributed storage system for structured data.

But When I run > cqlsh, I got the following:

Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
Badis Merabet
  • 13,970
  • 9
  • 40
  • 55
  • Does `nodetool status` give you any output? If not, then Cassandra is not running. – Aaron Apr 27 '18 at 15:52
  • Also, I think you're running into CASSANDRA-14173 (https://issues.apache.org/jira/browse/CASSANDRA-14173): https://stackoverflow.com/questions/48328661/cant-start-cassandra-after-os-patch-up – Aaron Apr 27 '18 at 15:55
  • `nodetool status` gives also an error of connection failure. – Badis Merabet Apr 27 '18 at 17:57
  • I have installed cassandra 3.11 on my local machine. I got this error few minutes at the start. and then `cqlsh` works. but I couldn't figure out why it worked suddenly. `sudo service cassandra status` show the same output on both local machine and on server. – Badis Merabet Apr 27 '18 at 17:59
  • There is nothing wrong with Cassandra or JDK. they are the same on server and on my local machine. the problem started when I upgraded cassandra on server. `/var/lib/cassandra/data` contains non compatible schemas. is there a way to restore my data ? – Badis Merabet Apr 27 '18 at 20:57

3 Answers3

2

Deleting content of directory /var/lib/cassandra/commitlog and restarting cassandra fixed my issue !

Badis Merabet
  • 13,970
  • 9
  • 40
  • 55
1

Check that Cassandra really accepts connections through loopback interface.

See what's in your cassandra.yaml under listen_address and listen_interface property

# cat /etc/cassandra/cassandra.yaml  | grep listen
...
listen_address: 172.17.0.2
...

If you see a non local address, it might happen that cqlsh needs to connect through another interface to your Cassandra instance. Try this command (by replacing the IP with the same IP in configuration:

# cqlsh 172.17.0.2

You can check which hosts are allowed co access your Cassandra service using the following command

# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:7199          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9042            0.0.0.0:*               LISTEN
tcp        0      0 localhost:38899         0.0.0.0:*               LISTEN
tcp        0      0 20f1802:afs3-fileserver 0.0.0.0:*               LISTEN
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path

0.0.0.0 means that my Cassandra instance allows connection from both loopback and ethernet interface.

Oresztesz
  • 2,294
  • 1
  • 15
  • 26
  • I got only this line uncommented: `listen_address: 127.0.0.1` – Badis Merabet Apr 27 '18 at 13:58
  • executing: `sudo lsof -i :9042` give empty result. so cassandra is not listening – Badis Merabet Apr 27 '18 at 14:05
  • You should look for port 9042 in case of Cassandra. I don't see it in your sent screenshot. Can you send the whole execution result of `netstat`? – Oresztesz Apr 27 '18 at 14:11
  • There is nothing listening on port 9042. that's the issue. I will send whole execution now. – Badis Merabet Apr 27 '18 at 14:12
  • 1
    Try to start Cassandra again. Also look for the logs if there was any error during startup. In logs look for the message: `state jump to normal`. This indicates that node is up-and-running. – Oresztesz Apr 27 '18 at 14:17
  • Yes, there is an error in log, but I am not able to understand it. https://imgur.com/a/Q8xxSgD – Badis Merabet Apr 27 '18 at 14:26
  • Unfortunately some details are missing from screenshot. I suspect that there is a port collision somewhere. Try to stop some of the services locally and start Cassandra again. – Oresztesz Apr 27 '18 at 14:29
  • i have chnaged the listen address and rpc address and port neither works for me – Irrfan23 Jan 29 '20 at 18:05
1

Delete all content from commitlog and it will work.

double-beep
  • 5,031
  • 17
  • 33
  • 41
Nilson Olivares
  • 11
  • 1