0

The ElasticSearch 6.2.2 service on this Ubuntu 16.04.3 machine has been running fine for a few months. I rebooted the machine today and found the ElasticSearch service had not started. So I tried sudo systemctl start elasticsearch.service. I was not given any error. But I noticed it still hadn't started.

Upon running systemctl status elasticsearch I get this:

`● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─override.conf
   Active: inactive (dead) (Result: exit-code) since Tue 2018-09-18 02:35:58 UTC; 19min ago
     Docs: http://www.elastic.co
  Process: 14147 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 14147 (code=exited, status=1/FAILURE)

Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Unit entered failed state.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Service hold-off time over, scheduling restart.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: Stopped Elasticsearch.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: elasticsearch.service: Start request repeated too quickly.
Sep 18 02:35:58 elastic-eu-vm-000 systemd[1]: Failed to start Elasticsearch.

The solution which seemed to help many people who couldn't start the ES service on Ubuntu found here (Can not start elasticsearch as a service in ubuntu 16.04), the one accepted as the answer, did not help me.

This is a production machine so I hope someone can help! Thanks!!

EDIT 1 - after much pain trying to track down the log file, it's now at the following url per suggestion from @ArnavRay https://file.io/ASnEG4

Stpete111
  • 3,109
  • 4
  • 34
  • 74

3 Answers3

1

It may happen because of two reason: first elasticsearch process is not stopped properly.

try

jps

and

ps aux |grep elasticsearch

to find out PID and

kill -9 [PID]

second the memory is not freed properly. to find out that try

free -m

there are several way to free the memory but I recommend reboot the server.

hamid bayat
  • 2,029
  • 11
  • 20
  • Hamid, I am new to Linux so might need more detail on your instructions. 1. run jps give me 23036 jps. What does this mean? 2. ps aux command - it gives me 4 different numbers. Which is the PID? 3. I have run the free -m command. More than half of my memory is free according to the results number. But in any case, what do you mean by "reboot the server?" Thank you for your time. – Stpete111 Sep 18 '18 at 12:54
  • Hamid, thanks for your input but none of our suggestions have resolved my issue. – Stpete111 Sep 18 '18 at 13:09
1

I Went through your logs. The x-pack license has expired. If you are using any of the features of x-pack, you have to get a new license key. If you are not using x-pack you can disable it and restart ElasticSearch. You can add the following lines to your elasticsearch.yml to disable x-pack features. 

xpack.graph.enabled: false
xpack.ml.enabled : false
xpack.monitoring.enabled: false
xpack.reporting.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false

According to your logs the ElasticSearch stopped right after it could not execute watchers. If you are using any watchers stop the queries as well to check if ElasticSearch starts. This should solve the issue.

ArnavRay
  • 321
  • 1
  • 9
  • Hi Arnav - thank you for your helpful and detailed reply. I have added all the above lines to the yml file, but still no luck. The latest log is at https://file.io/L7A1X0 but not sure this will help - I don't see anything at the bottom of it with the timestamp of when I tried to start the service after adding the lines to the yml file. So with that said, I need a little more guidance on what you mean by "If you are using any watchers stop the queries as well to check if ElasticSearch starts." How would I go about doing this? – Stpete111 Sep 18 '18 at 23:00
1

The only error on the logs are related to the x-pack security. Are you using any of the features of x-pack? Here is a link that describes what happens when the x-pack license expires. https://www.elastic.co/guide/en/x-pack/current/license-expiration.html

Try running elasticsearch-plugin remove x-pack

Default location for the elasticsearch-plugin is /usr/share/elasticsearch/bin/. You might need to use sudo. Remove all the x-pack related configuration from elasticsearch.yml and restart ElasticSearch.

https://github.com/deviantony/docker-elk/issues/154 - This has the same issue as yours.

If you are still facing issues you can try the following things.

  1. Increase the heap memory on jvm.options under /etc/elasticsearch/
  2. Use ls -a and check if any other elasticsearch.yml files has been created. If so remove them.

  3. Run these commands

    echo -e "elasticsearch - memlock unlimited\nstorm - nproc 257597" >> /etc/security/limits.conf ulimit -n 32768 ulimit -u 65536 echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf

  4. Try restarting the system. This has saved me a lot of headaches over the years :)
ArnavRay
  • 321
  • 1
  • 9
  • Arnav, thank you for your continued assistance. I am going to try all of your suggestions here today. With that said, I did notice that the latest time stamp of the logs I've provided are from when the problem first happened, and there have been no logs written ever since. So in other words, Elasticsearch isn't even writing to the logs now when I try to start up, so it's possible this could be a pre-elasticsearch issue. Nonetheless, I will use your above suggestions and see what I can come up with. I will revert shortly. – Stpete111 Sep 19 '18 at 12:51
  • Arnav - fyi, this github post you linked to is not similar to my problem. His Elasticsearch service will actually start. Mine will not. I can't make an API call to check the status of X-pack if ElasticSearch service isn't running. Ok now I will try your other suggestions. Just FYI, I have restarted the system after every change I've made to try to fix that - so I do that by default. – Stpete111 Sep 19 '18 at 13:13
  • 1
    Arnav - increasing the heap size in jvm.options allowed me to start the service! Thanks for this. I'm marking this as answer. However, solving this problem just brought me to the next one: https://stackoverflow.com/questions/52413413/elasticsearch-service-starts-but-can-not-be-reached-and-does-not-do-any-logging – Stpete111 Sep 19 '18 at 20:02
  • Great to hear this problem is solved. Let me check the next one. – ArnavRay Sep 19 '18 at 20:06