Elasticsearch service is failing when /tmp is mounted with noexec. When I try to remount it with exec, the service is getting started.
Is the below link still valid or there is permanent solution or workaround?
https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html
Q. I would like to know how can we change the path of Elasticsearch logs which its trying to write in /tmp? (something like '/var/' or '/opt' which is already mounted as exec)
Q. Also can you please tell, if this logs are temporary and elasticsearch service deletes it after some time ? if yes then only it make sense to change path to something like '/var' or '/opt'
Below are the logs.
1.
systemctl status elasticsearch.service
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─startup-timeout.conf
Active: failed (Result: exit-code) since Fri 2020-11-20 14:52:12 IST; 16min ago
Docs: http://www.elastic.co
Process: 48882 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 48882 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service
Nov 20 14:51:34 <HOSTNAME> systemd[1]: Starting Elasticsearch...
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will ...release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable sy...own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.
journalctl -fu elasticsearch
-- Logs begin at Tue 2020-11-17 13:09:05 IST. --
Nov 20 14:51:38 <HOSTNAME> elasticsearch[48882]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: ERROR: [1] bootstrap checks failed
Nov 20 14:52:12 <HOSTNAME> elasticsearch[48882]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Failed to start Elasticsearch.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: Unit elasticsearch.service entered failed state.
Nov 20 14:52:12 <HOSTNAME> systemd[1]: elasticsearch.service failed.
/var/log/elasticsearch/<cluster-name.log>
[2020-11-20T14:51:53,369][WARN ][o.e.b.Natives ] [elasticsearch_1] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: /tmp/elasticsearch-9221628356394379624/jna--1985354563/jna10392232823311871934.tmp: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
at java.lang.System.load(System.java:1870) ~[?:?]
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:333) ~[?:?]
at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.4.0.jar:7.4.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.4.0.jar:7.4.0]
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives ] [elasticsearch_1] cannot check if running as root because JNA is not available
[2020-11-20T14:51:53,380][WARN ][o.e.b.Natives ] [elasticsearch_1] cannot install system call filter because JNA is not available
[2020-11-20T14:51:53,381][WARN ][o.e.b.Natives ] [elasticsearch_1] cannot register console handler because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives ] [elasticsearch_1] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives ] [elasticsearch_1] cannot getrlimit RLIMIT_AS because JNA is not available
[2020-11-20T14:51:53,382][WARN ][o.e.b.Natives ] [elasticsearch_1] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2020-11-20T14:51:53,612][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_1] using [1] data paths, mounts [[/opt (/dev/mapper/appvg-lv_opt)]], net usable_space [121.8gb], net total_space [147.5gb], types [ext4]
[2020-11-20T14:51:53,614][INFO ][o.e.e.NodeEnvironment ] [elasticsearch_1] heap size [5.1gb], compressed ordinary object pointers [true]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node ] [elasticsearch_1] node name [elasticsearch_1], node ID [6OyoyA40S3-z6M7nAnCGIA], cluster name [<cluster name>]
[2020-11-20T14:51:53,617][INFO ][o.e.n.Node ] [elasticsearch_1] version[7.4.0], pid[48882], build[default/rpm/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z], OS[Linux/3.10.0-1127.19.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13/13+33]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node ] [elasticsearch_1] JVM home [/usr/share/elasticsearch/jdk]
[2020-11-20T14:51:53,618][INFO ][o.e.n.Node ] [elasticsearch_1] JVM arguments [-Xms5288m, -Xmx5288m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-9221628356394379624, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Xms5288m, -Xmx5288m, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=2772434944, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-11-20T14:51:55,967][INFO ][o.e.p.PluginsService ] [elasticsearch_1] loaded module [aggs-matrix-stats]
[2020-11-20T14:51:55,968][INFO ][o.e.p.PluginsService ] [elasticsearch_1] loaded module [analysis-common]
org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,962][ERROR][o.e.x.m.c.c.StatsCollector] [elasticsearch_1] collector [ccr_stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:11,963][ERROR][o.e.x.m.c.i.IndexStatsCollector] [elasticsearch_1] collector [index-stats] failed to collect data
java.lang.NullPointerException: null
at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:85) [x-pack-monitoring-7.4.0.jar:7.4.0]
at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:242) [x-pack-monitoring-7.4.0.jar:7.4.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-11-20T14:52:12,114][INFO ][o.e.t.TransportService ] [elasticsearch_1] publish_address {<IP of localhost>:5300}, bound_addresses {0.0.0.0:5300}
[2020-11-20T14:52:12,123][INFO ][o.e.b.BootstrapChecks ] [elasticsearch_1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-11-20T14:52:12,128][ERROR][o.e.b.Bootstrap ] [elasticsearch_1] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-11-20T14:52:12,132][INFO ][o.e.n.Node ] [elasticsearch_1] stopping ...
[2020-11-20T14:52:12,155][INFO ][o.e.n.Node ] [elasticsearch_1] stopped
[2020-11-20T14:52:12,156][INFO ][o.e.n.Node ] [elasticsearch_1] closing ...
[2020-11-20T14:52:12,166][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch_1] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:66) ~[?:?]
at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:87) ~[?:?]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at org.elasticsearch.node.Node.close(Node.java:881) ~[elasticsearch-7.4.0.jar:7.4.0]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.4.0.jar:7.4.0]
at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:186) ~[elasticsearch-7.4.0.jar:7.4.0]
