Puppetserver service fails to start

Question

I have a Vagrant CentOS VM running with ps.memory = 2048 RAM allocated.

When I try to start the puppetserver service:

$ puppet --version
4.4.0
$ sudo puppet resource service puppetserver ensure=running
Error: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
Error: /Service[puppetserver]/ensure: change from stopped to running failed: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
service { 'puppetserver':
  ensure => 'stopped',
}
$ journalctl -xn
No journal files were found.
$ systemctl status puppetserver.service
puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; disabled)
  Process: 4708 ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver (code=exited, status=0/SUCCESS)
 Main PID: 4709 (java);         : 4710 (bash)
   CGroup: /system.slice/puppetserver.service
           ├─4709 /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/...
           └─control
             ├─4710 /bin/bash /opt/puppetlabs/server/apps/puppetserver/ezbake-functions.sh wait_for_app
             └─4755 sleep 1

My JAVA_ARGS from /etc/sysconfig/puppetserver:

JAVA_ARGS="-Xms1g -Xmx1g -XX:MaxPermSize=1g"

As requested, the puppetserver.service file:

$ cat /usr/lib/systemd/system/puppetserver.service
[Unit]
Description=puppetserver Service
After=syslog.target network.target

[Service]
Type=simple
EnvironmentFile=/etc/sysconfig/puppetserver
User=puppet
TimeoutStartSec=120
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5

PermissionsStartOnly=true
ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver

ExecStart=/usr/bin/java $JAVA_ARGS \
          '-XX:OnOutOfMemoryError=kill -9 %%p' \
          -Djava.security.egd=/dev/urandom \
          -cp "${INSTALL_DIR}/puppet-server-release.jar" clojure.main \
          -m puppetlabs.trapperkeeper.main \
          --config "${CONFIG}" \
          -b "${BOOTSTRAP_CONFIG}" $@

KillMode=process

ExecStartPost=/bin/bash "${INSTALL_DIR}/ezbake-functions.sh" wait_for_app

SuccessExitStatus=143

StandardOutput=syslog

[Install]
WantedBy=multi-user.target

An attempt at running the ExecStartPost command by hand:

$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

RuntimeError: Got 2 failure(s) while initializing: File[/var/log/puppetlabs/puppetserver]: change from 0700 to 0750 failed: failed to set mode 0700 on /var/log/puppetlabs/puppetserver: Operation not permitted - No message available; File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available

So I tried again, but this time I changed some directory permissions, but still similar error (which doesn't make sense given I just changed the mode?):

$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/
$ sudo chown -R vagrant:vagrant /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/

$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0
RuntimeError: Got 1 failure(s) while initializing: File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available
                use at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1007

What could be the issue?

_"See 'systemctl status puppetserver.service' and 'journalctl -xn' for details."_ -- what do these say? — Jim Garrison, Mar 17 '16 at 21:14
I have shown the results of both of those commands in the question... — lollercoaster, Mar 17 '16 at 21:15
@JimGarrison the output of those two commands are right below — fge, Mar 17 '16 at 21:15
The full command to start the puppet server (that of `ExecStart`) — fge, Mar 17 '16 at 21:20
@fge: done, and included output in question. Great lead - some permissions issues, but not sure why. Running that command with sudo yields no errors (and simply hangs forever), but `sudo puppet resource service puppetserver ensure=running` fails in another SSH session, so it doesn't look like that is success either. — lollercoaster, Mar 17 '16 at 22:05
Can you try and run a simple jar with your Java invocation and the JVM args instead of the puppet server jar? Does that make a difference? Ie, try to run `/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -jar someVeryBasic.jar` — fge, Mar 17 '16 at 22:21
Hm, I don't have any jars - any simple one I can download somewhere? — lollercoaster, Mar 17 '16 at 22:35
why do you say this is an `OnOutOfMemoryError` ? Doesn't look like it from your post. — Daniele Segato, Mar 23 '16 at 11:29
I thought it was but it appears it may not be, you are correct — lollercoaster, Mar 23 '16 at 20:23

score 2 · Answer 1 · answered Mar 18 '16 at 02:03

2

Are you certain that it’s an OnOutOfMemory error? I ask because I found that the latest PuppetServer includes a newer version of logback, as shown by this message in /var/log/messages:

Mar 18 01:56:21 puppetserver java: Exception in thread "main" java.lang.AbstractMethodError: ch.qos.logback.core.net.SyslogAppenderBase.createOutputStream()Lch/qos/logback/core/net/SyslogOutputStream;
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.core.net.SyslogAppenderBase.start(SyslogAppenderBase.java:62)
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.classic.net.SyslogAppender.start(SyslogAppender.java:48)

If you see, this replace “classic.net.Syslog” in logback.xml with "core.net.Syslog"

sed -i_old -e 's/classic.net.Syslog/core.net.Syslog/' /etc/puppetlabs/puppetserver/logback.xml

If that's not the problem, please post your logfiles.

answered Mar 18 '16 at 02:03

Jo Rhett

141
5

Ah, yes I think you may be right. The problem could be that the `ExecStartPre` command is chowning the `/var/run/puppetlabs/puppetserver` to be user=puppet, group=puppet. I see that when I 1) chown to `vagrant:vagrant`, 2) try to start, 3) it continues to fail, 4) `/var/run/puppetlabs/puppetserver` is now owned by `puppet:puppet` (which would fit with perms errors above). The error still looks the same as when I posted in the question above however. – lollercoaster Mar 18 '16 at 17:42
I should add that commenting out the `ExecStartPre` line in env file doesn't seem to fix either. – lollercoaster Mar 18 '16 at 17:45
And running the `ExecStart` command with `sudo` still yields error: `RuntimeError: Got 2 failure(s) while initializing: File[/var/log/puppetlabs/puppetserver]: change from 0700 to 0750 failed: failed to set mode 0700 on /var/log/puppetlabs/puppetserver: Operation not permitted - No message available; File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available`. – lollercoaster Mar 18 '16 at 17:49

score 2 · Answer 2 · edited May 23 '17 at 12:06

You have provided log as

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0

So, it is clear that you are using jdk 8 which removed permgen space.

The Permanent Generation (PermGen) space has completely been removed and is kind of replaced by a new space called Metaspace. The consequences of the PermGen removal is that obviously the PermSize and MaxPermSize JVM arguments are ignored and you will never get a java.lang.OutOfMemoryError: PermGen error.

So please remove the -XX:MaxPermSize=1g portion from JAVA_ARGS of location /etc/sysconfig/puppetserver

JAVA_ARGS="-Xms1g -Xmx1g"

So

And then,

your command will be look like below:

$ /usr/bin/java -Xms1g -Xmx1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

or

$ /usr/bin/java -Xms1g -Xmx1g -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

Please try this 2 commands. I hope both will run successfully. For caution purpose, I have added both.

N.B: There is no need to change the file permission. keep them as before you using.

Otherwise, if you don't want to change anything, you can downgrade java 8 to java 7

Related Link:

score 1 · Answer 3 · answered Mar 26 '16 at 17:51

@lollercoster, you're starting your service with:

sudo puppet resource service puppetserver ensure=running

But your next command is not telling which user it is

/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

Please run whoami directly before your run the above command:

whoami
/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

Why this commands? I am pretty confident that your issue is a permission/usercontext issue and following commands make it more worse:

$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/
$ sudo chown -R vagrant:vagrant /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/

with the above cases you have to do a sudo on the right user context to have the files writable, but it depends if the other data is writable. So you can give it a try, but I would use the puppet user created for the service

User=puppet

So your puppet service runs as user puppet, but your logfiles are under vagrant user control, and not writable for puppet user.

So I would also try to switch back to:

$ sudo chown -R puppet /var/run/puppetlabs/
$ sudo chown -R puppet /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/

Java

Also I suggest you NOT to mess around with the JVM heap parameters, except you know what you do. Since you do not need to enforce a lower and a max bound on Java VM heap since JDK5. I would only limit to the upper bound

-Xmx1g

The heap and the real allocated mem will be managed well by the javaVM. As JVM needs more it will step by step increase the heap and keep the needed size (no downsizing to a lower value will occur). So you will have a better used RAM on your machine.

Also please switch to the Oracle JVM. I am facing often issues with security and Compatibility with OpenJDK. So my first test is to run it on the latest needed JDK of Oracle. In your case Oracle JDK8. But please try first the permission things I mentioned before.

Good luck, and please keep me updated.

score 0 · Answer 4 · answered Sep 29 '16 at 14:08

I have seen this behaviour before. Do not try to run the commands manually as root, because you will mess up the permissions. Instead read the logs carefully with journalctl -xe and try to understand why the service is failing. In my case I could not start the puppetserver service because there was already a private key, but not yet a public certificate.

ls /etc/puppetlabs/puppet/ssl/certs/`hostname -f`.pem
ls /etc/puppetlabs/puppet/ssl/private_keys/`hostname -f`.pem

If one of them is present without the other one, the service will fail to start up. You will see something like this in the log file:

Exception in thread "main" java.lang.IllegalStateException: Cannot initialize master with partial state; need all files or none.

If you are trying to install puppetserver for the first time, so there is no way to break an existing deployment, then you can probably just remove the exisiting private key and puppetserver will automatically generate a new pair.

rm /etc/puppetlabs/puppet/ssl/certs/`hostname -f`.pem
rm /etc/puppetlabs/puppet/ssl/private_keys/`hostname -f`.pem

And finally try again to start the puppetserver service.

service puppetserver start

score 0 · Answer 5 · edited Sep 17 '18 at 07:47

0

/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g

Xms and Xms are less than MaxPermSize. this worked for me.

edited Sep 17 '18 at 07:47

Vineeth Sai

3,389
7
23
34

answered Sep 17 '18 at 06:41

Manula Doratiyawa

1

Puppetserver service fails to start

5 Answers5