16

I upgraded from jenkins 2.219 to 2.272 (latest version as of this writing) and now getting the stack trace below when Jenkins starts.

Jenkins docs says that this happens due to the Configuration as Code plugin and to set the jvm args as -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000. The docs also say to increment the value until the error goes away but so far I'm at 480000 and still getting the error. I also don't see that I have the Configuration as Code plugin installed anyway.

How can this be fixed?

java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded
    at jenkins.model.Jenkins.save(Jenkins.java:3379)
    at jenkins.model.Jenkins.saveQuietly(Jenkins.java:3398)
    at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2637)
    at jenkins.model.Jenkins$16.run(Jenkins.java:3342)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:50)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1162)
    at jenkins.model.Jenkins.<init>(Jenkins.java:962)
    at hudson.model.Hudson.<init>(Hudson.java:85)
    at hudson.model.Hudson.<init>(Hudson.java:81)
    at hudson.WebAppMain$3.run(WebAppMain.java:295)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:312)
dukethrash
  • 1,449
  • 4
  • 15
  • 25
  • 1
    Me too facing the exact error – Developer404 Dec 28 '20 at 01:57
  • Same issue here as well... we also have Configuration as Code plugin and have tried ramping the delay up to stupidly long times. Having learned from a previous instance of this, have reverted to 2.267 war file from the 2.276 – houninym Jan 25 '21 at 12:51
  • fyi I had to "rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key" to upgrade then hit the IllegalStateException and did "yum downgrade https://archives.jenkins-ci.org/redhat/jenkins-2.255-1.1.noarch.rpm" to get jenkins working again – jamshid Sep 23 '21 at 19:23

9 Answers9

10

For anyone having this issue, yes downgrading to https://get.jenkins.io/war-stable/2.263.1/ first is required. Then restart the service, then upgrade all of your plugins, then try the LTS upgrade again, at least for me it worked just fine.

More details about the issue here: https://www.jenkins.io/doc/upgrade-guide/2.204/

SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded If you encounter this, you can tell the plugin to delay configuration for an amount of time to give Jenkins time to load the global configuration before the configuration is applied by the plugin.

To enable this set the io.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to a number of milliseconds to delay the initialisation. The required value will be dependant on aspects of your system (cpu/disk) and configuration, and how it can be found is mostly a trial and error. A suggestion would be to start with 5000 (5 Seconds) and then increment by 2000 (2 seconds) until you no longer exhibit the issue and finally add 1000 (1 second) for some extra safety. For example, to delay the configuration by 9 seconds you would use something like the following command java -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=9000 -jar jenkins.war. Exactly how and where you specify this option depends on the installation method used to install Jenkins

However for me the workaround didnt work without first downgrading, upgrading all plugins, then finally upgrading the core again.

Eric Weintraub
  • 967
  • 1
  • 10
  • 23
  • 1
    Thanks. simply replacing war file worked for me as someone had accidentally upgraded jenkins , when we had absolutely no need to. – tryingToLearn Sep 27 '21 at 06:43
  • 2
    That is a very very bad mindset to have. Jenkins always need to be up to date. Most of those updates are security issues. If you dont update Jenkins and I get on your network I can own the local administrator or root account on the box it is hosted on. Don't be the reason your source code is stolen or you face a ransomware attack, ALWAYS upgrade your stuff, its part of the management of the platform, if you dont have time to deal with it, dont use Jenkins. – Eric Weintraub Sep 28 '21 at 10:37
9

To avoid this issue beforehand when you actually have not installed the Casc plugin:

Update to https://github.com/jenkinsci/role-strategy-plugin/releases/tag/role-strategy-3.1 and you should be OK.

When jenkins is already in the faulty state:

download the hpi and drop the hpi file into JENKINS_HOME/plugins folder. https://updates.jenkins.io/download/plugins/role-strategy/

(information taken from: https://github.com/jenkinsci/configuration-as-code-plugin/issues/1531)

tobster
  • 155
  • 1
  • 8
8

I faced the same issue. I have downloaded the jenkins.war , the earlier version (2.263.1) from https://www.jenkins.io/download/ . Stopped the jenkins and replaced the war with my install directory(C:\Program Files\Jenkins). And started the jenkins.This works like a charm.

Developer404
  • 5,716
  • 16
  • 64
  • 102
5

You can download the hpi and drop the hpi file into JENKINS_HOME/plugins folder. Usually the jenkins plugin directory will be under - /var/lib/jenkins/plugins/.

The link to get Role strategy hpi plugin - https://updates.jenkins.io/latest/role-strategy.hpi. And restart Jenkins using this command: systemctl restart jenkins.

Tupteq
  • 2,986
  • 1
  • 21
  • 30
sambit
  • 339
  • 4
  • 12
3

My scenario was to upgrade the Jenkins to the latest version and then restore the data in it to work like it was working before.

While upgrading from Jenkins v2.263 to v2.303, I had the same error. The only thing which worked for me was to upgrade my global plugin i.e Role-based authorization strategy first, and then Jenkins. It worked like a crisp.

Sameed
  • 105
  • 2
  • 7
  • i still have same situation, when upgrading jenkins jenkins_2.263.4 to 2.289.3 or higher , is there any solution than down grading. i already have already upgraded role-strategy.hpi. to 3.2 . still no help . any other solution? – chandu Dec 12 '21 at 22:35
  • Hey chandu. I have already mentioned the solution other than downgrading the Jenkins version. You need to identify the plugin, which has the global view in your Jenkins. In my case, I was using a Role-based authorization strategy. I upgraded my plugin first, and then upgraded the Jenkins. The trick worked. – Sameed Dec 14 '21 at 08:12
  • thanks !, I already updated role-based authorization strategy plugin. But in my case, i have updated all other plugins that helped, i think its mainly matri-auth and ldap updates made difference, i guess. – chandu Dec 15 '21 at 20:48
2

We had the same issue after we upgraded to the latest version. After some searching we decided to go with a lower version which has LTS support, 2.263.1 in this case. That version worked instantly so I think this error is a bug.

Isegrimm
  • 21
  • 3
2

we has similar issue, resolved by downgrading the jenkins.war file downloaded from URL: https://get.jenkins.io/war-stable/2.263.1/

jayant kamthane
  • 21
  • 3
1

I have faced similar issue, and found that we don't need to downgrade version. Below are the steps I performed and it worked for me

  1. upgrade global plugin i.e Role-based authorization strategy first from Jenkins Console.

  2. Here are steps which I have performed: #on ubuntu, in /usr/share/jenkins:

sudo service jenkins stop
sudo mv jenkins.war jenkins.war.old
sudo wget https://updates.jenkins-ci.org/latest/jenkins.war
sudo chown -R jenkins:jenkins jenkins.war
sudo service jenkins start
Harsh Wardhan Gupta
  • 235
  • 3
  • 5
1

I have the same error on Jenkins 2.332.2 and updated plugin role-strategy v483.v17281966f5c3.
I had to download the version 3.2.0 of the plugin and put it directly in the jenkins plugins folder. (previously I remove the existing '.jpi' file)
Then restarting Jenkins /etc/init.d/jenkins restart, and the issue has gone.
Thank you for the intel!

Luc Demeester
  • 305
  • 2
  • 4