7

Plugins were updated on our Jenkins, after which Jenkins entered the "safe-shutdown" mode, blocking all of our nightly scheduled jobs. We don't know who (or "what") started the updates, and I cannot find any sort of log that refers to the plugin updates. All we know is the time they were updated (through "Date modified" on the actual plugin *.jpi files).

If it helps, we are using Active Directory authentication, with role-based rights management, so we actually know who (normal + service users) has rights on this, but no one seems to have done it, which makes me think it may have been triggered through script or scheduled task somewhere somehow.

Is there any way to find out who started the updates or how they were started? Thanks.

DevsCreateWorkForMe
  • 73
  • 1
  • 4

2 Answers2

0

The only place I found even a lit bit of information was in a Catalina log file.

catalina.2019-08-07.log:07-Aug-2019 16:37:10.695 INFO [Update center installer thread [#1]] hudson.model.UpdateCenter$DownloadJob.run Starting the installation of Credentials Binding on behalf of aakoch

I know you asked this 2 months ago, but I was having an issue also and found this question. So I'm adding what little info I have for the next person.

I'm going to add the stacktrace I had. Maybe it will show up in a search result.

Loading library jenkins-library@master
java.lang.NullPointerException
    at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
    at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
    at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: Loading libraries failed

1 error

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

Issue was that the information I had defined for the library got erased after doing some plugin update. I'm not sure which one. I just redefined the info and I was good to go.

aakoch
  • 1,164
  • 1
  • 9
  • 18
  • Hi Thank you so much. I did actually look at those logs before I first posted this, but it seems I was blind enough to just overlook those "Starting installation of..." messages - that is my fault. If you're interested, our log showed an obscure service user had started the updates. When brought up, one of our juniors confessed "Oh yeah, I changed that user in a scheduled task, because the one configured was obsolete... What does the task do? I don't know" Be assured, he has since been heavily ridiculed for this. Thank you again! – DevsCreateWorkForMe Aug 12 '19 at 11:38
  • Where would one find such a file? – Florian Straub Mar 09 '20 at 07:46
  • 1
    @FlorianStraub it really depends on how it's installed. Mine is tomcat/jenkins/logs – aakoch Mar 17 '20 at 13:26
0

Meanwhile we have a job running the python script below @daily in order to save the current plugin names and versions in a json file on the server:

import os, json, base64, urllib.request, ssl, datetime, time

request = urllib.request.Request("https://jenkins-server-name/pluginManager/api/json?depth=1")
#according to https://stackoverflow.com/a/28052583/4609258 the following is ugly
# ... but still ok if you basically access localhost, I think
context = ssl._create_unverified_context() 
base64string = base64.b64encode(bytes('%s:%s' % ('jenkins user name', '34 characters long api key'),'ascii'))
request.add_header("Authorization", "Basic %s" % base64string.decode('utf-8'))

with urllib.request.urlopen(request, context=context) as url:
    parsed = json.loads(url.read().decode())
    f = open(datetime.datetime.now().strftime("%Y-%m-%d_%H-%M") + "_plugins.json", "w")
    f.write(json.dumps(parsed, indent=4, sort_keys=True))
    f.close()

If there is a problem, one can compare these daily files with each other, to see what was changed when.

Florian Straub
  • 826
  • 9
  • 18