11

My Jenkins build hangs between build and post-build steps.

The console output shows there is a 6-minute wait (but I've seen waits of up to one hour):

10:53:26 BUILD FAILED in 1m 7s
10:53:26 4 actionable tasks: 4 executed
10:53:26 Build step 'Invoke Gradle script' changed build result to FAILURE
10:53:26 Build step 'Invoke Gradle script' marked build as failure
11:09:29 [CucumberReport] Using Cucumber Reports version 4.9.0

I found this and this questions that have similar issues, and they say the solution is setting -DSoftKillWaitSeconds=0 in jenkins.xml.

However, I need a way to set the option for particular jobs only, without messing with global Jenkins settings (I wouldn't want to mess with other projects).

EDIT:

When I manually abort the job, before the [CucumberReport] step, Cucumber reports are still generated.

I also checked Abort the build if it's stuck checkbox in Build Environment options, with Time-out strategy set to No Activity (Timeout seconds = 2).

When I build the project with these settings, the build will fail with "Aborted after 0 seconds" shown in Build History, as before, but the console output will be the same. (Nothing changes, Cucumber Reports will be generated but after a certain timeout).

Mate Mrše
  • 7,997
  • 10
  • 40
  • 77
  • What do you want to achieve with the parameter? Maybe there is an easier solution for this. – Michael Kemmerzell Jan 02 '20 at 11:26
  • I want to cut wait time after the build phase. As you can see from the console output, there is a 6-minute wait between build and post-build steps. – Mate Mrše Jan 02 '20 at 11:31
  • Can you provide some parts of your pipeline? I doubt jenkins is simply stuck - something has to happen here. – Michael Kemmerzell Jan 02 '20 at 11:32
  • Can you be more specific on what parts of the pipeline would be of interest so I can edit and share some because I'm unable to share actual code? – Mate Mrše Jan 02 '20 at 11:39
  • Can't you just set a [`timeout`](https://jenkins.io/doc/book/pipeline/syntax/#options) option in your pipe? You should definitely post a simplified version of your pipe. It's hard to "guess" what might be wrong. Also, this question is probably better suited to be at https://devops.stackexchange.com/ – tftd Jan 08 '20 at 03:08
  • @tftd I'm new to Jenkins and I'm using a freestyle project so not sure about what do you exactly mean by "pipe". – Mate Mrše Jan 08 '20 at 07:42

2 Answers2

4

It is not possible to select a job-specific value for SoftKillWaitSeconds (the value is derived from the Jenkins core at a point where the job name is not known).

My recommendation is to fix the abort handling in your job itself, so it will not depend on a "soft kill timeout". If you're running on a Unix-ish system, you can ensure this by running your job in a new process group (set -m in bash) and (for example) setting up a proper exit trap.

Alex O
  • 7,746
  • 2
  • 25
  • 38
  • Since I'm new to Jenkins, could you expand on "running your job in a new process group (set -m in bash) and (for example) setting up a proper exit trap"? – Mate Mrše Jan 10 '20 at 11:00
  • This is unrelated to Jenkins per-se. You can use this approach if your job involves an "Execute shell" build step (or if you can change it so it does). Then use "set -m" to create a new process group, and use a trap like shown [here](https://stackoverflow.com/questions/360201/how-do-i-kill-background-processes-jobs-when-my-shell-script-exits) to ensure that all processes in the group will get killed. – Alex O Jan 10 '20 at 13:26
3

We are using the Build-timeout plugin to kill stuck jobs with timeout strategy set to No Activity or Absolute. For me, this is a good approach when you are using freestyle projects. The reason why your build is "Aborted after 0 seconds" is that most likely there are unfinished child processes. From documentation:

Because Java only allows threads to be interrupted at a set of fixed locations, depending on how a build hangs, the abort operation might not take effect. For example,

  • if Jenkins is waiting for child processes to complete, it can abort right away.
  • if Jenkins is stuck in an infinite loop, it can never be aborted.
  • if Jenkins is doing a network or file I/O within the Java VM (such as lengthy file copy or SVN update), it cannot be aborted.

You could try the absolute timeout strategy. You can define a global variable, so that you do not repeat the timeout value in jobs:

  1. Go to "Manage Jenkins" > "Configure System".
  2. Check "Environment variables" in "Global properties".
  3. Add an environment variable name="GLOBAL_TIMEOUT_MINUTES" value="20".
  4. Go to a configuration page of a project.
  5. Check "Abort the build if it's stuck" in "Build Environment".
  6. Select "Absolute" for "Time-out strategy". Of course, also applicable to other strategies.
  7. Set "${GLOBAL_TIMEOUT_MINUTES}" for "Timeout".
  8. Set timeout action "Abort the build".

If this is not working, you could try to look in the logs https://your-jenkins-server/log or in a thread dump.
The hanging may be caused by new/old version of a plugin. Try to find what are the unfinished child processes. Try to disable post-build actions one by one to find the one that may be the cause of the issue. You can see https://superuser.com/questions/1401879/debugging-what-happens-when-a-jenkins-build-unexpectedly-pauses-or-hangs

K. B.
  • 3,342
  • 3
  • 19
  • 32
  • Upvoted for useful info, however, doesn't solve my problem. I already tried build timeout plugin but with no success. – Mate Mrše Jan 13 '20 at 11:33