Jenkins slave jobs failing on "Unexpected termination of channel"

Question

I am currently seeing a set of errors across my builds.

Is this expected behaviour if you loose Jenkins (say to a box crash, or a kill -9)?

Or is there something worse going on (like a bad network connection)?

The stack and error is:

hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
        at $Proxy175.join(Unknown Source)   at
hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861)     at
hudson.Launcher$ProcStarter.join(Launcher.java:345)     at
hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
        at
hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
        at
hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
        at hudson.model.Build$RunnerImpl.build(Build.java:178)  at
hudson.model.Build$RunnerImpl.doRun(Build.java:139)     at
hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473)
        at hudson.model.Run.run(Run.java:1410)  at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)     at
hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:238) Caused by:
hudson.remoting.RequestAbortedException: java.io.IOException:
Unexpected termination of the channel   at
hudson.remoting.Request.abort(Request.java:273)     at
hudson.remoting.Channel.terminate(Channel.java:732)     at
hudson.remoting.Channel$ReaderThread.run(Channel.java:1157) Caused by:
java.io.IOException: Unexpected termination of the channel  at
hudson.remoting.Channel$ReaderThread.run(Channel.java:1133) Caused by:
java.io.EOFException    at
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)

Just putting it out here, these kinds of issues are due to network. Since my jenkins server and slave are in the same network, I edited my /etc/hosts file and added DNS entry for jenkins server with private IP. That solved the issue for me. Also, I tried to equate the java servers on both my master and slave. restarted both master and slave. — Luv33preet, Jun 10 '22 at 13:29

score 47 · Accepted Answer · edited Jun 22 '16 at 12:45

47

You'll see that error if the Jenkins master loses connectivity with the slave. It could be due to any of the three issues you listed above:

Manually killing the slave process
The slave server becoming unavailable
A network problem between the master and the slave

edited Jun 22 '16 at 12:45

HRgiger

2,750
26
37

answered Apr 03 '12 at 17:41

gareth_bowles

20,760
5
52
82

44

If it's simply slow response from the slave, try turning off the response time check as a temporary fix.Go to Manage Jenkins > Manage Nodes > Configure, and uncheck the "Response time" box. That will get you through until you can give it more resources if it's just a slow slave. – Joe McMahon Jun 19 '13 at 19:37
1

@gareth_bowles, How can we say it as a network problem between the master and the slave – Karthi Aug 30 '17 at 12:08

score 8 · Answer 2 · edited Apr 04 '16 at 05:22

8

I resolved it by a simple restart of the master Jenkins server.

edited Apr 04 '16 at 05:22

Robert Brooker

2,148
24
22

answered Apr 30 '15 at 14:32

schoenk

824
3
13
31

Don't know why someone downvoted you. This was that solution for me. https://groups.google.com/forum/#!topic/jenkinsci-users/ZsNnfs5bEAs – Robert Brooker Apr 04 '16 at 04:13
This also worked for me, jenkins 2.289.3 with nodes running java 8 – joonasj Sep 28 '21 at 07:45

score 5 · Answer 3 · answered Aug 13 '13 at 05:42

5

Since 1.520, Jenkins requires Java 6 or later, one of the possibility of this error is that you dont have java 6 or later on slave/remote. Resolution is to update your remote machine with java 6 or later or set correct environment variable which SSH uses while it logs in into slave machine.

answered Aug 13 '13 at 05:42

Halim

2,090
2
16
9

Dear Halim, you just made my day. – Julien B. May 15 '14 at 09:04
Thanks. I was going to leave a note about this but you had the answer. I switched from JDK 6 to 7 which solved the issue. The troubleshooting steps were to look at the Jenkins slave log (on the jenkins master server) which indicated class not found errors which led me down the path of trying a different jdk. – leeman24 Feb 14 '17 at 15:51

score 3 · Answer 4 · answered Apr 26 '12 at 10:16

Another thing where you can see this error (which actually leads to this) is lack of memory/cpu cores for build. I spent some time testing it, and resolved this issue on our site by increasing java heap size (and as well adding more memory to machines), increasing number of cores (so one machine have at least 4G of RAM with two cores available with one build slot), and also do some tuning (if you are on linux) number of open file descriptors and processes which can be run concurrently.

Madhusuthanan Seetharam · Answer 5 · 2014-03-28T17:39:35.797

3

For me the node configuration had a wrong "Remote FS Root". So check if all the config parameters on the master for this slave node are correct. Jenkins doesn't complain a thing but terminates the connection.

I struggled a lot before determining it was the node config. At least the error message on the master should be more explicit or informative.

edited Mar 28 '14 at 17:39

answered Jun 13 '13 at 13:48

Madhusuthanan Seetharam

779
6
12

1

is there a way to determine if the Remote FS Root is correct? – fuzzi May 29 '18 at 15:09

score 3 · Answer 6 · answered Nov 29 '13 at 10:26

The version of java has to be 1.6 (latest) or newer for Jenkins 1.531 that I am using.

For one of my slave nodes I found that I had to set the Java path in the 'Launch method' Advanced button 'JavaPath' for the node before this would work correctly. The issue was that the node kept using an older version of Java which I was not able to upgrade.

score 2 · Answer 7 · answered Nov 30 '17 at 13:11

IF you are using aws ELB for jenkins master and jenkins slave communication ELB Idle time is causing this issue. Please go and update your idle time using http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html#config-idle-timeout-console

Set it to more than 60 seconds

score 1 · Answer 8 · answered Aug 07 '13 at 21:02

I know this question is kind of old but I came across this problem the other day.

For me the problem was occurring during the install of the JDK. The install failed because the security settings of Windows were set so the notification box that says: "You need to give permission for this operation" was popping up, causing the install to fail.

I changed the notification settings to "Never notify" and it solved this issue.

score 1 · Answer 9 · answered Nov 28 '17 at 23:56

I had similar issues with Jenkins Slaves terminating when I upgraded Jenkins to version 2.9.2 from 2.1.5.

All issues related to upgrade like unexpected termination of Slave fixed by installing Java 8 version on Centos.

yum install java-1.8.0-openjdk-devel

orodbhen · Answer 10 · 2017-11-17T18:45:49.827

On CentOS Linux 7, I found that the Jenkins slave doesn't work with OpenJDK. I tried Java 7 and 8. It requires Oracle Java.

So, if you're on Linux, install Oracle Java, and set the path to point to it. You can do this in a number of ways, e.g.:

Set $JAVA_HOME for the user you're launching the slave as, and add it to $PATH.
Use update-alternatives to set the default java.
Set JavaPath, under the node's Advanced settings in Jenkins.

Jenkins slave jobs failing on "Unexpected termination of channel"

10 Answers10

Linked