Exception in thread "main" java.io.StreamCorruptedException: invalid stream header: 0BDAACED

Question

I am trying to execute jenkins cli command from master.

ssh user@192.168.1.2 -C "/usr/bin/java -jar /home/user/slave.jar"

Getting following error:

<===[JENKINS REMOTING CAPACITY]===>Exception in thread "main" java.io.StreamCorruptedException: invalid stream header: 0BDAACED
  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:808)
  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
  at hudson.remoting.ChannelBuilder.makeTransport(ChannelBuilder.java:430)
  at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:389)
  at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:310)
  at hudson.remoting.Launcher.main(Launcher.java:528)
  at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:468)
  at hudson.remoting.Launcher.run(Launcher.java:242)
  at hudson.remoting.Launcher.main(Launcher.java:195)
ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected EOF
  at hudson.remoting.Request.abort(Request.java:303)
  at hudson.remoting.Channel.terminate(Channel.java:847)
  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:92)
  at ......remote call to ubuntu-slave(Native Method)
  at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
  at hudson.remoting.Request.call(Request.java:172)
  at hudson.remoting.Channel.call(Channel.java:780)
  at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:508)
  at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:381)
  at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:131)
  at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:253)
  at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unexpected EOF
  at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
  at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
ERROR: Connection terminated
java.io.IOException: Unexpected EOF
  at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:99)
  at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
ERROR: Process terminated with exit code 1

Please tell me what could be the problem?

Answer 1

The short answer

Jenkins' slave.jar communicates with the Jenkins server through the slave's stdin/stdout. You rely on ssh to transfer stdin/stdout to/from the slave.

Something that you are not aware of tampers with the stdin before it reaches the slave, and the resulting communication protocol violation causes the exception that you see.

The longer explanation and a fix

I suppose that you have the ssh command from your question, which starts the slave, in a shell script on the jenkins server, and that this shell script also has other commands in it that precede this command. Because this is how it was in my case, inspired by the jenkins remoting documetation that suggests you could do things like copying the correct slave.jar to the slave.

So we are both using the "Launch agent via execution of command on the server" option to start the slave agent. In my case I do this to be able to use an ssh jump host to reach the slave, but this is not of relevance to this answer.

I will now give examples of non-working and working shell scripts to start a slave.jar on a remote node, and then come up with reasoning of what might cause the observed behaviour. I use /bin/bash as the shell, feel free to use others.

1) These versions both work if you already have slave.jar on the slave

#!/bin/bash                                                                     
ssh user@host "java -jar /home/user/slave.jar"

.

#!/bin/bash                                                                     
exec ssh user@host "java -jar /home/user/slave.jar"

Both versions have only a single command in them. Nothing else is in here that tampers with stdin or stdout. In the first version, the shell lingers around and forwards stdin/stdout to/from the ssh command. In the second version, the shell is replaced by the ssh process and directly inherits its stdin/stdout. Both versions work ok, and the additional shell process in the first version should not matter on any system.

2) This version copies slave.jar from some location on the server to the slave before executing it on the slave

#!/bin/bash
scp -q /some/location/slave.jar user@host:.
exec ssh user@host "java -jar /home/user/slave.jar"

Of course, this only works if you have /some/location/slave.jar on the server.

3) This version tries to do some additional cleanup on the slave before starting the slave agent.

#!/bin/bash
ssh user@host "rm -rf /home/user/tmp/jenkins"
exec ssh user@host "java -jar /home/user/slave.jar"

The /home/user/tmp/jenkins location on the slave is just an example. This version fails after a 4 minute timeout. With the exact error message from the question. The failure is not caused by anything important missing from /home/user/tmp/jenkins, as you will see in the next example:

4) Different ways to make example 3 work

#!/bin/bash
ssh user@host "rm -rf /home/user/tmp/jenkins" </dev/null
exec ssh user@host "java -jar /home/user/slave.jar"

.

#!/bin/bash
exec ssh user@host "rm -rf /home/user/tmp/jenkins && java -jar /home/user/slave.jar"

In the first fix, we make sure stdin to communicate with the slave is not forwarded to the remote rm command. rm does not read its stdin, but apparently ssh does not know that and buffers some bytes from its own stdin and forwards them to the remote command just in case it is needed? This is fixed by forwarding /dev/null as stdin to the rm command instead of the stdin destined for the slave communication.

In the second fix, only one ssh command is used, the rm command again does not read anything from stdin, and the slave.jar receives the untampered stream.