Jenkins pipeline java.io.NotSerializableException java.util.regex.Matcher error even with @NonCPS

Question

I am using @NonCPS in front of my Jenkinsfile function which performs a regex match and i'm still getting java.io.NotSerializableException java.util.regex.Matcher error even with the @NonCPS annotation.

Note, it calls the function many times and the exception only occurs once a match is actually made.

Here is my code:

@NonCPS
def extractEndTime(logLine) {
    def MY_REGEX = /.*(20[0-9]{2}-[0-9]{2}-[0-9]{2}).([0-9]{2}:[0-9]{2}:[0-9]{2}).*\"\w+\"\sthe text\s(\w+)\./
    m = (logLine =~ TEST_LOGLINE_END_REGEX)
    if (m.count) {
        return [m[1],m[2],m[3]]
    } else {
        return null
    }
}

The output when doing a jenkins build:

GitHub has been notified of this commit’s build result
java.io.NotSerializableException: java.util.regex.Matcher
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
    at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
    at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
    at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
    at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
    at java.util.LinkedHashMap.internalWriteEntries(LinkedHashMap.java:333)
    at java.util.HashMap.writeObject(HashMap.java:1354)
    at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
    at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
    at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
    at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
    at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
    at com.cloudbees.groovy.cps.SerializableScript.writeObject(SerializableScript.java:26)
    at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
    at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
    at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
    at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
    at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
    at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
    at java.util.HashMap.writeObject(HashMap.java:1354)
    at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
    at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:433)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:412)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:357)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: an exception which occurred:
    in field delegate
    in field closures
    in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@42b37962
Finished: FAILURE

score 51 · Answer 1 · edited Jun 20 '20 at 09:12

51

Jenkins require all variables to be serializable because the state of the pipeline is periodically saved to disk in case of interrupts like a server restarts. This feature allows pipelines to maintain their state and continue even after the server is restarted. Variables of type Matcher are not serializable and require some additional work by the developer.

From jenkinsci/pipeline-plugin Serializing Local Variables:

However the safest approach is to isolate use of nonserializable state inside a method marked with the annotation @NonCPS. Such a method will be treated as “native” by the Pipeline engine, and its local variables never saved.

The code example provided:

@NonCPS
def version(text) {
  def matcher = text =~ '<version>(.+)</version>'
  matcher ? matcher[0][1] : null
}

Additional material backing this can be found on Pipeline Groovy Plugin Technical Design, here they discuss more technical details and behavior of methods marked with @NonCPS.

Pipeline scripts may mark designated methods with the annotation @NonCPS. These are then compiled normally (except for sandbox security checks), and so behave much like “binary” methods from the Java Platform, Groovy runtime, or Jenkins core or plugin code. @NonCPS methods may safely use non-Serializable objects as local variables, though they should not accept nonserializable parameters or return or store nonserializable values. You may not call regular (CPS-transformed) methods, or Pipeline steps, from a @NonCPS method, so they are best used for performing some calculations before passing a summary back to the main script. Note in particular that @Overrides of methods defined in binary classes, such as Object.toString(), should in general be marked @NonCPS since it will commonly be binary code calling them.

See: Serializing Local Variables and Pipeline Groovy Plugin Technical Design

edited Jun 20 '20 at 09:12

Community

1
1

answered Jan 26 '18 at 16:30

SINGULARITY

1,124
11
11

Because `m` is declared the way it is (no `def` or anything to make it local), I believe it gets scoped as a "global" which is why the serialization exception happens. – mkobit Jan 26 '18 at 18:56
1

"Pipeline restricts all variables to Serializable types" - [Best Practices for Scalable Pipeline Code](https://jenkins.io/blog/2017/02/01/pipeline-scalability-best-practice/) – SINGULARITY Jan 26 '18 at 19:20
In the same documentation: _"While normal Pipeline is restricted to serializable local variables (see appendix at bottom), @NonCPS functions can use more complex, nonserializable types internally (for example regex matchers, etc). "_ - I am fairly confident it is because `m` is a [script global](https://stackoverflow.com/a/6315609/627727) and a non-serializable type is assigned to it in the scope of the `@NonCPS`, which is why I think the other answer is correct. – mkobit Jan 26 '18 at 19:39
1

I cannot find any documentation that says _script globals_ are annotated as `@NonCPS`. The Best Practices link from above does not list _scipt globals_ as common non-serializable types. Moreover, if scripted globals can be used as non-serializable types, they would not be safe to use unless they're scoped to a function that has been annotated with the `@NonCPS` attribute. – SINGULARITY Jan 26 '18 at 21:14
@SINGULARITY thanks for your answer and linking to that documentation. It's incredibly helpful! – Shea Jul 05 '18 at 14:53
1

I think using `@NonCPS` functions is the solution, perhaps you could add that to your answer instead of only linking to the solution; links tend to break. – Hans Wouters Jul 30 '18 at 14:01
this is happening to us only on parallel stages... When we moved them into synchronous execution, the error went away... other solution was to null the references of the matcher references before the method returns... – Marcello DeSales Jul 23 '19 at 23:47
Great tech answer. Saved my day - literally. – Marek R Oct 10 '19 at 11:15

score 46 · Accepted Answer · edited Jan 26 '18 at 16:32

46

Probably due to the m variable scope. Try restrict it with def like this:

def m = (logLine =~ TEST_LOGLINE_END_REGEX)

Without def a variable is created in a global script binding and hence still exists after exiting from the method.

edited Jan 26 '18 at 16:32

mkobit

43,979
12
156
150

answered Nov 07 '16 at 00:29

izzekil

5,781
2
36
38

@DanielPark, if this is a good answer, you should mark it as the accepted one and also up-voting is good practice :-) – Eldad Assis Nov 07 '16 at 09:45
7

I don't believe scoping is the issue here because Jenkins wants the pipeline to be be serializable and the `java.io.NotSerializableException: java.util.regex.Matcher` exception is thrown because the pipeline is trying to save the state of a variable that's not serializable. See my answer below. – SINGULARITY Jan 26 '18 at 17:50
3

I am already using `def` and yet I get this error. So far failing to understand what is causing this error. :( – haridsv Mar 14 '19 at 16:25
I have a couple of `def g = m.group(1)` sort of calls and if I comment them the error goes away. This is really weird! – haridsv Mar 14 '19 at 16:29
3

I actually made a wrong conclusion above. The `Matcher.group()` calls were not the issue, but the next line using the extracted fields, even if it was a simple echo, like `echo "g1=${g1} g2=${g2}"`. Since `g1` and `g2` are simple strings with no references to `Matcher`, this made no sense, but this is not the first time I saw unexplainable errors in Jenkins pipeline. The workaround was to push the code that used the `Matcher` into a function and return the groups from it. This got the issue resolved though I didn't declare it as `NonCPS`. – haridsv Mar 14 '19 at 17:33
Instead of creating a new function, I was able to avoid the error by never creating a variable for the Matcher. Instead I shoved all code involving the regex into one line. In my case it was: `def MYBUILDUSER = ("curl ${BUILD_URL}/api/json".execute().text =~ /"Started by (user )?(.*?)\"/)[-1][-1]` – twasbrillig May 05 '20 at 17:24
2

Just surprised that this answer is accepted by the question owner and upvoted by so many people. This answer doesn't resolve the issue at all. Just like @SINGULARITY commented, the reason is that Matcher is not serializable. I was really mislead by this answer. I suggest the question owner to remove the acceptance mark and downvote this answer. – Vespene Gas Aug 12 '21 at 08:15
THIS IS NOT A WORKING SOLUTION ! I have no idea why this one got accepted as it does NOT solve the problem. – Tobias Gierke Aug 08 '23 at 08:08

score 9 · Answer 3 · answered Aug 26 '21 at 11:16

Ran into this recently. One way to avoid this is to simply set the local matcher-object to null after extracting the parts you need into serializable types. Make sure it's null before doing additional steps/actions.

script {
    def label = 'v1.2.3'  // From git describe etc.
    def regex = '^v([0-9]+).([0-9]+).([0-9]+)$'
    def matcher = label =~ regex

    if (matcher) {
        def tag = matcher[0][0]
        def maj = matcher[0][1] as int
        def min = matcher[0][2] as int
        def patch = matcher[0][3] as int

        matcher = null
        echo "Found ${maj} ${min} ${patch} from ${tag}"
        // Additional steps
    } else {
        matcher = null
        echo "Nothing found"
    }
}

score 6 · Answer 4 · edited Jul 30 '20 at 05:57

6

The following works (note the [0]):

def m = (it =~ /abcd/)[0]

but this results in a java.io.NotSerializableException:

def m = (it =~ /abcd/)

edited Jul 30 '20 at 05:57

Kobi

135,331
41
252
292

answered Mar 22 '19 at 06:26

Mervyn Zhang

81
1
3

score 5 · Answer 5 · answered May 14 '18 at 13:22

5

Just for clarity, the following worked for me.

The extra work for the developer, as an example, see the code below.

my.Parameter is NOT Serializable

so I make a method, without a def and annotate it @NonCPS and call that from the node block

  node('MySys') {
    echo 'hello'
    avoidCPS()
    echo 'finished'
  }

  @NonCPS
  avoidCPS () {
    // use my.Parameter here
    my.Parameter p = new my.Parameter()
    ... do some more ...
  }

answered May 14 '18 at 13:22

Mike

381
1
4
12

do we must remove the 'def' statement when using non-serializable class? – Hobin C. Mar 20 '20 at 06:43

Jenkins pipeline java.io.NotSerializableException java.util.regex.Matcher error even with @NonCPS

5 Answers5