Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap

Question

Solved: Thanks to below answer from S.Richmond. I needed to unset all stored maps of the groovy.json.internal.LazyMap type which meant nullifying the variables envServers and object after use.

Additional: People searching for this error might be interested to use the Jenkins pipeline step readJSON instead - find more info here.

---

I am trying to use Jenkins Pipeline to take input from the user which is passed to the job as json string. Pipeline then parses this using the slurper and I pick out the important information. It will then use that information to run 1 job multiple times in parallel with differeing job parameters.

Up until I add the code below "## Error when below here is added" the script will run fine. Even the code below that point will run on its own. But when combined I get the below error.

I should note that the triggered job is called and does run succesfully but the below error occurs and fails the main job. Because of this the main job does not wait for the return of the triggered job. I could try/catch around the build job: however I want the main job to wait for the triggered job to finish.

Can anyone assist here? If you need anymore information let me know.

Cheers

def slurpJSON() {
return new groovy.json.JsonSlurper().parseText(BUILD_CHOICES);
}

node {
  stage 'Prepare';
  echo 'Loading choices as build properties';
  def object = slurpJSON();

  def serverChoices = [];
  def serverChoicesStr = '';

  for (env in object) {
     envName = env.name;
     envServers = env.servers;

     for (server in envServers) {
        if (server.Select) {
            serverChoicesStr += server.Server;
            serverChoicesStr += ',';
        }
     }
  }
  serverChoicesStr = serverChoicesStr[0..-2];

  println("Server choices: " + serverChoicesStr);

  ## Error when below here is added

  stage 'Jobs'
  build job: 'Dummy Start App', parameters: [[$class: 'StringParameterValue', name: 'SERVER_NAME', value: 'TestServer'], [$class: 'StringParameterValue', name: 'SERVER_DOMAIN', value: 'domain.uk'], [$class: 'StringParameterValue', name: 'APP', value: 'application1']]

}

Error:

java.io.NotSerializableException: groovy.json.internal.LazyMap
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:569)
    at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
    at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
    at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
    at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
    at java.io.ObjectOutputStream.writeObject(Unknown Source)
    at java.util.LinkedHashMap.internalWriteEntries(Unknown Source)
    at java.util.HashMap.writeObject(Unknown Source)
...
...
Caused by: an exception which occurred:
    in field delegate
    in field closures
    in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@5288c

Answer 1

Use JsonSlurperClassic instead.

Since Groovy 2.3 (note: Jenkins 2.7.1 uses Groovy 2.4.7) JsonSlurper returns LazyMap instead of HashMap. This makes new implementation of JsonSlurper not thread safe and not serializable. This makes it unusable outside of @NonDSL functions in pipeline DSL scripts.

However you can fall-back to groovy.json.JsonSlurperClassic which supports old behavior and could be safely used within pipeline scripts.

Example

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node('master') {
    def config =  jsonParse(readFile("config.json"))

    def db = config["database"]["address"]
    ...
}    

ps. You still will need to approve JsonSlurperClassic before it could be called.

Answer 2

I ran into this myself today and through some bruteforce I've figured out both how to resolve it and potentially why.

Probably best to start with the why:

Jenkins has a paradigm where all jobs can be interrupted, paused and resumable through server reboots. To achieve this the pipeline and its data must be fully serializable - IE it needs to be able to save the state of everything. Similarly, it needs to be able to serialize the state of global variables between nodes and sub-jobs in the build, which is what I think is happening for you and I and why it only occurs if you add that additional build step.

For whatever reason JSONObject's aren't serializable by default. I'm not a Java dev so I cannot say much more on the topic sadly. There are plenty of answers out there about how one may fix this properly though I do not know how applicable they are to Groovy and Jenkins. See this post for a little more info.

How you fix it:

If you know how, you can possibly make the JSONObject serializable somehow. Otherwise you can resolve it by ensuring no global variables are of that type.

Try unsetting your object var or wrapping it in a method so its scope isn't node global.

Answer 3

EDIT: As pointed out by @Sunvic in the comments, the below solution does not work as-is for JSON Arrays.

I dealt with this by using JsonSlurper and then creating a new HashMap from the lazy results. HashMap is Serializable.

I believe that this required whitelisting of both the new HashMap(Map) and the JsonSlurper.

@NonCPS
def parseJsonText(String jsonText) {
  final slurper = new JsonSlurper()
  return new HashMap<>(slurper.parseText(jsonText))
}

Overall, I would recommend just using the Pipeline Utility Steps plugin, as it has a readJSON step that can support either files in the workspace or text.

Answer 4

I want to upvote one of the answer: I would recommend just using the Pipeline Utility Steps plugin, as it has a readJSON step that can support either files in the workspace or text: https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readjson-read-json-from-files-in-the-workspace

script{
  def foo_json = sh(returnStdout:true, script: "aws --output json XXX").trim()
  def foo = readJSON text: foo_json
}

This does NOT require any whitelisting or additional stuff.

Answer 5

This is the detailed answer that was asked for.

The unset worked for me:

String res = sh(script: "curl --header 'X-Vault-Token: ${token}' --request POST --data '${payload}' ${url}", returnStdout: true)
def response = new JsonSlurper().parseText(res)
String value1 = response.data.value1
String value2 = response.data.value2

// unset response because it's not serializable and Jenkins throws NotSerializableException.
response = null

I read the values from the parsed response and when I don't need the object anymore I unset it.

Answer 6

A slightly more generalized form of the answer from @mkobit which would allow decoding of arrays as well as maps would be:

import groovy.json.JsonSlurper

@NonCPS
def parseJsonText(String json) {
  def object = new JsonSlurper().parseText(json)
  if(object instanceof groovy.json.internal.LazyMap) {
      return new HashMap<>(object)
  }
  return object
}

NOTE: Be aware that this will only convert the top level LazyMap object to a HashMap. Any nested LazyMap objects will still be there and continue to cause issues with Jenkins.

Answer 7

You can use the following function to convert LazyMap to a regular LinkedHashMap (it will keep the order of original data):

LinkedHashMap nonLazyMap (Map lazyMap) {
    LinkedHashMap res = new LinkedHashMap()
    lazyMap.each { key, value ->
        if (value instanceof Map) {
            res.put (key, nonLazyMap(value))
        } else if (value instanceof List) {
            res.put (key, value.stream().map { it instanceof Map ? nonLazyMap(it) : it }.collect(Collectors.toList()))
        } else {
            res.put (key, value)
        }
    }
    return res
}

... 

LazyMap lazyMap = new JsonSlurper().parseText (jsonText)
Map serializableMap = nonLazyMap(lazyMap);

or better use a readJSON step as noticed in earlier comments:

Map serializableMap = readJSON text: jsonText

Answer 8

According to best practices posted on Jenkins blog (Pipeline scalability best practice), it is strongly recommended to use command-line tools or scripts for this kind of work :

Gotcha: especially avoid Pipeline XML or JSON parsing using Groovy’s XmlSlurper and JsonSlurper! Strongly prefer command-line tools or scripts.

i. The Groovy implementations are complex and as a result more brittle in Pipeline use.

ii. XmlSlurper and JsonSlurper can carry a high memory and CPU cost in pipelines

iii. xmllint and xmlstartlet are command-line tools offering XML extraction via xpath

iv. jq offers the same functionality for JSON

v. These extraction tools may be coupled to curl or wget for fetching information from an HTTP API

Thus, it explains why most solutions proposed on this page are blocked by default by Jenkins security script plugin's sandbox.

The language philosophy of Groovy is closer to Bash than Python or Java. Also, it means it's not natural to do complex and heavy work in native Groovy.

Given that, I personally decided to use the following :

sh('jq <filters_and_options> file.json')

See jq Manual and Select objects with jq stackoverflow post for more help.

This is a bit counter intuitive because Groovy provides many generic methods that are not in the default whitelist.

If you decide to use Groovy language anyway for most of your work, with sandbox enabled and clean (which is not easy because not natural), I recommend you to check the whitelists for your security script plugin's version to know what are your possibilities : Script security plugin whitelists

Answer 9

The way pipeline plugin has been implemented has quite serious implications for non-trivial Groovy code. This link explains how to avoid possible problems: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#serializing-local-variables

In your specific case I'd consider adding @NonCPS annotation to slurpJSON and returning map-of-maps instead of JSON object. Not only the code looks cleaner, but it's also more efficient, especially if that JSON is complex.

Answer 10

The other ideas in this post were helpful, but not quite all I was looking for - so I extracted the parts that fit my need and added some of my own magix...

def jsonSlurpLaxWithoutSerializationTroubles(String jsonText)
{
    return new JsonSlurperClassic().parseText(
        new JsonBuilder(
            new JsonSlurper()
                .setType(JsonParserType.LAX)
                .parseText(jsonText)
        )
        .toString()
    )
}

Yes, as I noted in my own git commit of the code, "Wildly-ineffecient, but tiny coefficient: JSON slurp solution" (which I'm okay with for this purpose). The aspects I needed to solve:

1. Completely get away from the java.io.NotSerializableException problem, even when the JSON text defines nested containers
2. Work for both map and array containers
3. Support LAX parsing (the most important part, for my situation)
4. Easy to implement (even with the awkward nested constructors that obviate @NonCPS)

Answer 11

Noob mistake on my part. Moved someones code from a old pipeline plugin, jenkins 1.6? to a server running the latest 2.x jenkins.

Failed for this reason: "java.io.NotSerializableException: groovy.lang.IntRange" I kept reading and reading this post multiple times for the above error. Realized: for (num in 1..numSlaves) { IntRange - non-serializable object type.

Rewrote in simple form: for (num = 1; num <= numSlaves; num++)

All is good with the world.

I do not use java or groovy very often.

Thanks guys.

Answer 12

I found more easy way in off docs for Jenkins pipeline

Work example

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

@NonCPS
def jobs(list) {
    list
        .grep { it.value == true  }
        .collect { [ name : it.key.toString(),
                      branch : it.value.toString() ] }

}

node {
    def params = jsonParse(env.choice_app)
    def forBuild = jobs(params)
}

Due to limitations in Workflow - i.e., JENKINS-26481 - it's not really possible to use Groovy closures or syntax that depends on closures, so you can't > do the Groovy standard of using .collectEntries on a list and generating the steps as values for the resulting entries. You also can't use the standard > Java syntax for For loops - i.e., "for (String s: strings)" - and instead have to use old school counter-based for loops.

Answer 13

If you can't use JsonSlupurClassic, here a simple way to transform LazyMap to LinkedHashMap.

def deserialize(String jsonStr) {
    def obj = new JsonSlurper().parseText(jsonStr)
    if (obj instanceof Map) {
        return cloneMap(obj)
    } else if (obj instanceof List) {
        return cloneList(obj)
    } else {
        return obj
    }
    return
}

def cloneMap(Map map) {
    def res = [:]
    map.each {e ->
        def key = e.getKey()
        def value = e.getValue()
        if (value instanceof Map) {
            res.put(key, cloneMap(value))
        } else if (value instanceof List) {
            res.put(key, cloneList(value))
        } else {
            res.put(key, value)
        }
    }
    return res
}

def cloneList(List list) {
    def res = []
    list.each {i ->
        if (i instanceof List) {
            res.add(cloneList(i))
        } else if (i instanceof Map) {
            res.add(cloneMap(i))
        } else {
            res.add(i)
        }
    }
    return res
}