1

I have a matrix pipeline job which performs multiple stages (something like ~200) most of which are functional tests whose results are recorded by the following code:

        stage('Report') {
            script {
               def summary = junit allowEmptyResults: true, testResults: "**/artifacts/${product}/test-reports/*.xml"
               def buildURL = "${env.BUILD_URL}"
               def TestAnalyzer = buildURL.replace("/${env.BUILD_NUMBER}", "/test_results_analyzer")
               def TestsURL = buildURL.replace("job/${env.JOB_NAME}/${env.BUILD_NUMBER}", "blue/organizations/jenkins/${env.JOB_NAME}/detail/${env.JOB_NAME}/${env.BUILD_NUMBER}/tests")
               slackSend (
                  color: summary.failCount == 0 ? 'good' : 'warning',
                  message: "*<${TestsURL}|Test Summary>* for *${env.JOB_NAME}* on *${env.HOSTNAME} - ${product}* <${env.BUILD_URL}| #${env.BUILD_NUMBER}> - <${TestAnalyzer}|${summary.totalCount} Tests>, Failed: ${summary.failCount}, Skipped: ${summary.skipCount}, Passed: ${summary.passCount}"
               )
            }
        }

The problem is that this Report stage regularly fails with the following error:

> Archive JUnit-formatted test results                        9m 25s
[2022-11-16T02:51:49.569Z] Recording test results
Java heap space

I have increased the heap space of the jenkins server to 8GB by modifying the systemd service configuration this way:

software-dev@magnet:~$ sudo cat /etc/systemd/system/jenkins.service.d/override.conf
[Service]
Environment="JAVA_OPTS=-Djava.awt.headless=true -Xmx8g"

which was taken into account, because I verified with the following command:

software-dev@magnet:~$ tr '\0' '\n' < /proc/$(pidof java)/cmdline
/usr/bin/java
-Djava.awt.headless=true
-Xmx10g
-jar
/usr/share/java/jenkins.war
--webroot=/var/cache/jenkins/war
--httpPort=8080

I just increased the Heap size to 10GB and I'll wait for the result of this night's build, but I have the feeling that this amount of Heap space really looks excessive, so I'm suspecting that a plugin, maybe the JUnit one, may be buggy and could consume too much memory.

Is anyone aware of such a thing? Could there be workarounds?

More importantly, which methods could I use to try to track if one plugin is consuming too much? I have notions of Java since my CS degree, but I'm not familiar with the jenkins development ecosystem.

Thank you by advance.

ncarrier
  • 433
  • 3
  • 14
  • 1
    Looks ancient, and doubtly ever fixed: https://issues.jenkins.io/browse/JENKINS-8404, https://www.google.com/search?q=%22Recording+test+results%22+java+heap+space, https://stackoverflow.com/q/13381977/592355 (most promising workaround: `MAVEN_OPTS`!?) ;(;( – xerx593 Nov 16 '22 at 10:08
  • 1
    Yes I had seen this one. How much ancient it is, isn't really reassuring... Plus if the number of tests was the cause, then it a big issue, we don't have more than 500 tests, nothing comparable to the 40000 mentioned... And we're not using maven :( (I don't even know what it is ^^') – ncarrier Nov 16 '22 at 13:47

1 Answers1

0

You can try splitting the tests into chunks/batch/groups but this solution requires changes in the code.

More details

https://semaphoreci.com/community/tutorials/how-to-split-junit-tests-in-a-continuous-integration-environment

Grouping JUnit tests

Joao Vitorino
  • 2,976
  • 3
  • 26
  • 55
  • 1
    Thank you for your suggestion! Unfortunately, this is not a java project. It is a project which builds firmwares for products, on which, after the build and deployment, we do functional (or integration) testing with pytest. Those tests generate junit result files for jenkins integration. So I don't feel like the solution you proposed can apply to my situation, but I may have missed something. Also, the tests are grouped per matrix axis and the failures are in the per axis report stages. so in the end, there aren't so many tests per axis, probably not more than 150. – ncarrier Nov 18 '22 at 07:28