24

I have a JUnit test using Assumption to skip the test if the developer's computer doesn't have the pre-requisite software for running it. Despite being "junit", it's an integration test. Something like this:

int isSoftwarePresent = new ProcessBuilder("check software presence").start().waitFor();
Assume.assumeThat("Software not present", isSoftwarePresent, is(equalTo(0)));

However, at one point I realized the test had stopped running on the automated build on Jenkins, due to that assumption, and eventually a regression was introduced which the test was supposed to stop.

To put in other words, the required software went missing from Jenkins slave environment, which caused the test to be skipped.

The automated test is run by maven with the FailSafe plugin, on a Jenkins Pipeline build plan. How can I detect that my environment is Jenkins so that I can make the assumption condition more strict?

That is, I want the condition to be something like this:

boolean isJenkinsBuild = /* true if this is being run inside a Jenkins build, false otherwise */;
boolean isSoftwarePresent = new ProcessBuilder("check software presence").start().waitFor() == 0;
Assume.assumeTrue("Software not present", isSoftwarePresent || isJenkinsBuild);

Or even,

@Test
void testJenkinsEnvironment() {
    ...
    Assume.assumeTrue(isJenkinsBuild);
    Assert.assertTrue(isSoftwarePresent);
}

@Test
void testFeature() {
    ...
    Assume.assumeTrue(isSoftwarePresent);
    ...
}
Daniel C. Sobral
  • 295,120
  • 86
  • 501
  • 681
  • 3
    This sounds like you're over-engineering. Configure jenkins to automatically fail long running jobs after some time period. Also, have your Ops team install the required software on the CI server. EDIT: If your test is written to detect its runtime environment, how do you intend on testing the test? – Dave Jul 02 '18 at 20:35
  • @Dave Your comment is so confusing! The build isn't failing, it's passing because the test is being ignored. Maybe that's why what you say doesn't make sense to me? I've edited the question with more details. I don't think there is anything over-engineering there, but, if there is something, do point it out in the code samples. – Daniel C. Sobral Jul 02 '18 at 21:53
  • I'd love to debate this more, but it's rather a philosophical debate about test design, so not really within the remit of StackOverflow. Basically, I know your test isn't failing - it's being skipped. I'm suggesting that either a skipped test, or a long running test should fail the build - which would be reported on and issue regression wouldn't have occurred. The over-engineering is designing your tests to detect the test environment (and something car manufacturers got in a lot of trouble for). – Dave Jul 03 '18 at 19:13
  • @Dave Well, that can't be helped. That particular software dependency cannot be imposed on all developers working with that particular repository. – Daniel C. Sobral Jul 03 '18 at 19:30
  • Surely then, any developer that doesn't have the software installed, shouldn't be used as a jenkins slave for that particular build job? I honestly find it strange that a developer machine would be used as a jenkins slave, but that's a completely separate debate. – Dave Jul 03 '18 at 19:32
  • @Dave Developer machines are not used as Jenkins slaves, but they do have to run tests too. – Daniel C. Sobral Jul 09 '18 at 17:42
  • but your question clearly states "automated build on Jenkins... run by Maven... on a Jenkins pipeline...". If the developer machines lacking the required software are running tests, isn't this what the concept of Profiles was invented for? Software dependencies can (and should be) mandatory on Jenkins slaves, and those tests can be excluded within the Maven Profile for developer machines (at my office: mvn clean failsafe:integration-test -Pdev-int ). Conversely if your devs are particularly precious, you could enable a profile only on the CI builds. – Dave Jul 10 '18 at 19:12
  • @Dave Profiles could be used, but how would it target the test? File name pattern is precarious, and we replaced that with JUnit categories, but on JUnit 4 category selection doesn't work very well when there are many, and I would be wary of adding another one. Even if I did add another one, it would conflate the idea of "this is running on Jenkins" with "this test may be skipped in the absence of pre-requisites". So, unless I'm missing something, I'd still be using test-side mechanics to toggle the test even if I use a profile. – Daniel C. Sobral Jul 11 '18 at 05:00
  • I'm assuming that your problematic tests are constrained to a set of classes (and if not, the project should likely be re-architected to isolate those problematic tests). Then, I would personally setup Maven profiles - one enabled by default for CI use, one enabled for dev's to run tests locally (when they don't have the required software). Within the "dev" profile, exclude the test classes within the FailSafe plugin. Dev's without the software don't have to run the problematic tests, but the CI server does, and tests don't know about the context. I can add an "answer" for context if helpful. – Dave Jul 11 '18 at 19:00
  • @Dave No, I *cannot* exclude the test from running, because people who do work with that part of the system need to execute it. Every dev who has the software needs to run it. Every dev who does not have the software should not run it. Jenkins should always run it. – Daniel C. Sobral Jul 13 '18 at 02:32
  • So, Maven Profiles is the obvious answer. CI profile runs the test, "dev-with-software" profile runs the test, "dev-without-software" profile excludes the specific test classes from execution. Individual developers choose the profiles that they have active (defaulted in settings.xml or just at runtime). This way, the test doesn't need to "know too much" (ie, anything about the context). – Dave Jul 13 '18 at 20:31
  • @Dave Individual developers don't use profiles. It's human nature -- they'll go for the least amount of characters they can type. Tests should be as fool-proof as possible, and relying on developers to use profiles is not. – Daniel C. Sobral Jul 15 '18 at 05:48
  • Funny, this is exactly what my last 3 employers mandated. Tests were always written to ensure that if you didn't run the correct profile, tests didn't pass. Obviously with a sensible default. Changing your test cases so that they're aware of the test context is a bad architecture/design decision. Let the IDE remember which profiles to run, and then it's one-time setup for individual developers. – Dave Jul 16 '18 at 06:16

2 Answers2

32

Since they are easy to test for and passed to every program that gets executed by the build, I opted to base the check on environment variables.

Though rohit answer shows one way of setting variables on Jenkinsfile, I'd rather rely on things that Jenkins itself does, so I checked the environment variables set on my Jenkins jobs, and there's many options to pick from:

General Jenkins Information

These seem to be constant on a Jenkins instance.

  • HUDSON_URL
  • JENKINS_URL

Not Jenkins-specific, but the following is also useful:

  • USER

Our Jenkins runs under user jenkins, so testing that value is also a possibility.

Job Information

These have constant values across builds for the same job.

  • JOB_URL
  • JOB_NAME
  • JOB_BASE_NAME
  • JOB_DISPLAY_URL

Build Information

These have constant values across the same build (that is, for different sh invocations).

  • BUILD_URL
  • BUILD_TAG
  • RUN_CHANGES_DISPLAY
  • RUN_DISPLAY
  • BUILD_DISPLAY_NAME
  • BUILD_ID
  • BUILD_NUMBER

Node Information

These are related to the node on which the current command is being executed.

  • JENKINS_HOME
  • JENKINS_NODE_COOKIE
  • NODE_LABELS
  • NODE_NAME

Special

This one changes with each build, may change from node to node, and may even change with Jenkinsfile configuration:

  • WORKSPACE

Miscellaneous

I'm not sure about these. They are definitely from Jenkins, but I don't know what their lifecycle is.

  • HUDSON_SERVER_COOKIE
  • JENKINS_SERVER_COOKIE
  • HUDSON_COOKIE

Final Considerations

For my own purposes, I decided to go with JENKINS_HOME. The name is very Jenkins-specific, and it seems more fundamental than, say, JENKINS_URL. While it doesn't imply the build is being run by Jenkins, it will always be set in that case. I don't mind false positives, as long as I don't get false negatives.

Daniel C. Sobral
  • 295,120
  • 86
  • 501
  • 681
5

Couple of ways you can achieve this:

  1. You can make you application accept arguments and then pass a TRUE/FALSE value indicating it is running from Jenkins or not.

  2. You can also read the system properties of os

e.g. System.getProperty("os.arch");

But this will not work if your Jenkins environment and your workspace are on the same machine

  1. You can just set an env variable in your pipeline(exists only in pipeline) and in the application you can read that value

Like so :

Pipeline- option1

     pipeline {
            environment {
                FROM_JENKINS= "TRUE" 
            } stage('test'){ 

                   sh "mvn test"
             }
        }

Pipeline- option2

     pipeline {
            stage('test'){ 

                    sh '''FROM_JENKINS="TRUE" // setting the env variable in the same shell where you are running mvn
                        mvn test'''
             }
        }

Application

boolean isJenkinsBuild = Boolean.valueOf(System.getenv("FROM_JENKINS"));
boolean isSoftwarePresent = new ProcessBuilder("check software presence").start().waitFor() == 0;
Assume.assumeTrue("Software not present", isSoftwarePresent || isJenkinsBuild);

Hope it helps :)

rohit thomas
  • 2,302
  • 11
  • 23