9

We are having issues setting the Jenkins environment variables on our dynamic EC2-Fleet.

We already have a fixed master (linux) and a fixed Windows slave but wanted to add slaves dynamically when the load on the system becomes heavy. For this we created a Spot Request Instance in AWS spinning up linux machines from an AMI and control this via the EC2-fleet-plugin in Jenkins.

Before this EC2-fleet can be of any help, our jobs must be able to run on its nodes. Most of our jobs use Jenkinsfiles and need certain environment variables to be set but the EC2-fleet-plugin does not provide the possibility to set environment variables (https://issues.jenkins-ci.org/browse/JENKINS-36544).

As suggested on this ticket (JENKINS-36544), we tried to set the environment variables in "System Configuration" for the dynamic ec2 slaves and set the environment variables for the other nodes on the "Node Configuration" overriding the "System Configuration", or so we thought. This should work if this bug wouldn't exist: https://issues.jenkins-ci.org/browse/JENKINS-44425. Because of this bug the "System Configuration" overrides the "Node configuration" instead of vice versa. So we can't use this as the existing nodes would not have the correct environment variables anymore.

As a last resort we tried to set the environment variables on the dynamic ec2 slaves by creating an /etc/profile.d/jenkinsvars.sh on the AMI used by the Spot Request Instance. This script would be automatically run on login system wide (https://help.ubuntu.com/community/EnvironmentVariables#A.2Fetc.2Fprofile.d.2F.2A.sh). Next to that we also attempted to set them in /home/ubuntu/.profile on the AMI singling out the ubuntu user which is the user running the Jenkins agent (https://help.ubuntu.com/community/EnvironmentVariables#A.2BAH4-.2F.profile). But it appears Jenkins does not use these environment variables but its own...

A way that works is to adapt the jobs to load a groovy file that's part of the AMI to set the environment variables we need but that would mean to change almost all jobs we have, next to all Jenkinsfiles that are included in our repositories (Bitbucket project). We would like to avoid this....

Marco Andries
  • 111
  • 7
  • Can you specify one of environment variables that you need to set that gets overridden on the linux spot instance? – Taterhead Sep 30 '19 at 10:48
  • Those are really specific so I don't think that info will help but we use "gsExec" and "compareExec" to respectively define the path to GhostScript and ImageMagick... – Marco Andries Oct 01 '19 at 11:12

4 Answers4

1

Try the following strategy:

  1. Leverage User Data to run a shell script when the Spot Instance launches. It is the primary recommended way by Amazon and the plugin authors.
  2. Instead of saving variables into the environment, have the user data script save them into either /var/tmp or /etc/profile or in parameter store. Refer to answers in this SO question. If you want to encrypt your info use KMS parameter store, if you dont care use one of the others.Choose one of the answers to best fit your needs.
  3. Alter your Jenkins job to pause until your user data script has completed running (refer to the documentation from the plugin)
  4. Change your Jenkins job to pick up the variables from the location you chose in step 2.
Taterhead
  • 5,763
  • 4
  • 31
  • 40
  • Hi thanks for the suggestion but like I mentioned in the original question, we would like to avoid having to change our Jenkins jobs.... – Marco Andries Oct 03 '19 at 07:10
0

Try restarting the server environment. Just saying.

bijaykumarpun
  • 677
  • 4
  • 9
-1

So we can't use this as the existing nodes would not have the correct environment variables anymore.

Update your existing nodes to load the environment variables when they are provisioned / started, then remove them from the System configuration, then add them to the Node configuration.

You could also try setting the Slave command prefix field to ENV_VAR1=val1 ENV_VAR2=val2, although I haven't tried that.

Thirdly, you can try putting your variables directly into /etc/profile which should always be loaded no matter what user you are logging in as.

However, the easiest by far is to make all of your drones/agents exactly the same and set your environment variables in whatever scripts you run to build your projects. Use docker to pull dependencies to the agents as necessary during the job and to set up specific environments for your applications. This greatly simplifies the maintenance and configuration of your agents.

Brandon
  • 1,956
  • 18
  • 18
  • Brandon, thank you for the suggestions but as far as I know these won't work... Adding node Jenkins environment variables for the existing slaves is exactly what we did (see original question). As we need the EC2 instances also to have Jenkins environment variables, we have to use the System Configuration to set those. Due to the existing bug, those override the node environment variables. 2 & 3. Like indicated, the Jenkins environment variables are not the same as the environment variables. The environment variables are already set, that's not an issue. 4. We have Linux & Windows nodes – Marco Andries Oct 11 '19 at 08:13
  • `remove them from the System configuration` By this I meant define all environment variables that are specific to any particular slave in the Node configuration, and only have environment variables that apply to all slaves in the global configuration `Like indicated, the Jenkins environment variables are not the same as the environment variables.` If the environment variables are not being set, you are not doing it correctly. Jenkins does have access to env vars set on the node itself. See this answer: https://stackoverflow.com/a/9029251/535827 – Brandon Oct 13 '19 at 21:23
  • There is also a bug in very old versions of Jenkins that prevents env var updates from being recognized w/o destroying and recreating the slave in Jenkins. Ensure you are not on an old version of Jenkins. I know for a fact it is possible, you just have to set them in the right place (correct user, correct shell, correct file for non-interactive login shell). You can also look at this plugin which may help: https://wiki.jenkins.io/display/JENKINS/EnvInject+Plugin – Brandon Oct 13 '19 at 21:28
  • Hi Brandon, I have tried your suggestions and found out that actually the machine from which we create the AMI, works as expected. Jenkins has the environment variables. For the spot instances created from the AMI, although they have the environment variables, Jenkins does not have the environment variables. – Marco Andries Oct 29 '19 at 10:10
-1

The Jenkins version or version of the EC2 plugin are missing in the question, but according to the description in this merged pull request, this bug should be fixed now: https://github.com/jenkinsci/ec2-plugin/pull/440#issuecomment-597160730

Jenkins version:

so this change works in both <=2.204, and >=2.205

EC2 plugin version: >=ec2-1.50

JENKINS-36544 - Fix Node Properties on Jenkins 2.205+ (#440) @jhansche

From the Pull Request description:

  • Navigate to the cloud configuration screen (this moves to a new page >=2.205)
  • Click "Add a new cloud"
  • Click "Amazon EC2"
  • Under the "AMIs" section, click "Add"
  • At the bottom of the AMI block, expand "Advanced"
  • Expect to see "Node Properties" block at the bottom of the block

Node Properties has the Environment variables.

Amedee Van Gasse
  • 7,280
  • 5
  • 55
  • 101