Terraform cannot pull modules as part of jenkins pipeline

Question

I have a jenkinsfile that was working and able to deploy some infrastructure automatically with terraform. Unfortunately after adding a terraform module with a git source it stopped working with the following error:

+ terraform init -input=false -upgrade

Upgrading modules...

- module.logstash

  Updating source "git::https://bitbucket.org/*****"

Error downloading modules: Error loading modules: error downloading 'https://bitbucket.org/*****': /usr/bin/git exited with 128: Cloning into '.terraform/modules/34024e811e7ce0e58ceae615c545a1f8'...

fatal: could not read Username for 'https://bitbucket.org': No such device or address



script returned exit code 1

The urls above were obfuscated after the fact. Below is the cut down module syntax:

module "logstash" {
  source             = "git::https://bitbucket.org/******"
  ...
}

Below is the Jenkinsfile:

pipeline {
  agent {
    label 'linux'
  }
  triggers {
    pollSCM('*/5 * * * *')
  }
  stages {
    stage ('init') {
      steps {
        sh 'terraform init -input=false -upgrade'
      }
    }
    stage('validate') {
      steps {
        sh 'terraform validate -var-file="production.tfvars"'
      }
    }
    stage('deploy') {
      when {
        branch 'master'
      }
      steps {
        sh 'terraform apply -auto-approve -input=false -var-file=production.tfvars'
      }
    }
  }
}

I believe this to be a problem with terraform internally using git to checkout the module but Jenkins has not configured the git client within the pipeline job itself. Preferably I would be able to somehow pass the credentials used by the multibranch pipeline job into the job itself and configure git but I am at a loss of how to do that. Any help would be appreciated.

@ydaetskcoR no it's not unfortunately. Both the parent project and module are private repos in bitbucket — George Richardson, Oct 11 '18 at 11:23
@MattSchuchard I am aware that this is a git credential issue, however it is the act of passing the checkout credentials in from the Jenkins build which I am having trouble with, not configuring git itself which I could do manually. — George Richardson, Oct 11 '18 at 13:16
i do not see any `withCredentials` in your pipeline code though. — Matthew Schuchard, Oct 11 '18 at 15:10

score 2 · Accepted Answer · answered Oct 12 '18 at 07:41

So I found a non-ideal solution that requires you to specify the credentials inside your Jenkinsfile rather than automatically using the credentials used by the job for checkout.

withCredentials([usernamePassword(credentialsId: 'bitbucketcreds', passwordVariable: 'GIT_PASS', usernameVariable: 'GIT_USER')]) {
  sh "git config --global credential.helper '!f() { sleep 1; echo \"username=${env.GIT_USER}\\npassword=${env.GIT_PASS}\"; }; f'"
  sh 'terraform init -input=false -upgrade'
  sh 'git config --global --remove-section credential'
}

The trick is to load the credentials into environment variables using the withCredentials block and then I used the answer from this question to set the credential helper for git to read in those creds. You can then run terraform init and it will pull down your modules. Finally it clears the modified git settings to hopefully avoid contaminating other builds. Note that the --global configuration here is probably not a good idea for most people but was required for me due to a quirk in our Jenkins agents.

If anyone has a smoother way of doing this I would be very interested in hearing it.

Terraform cannot pull modules as part of jenkins pipeline

1 Answers1