7

Looking for the correct/working way to source a Terraform module that is hosted within a private Azure DevOps git repo via SSH (see TF Docs on Git repo sourcing via ssh).

It's undesirable to have a PAT token as it will eventually expire and renewal can't be automated yet (although coming soon apparently). Despite that, have validated the https method but I would like to avoid needing to edit the source if I can as module source references can't include terraform vars at init time.

I have followed the following steps to generate an ssh private/public key pair and those files have been added to my local .ssh folder @ C:\Windows\Users\<username>\.ssh.

Example terraform code below:

module "test" {
    source = "<ref>"
}

Where <ref> has been tested with:

  • git::https://<pat token>:dev.azure.com/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref> - worked/validated https approach as last resort
  • git::ssh://git@ssh.dev.azure.com:v3/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref>
  • git::ssh://git@ssh.dev.azure.com/v3/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref> - see terraform/issues/18869

After reading about multiple keys here:

Generally, if you configure multiple keys for an SSH client and connect to an SSH server, the client can try the keys one at a time until the server accepts one. However, this doesn't work with Azure DevOps for technical reasons related to the SSH protocol and how our Git SSH URLs are structured. Azure DevOps will blindly accept the first key that the client provides during authentication. If that key is invalid for the requested repo, the request will fail with the following error: remote: Public key authentication failed. fatal: Could not read from remote repository.

I played around with my .ssh/config file including

  • Removing all other keys other than the automation ssh key pair that I wanted to use specifically for this case (as opposed to my own personal SSH key identifying me)
  • Updating .ssh/config to look like:
Host automation
    HostName ssh.dev.azure.com
    IdentityFile ~/.ssh/automation_account
    IdentitiesOnly yes
Host azuredevops
    HostName ssh.dev.azure.com
    IdentityFile ~/.ssh/azuredevops
    IdentitiesOnly yes

And then within the <ref> trying:

  • git::ssh://git@automation/v3/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref>
  • git::ssh://automation/v3/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref>

Where all of these attempts result in:

Permission denied, please try again.
git@ssh.dev.azure.com: Permission denied (password,publickey).
fatal: Could not read from remote repository.

Or

does not appear to be a git repository
fatal: Could not read from remote repository.

The ssh key that I am using, have validated that it is registered with Azure DevOps by getting the thumbprint ssh-keygen -E md5 -lf <path to private key> and getting back 2048 MD5:ab:e2:... automation (RSA) which I can confirm exists within Azure DevOps under my SSH keys.

I have validated that the key allows me to access one of my target TF repos by adding a new git remote using that specific key. git remote -v gives me:

aorigin       git@automation:v3/<org name>/<project name>/_git/<repo name> (fetch)
aorigin       git@automation:v3/<org name>/<project name>/_git/<repo name> (push)
origin  git@ssh.dev.azure.com:v3/<org name>/<project name>/_git/<repo name> (fetch)
origin  git@ssh.dev.azure.com:v3/<org name>/<project name>/_git/<repo name> (push)

And a git pull aorigin works as expected. Origin is the original SSH clone url from the Azure DevOps gui.

Almost certain that I am missing something obvious but after extensive Googling and a number of different configurations, cannot for the life of me get it to work. Help/pointers/suggestions appreciated

Versions of tools:

  • Terraform v0.13.4
  • git version 2.17.1.windows.2
  • OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
Jamie
  • 511
  • 7
  • 14

3 Answers3

6

Yes this is possible, I have done it myself on multiple projects successfully:

For VSTS URLs:

source = "git::ssh://OrgName@vs-ssh.visualstudio.com/v3/OrgName/ProjectName/RepoName//DirectoryName"

or for Azure DevOps URLs:

source = "git::ssh://git@ssh.dev.azure.com/v3/OrgName/ProjectName/RepoName//DirectoryName"

Please note, even if you have the source set correctly, if your known hosts file is not up to date, you may get an error saying to check your permissions which is quite misleading.

Run ssh-keyscan on ssh.dev.azure.com or vs-ssh.visualstudio.com to get the known hosts entry.

You can also pull from Private Github Repositories using SSH as-well.

ref: https://jimferrari.com/2022/01/12/use-terraform-modules-from-private-github-in-azure-devops/

jfdevops
  • 171
  • 2
  • 2
1

Following this issue and the Terraform documentation, I would try the URL

git::automation:v3/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref>

(Adding User git to my .ssh/config file, right after Hostname, in order to not have to add the user)

The OP Jamie performed the following steps:

  • Updated Git to mentioned version
  • Regenerated some new keys using steps in MS Docs
  • Added all keys using ssh-add
  • Added public keys to Azure DevOps
  • Made the following changes to .ssh/config file:
Host ssh.dev.azure.com 
  IdentityFile ~/.ssh/me
  IdentitiesOnly yes

Host automation
  HostName ssh.dev.azure.com
  User git
  IdentityFile ~/.ssh/auto 
  IdentitiesOnly yes 

Host automationsec 
  HostName ssh.dev.azure.com 
  User git 
  IdentityFile ~/.ssh/auto_sec 
  IdentitiesOnly yes

Tested with the two following cases:

module "test" {
  source     = "git::automation:v3/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref>"
}

and

module "test" {
  source     = "git::automationsec:v3/<org name>/<project name>/_git/<repo name>//<sub path>?ref=<version ref>"
}

The first passwordless entry works, the second case doesn't. Unsure of the root cause as to why a ssh key with a passphrase isn't accepted

Jamie
  • 511
  • 7
  • 14
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Many thanks for the response! I went back to the start and generated some new keys and found that when the ssh key doesn't include a passphrase, works fine. When there is a passphrase it returns ```C:\Program Files\Git\cmd\git.exe exited with 128: Cloning into '.terraform\modules\test'... Permission denied, please try again. Permission denied, please try again. git@ssh.dev.azure.com: Permission denied (password,publickey). ``` Happy to mark your response as an answer altho would be good to know why it fails when using a passphrase – Jamie Apr 14 '21 at 17:59
  • @Jamie No problem. Let me know how you have solved the issue. – VonC Apr 14 '21 at 17:59
  • @Jamie Great, thank you. I have included your comment in the answer for more visibility. – VonC Apr 14 '21 at 18:06
  • 1
    @Jamie You can edit the answer if you want to add more details. – VonC Apr 14 '21 at 18:08
  • Accepting answer as I can now resolve modules hosted in an Azure DevOps git repo. Still need to do some digging on the root cause of why the ssh key with a passphrase doesn't work. Will update when/if I find that out. – Jamie Apr 14 '21 at 18:26
  • @Jamie OK. I will look forward to your follow-up. – VonC Apr 14 '21 at 18:28
  • @Jamie Thank you for the edit! Much clearer now. – VonC Apr 16 '21 at 07:55
0

With respect to using an ssh key with a passphrase, we'd normally use ssh-agent for this with ssh-add to load the key. After struggling with the same issues trying @Vonc 's solution above, I dug further into terraform docs and discovered the following:

agent - Set to false to disable using ssh-agent to authenticate. On Windows the only supported SSH authentication agent is Pageant.

( https://www.terraform.io/docs/language/resources/provisioners/connection.html )

x0n
  • 51,312
  • 7
  • 89
  • 111