5

At first, I was trying to use ansible for deployment after gitlab CI built, but it shows "host unreachable" in result.

After some trial and error, I found out the problem is ssh permission denied when ssh by private key into my AWS EC2 instance to deploy.

My .gitlab-ci.yml config is something like this:

.gitlab-ci.yml

image: ansible/ubuntu14.04-ansible:stable

stages:
  - deploy

deploy_web:
  stage: deploy
  script:
   - "echo Ansible"
   - "echo Environment: ${ENV}"
   - "echo TAG: ${TAG}"

   - "echo ${VAULT_PASS} > vault_pass.txt"
   - "mkdir sshkey"
   - "echo ${SSH_KEY_APP} > ./sshkey/app-key.pem"
   - "chmod 600 ./sshkey/app-key.pem"
   - "export SSH_KEY_DIR=`pwd`/sshkey"
   - "export ANSIBLE_HOST_KEY_CHECKING=False"
   - "ssh-keyscan foobar.io >> ~/.ssh/known_hosts"
   - "ssh -v -i ./sshkey/app-key.pem ubuntu@foobar.io" // for debugging

   - "ansible-playbook -i ${ENV} servers.yml --vault-password-file vault_pass.txt -vvvv --tags=${TAG}"

When gitlab CI builds this, it basically gives these ssh error messages:

OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to foobar.io [12.34.56.78] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file ./sshkey/app-key.pem type -1
debug1: identity file ./sshkey/app-key.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA be:b1:53:76:aa:bf:65:ea:b4:1b:7a:8f:cc:7c:2a:79
debug1: Host 'foobar.io' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
Warning: Permanently added the ECDSA host key for IP address '12.34.56.78' to the list of known hosts.
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: ./sshkey/app-key.pem
debug1: key_parse_private2: missing begin marker
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: No more authentication methods to try.

Permission denied (publickey).

Also tried using absolute path:

$ cat /builds/foobar/bar/sshkey/app-key.pem
-----BEGIN RSA PRIVATE KEY-----
 ...(the key)...
 -----END RSA PRIVATE KEY-----

$ ssh -v -i /builds/foobar/bar/sshkey/app-key.pem ubuntu@foobar.io
Permission denied (publickey).

These are what I have tried:

  • try using shell executor for gitlab CI runner -> failed
  • run the scripts in a local docker container -> success
  • ssh into the runner instance manually (not through CI) and run the scripts in shell -> success
  • ssh into the runner instance manually and run the scripts in docker container -> success

As a conclusion - It only fails when run by gitlab CI, so I wonder if there are any additional configuration I haven't noticed to do things like this...

Many thanks for anyone can help!

The real problem is

When echo-ing a multiline environment variable, quotes are needed. So basically every line of the key ends with ^M, which shows correctly in gitlab's console but actually unable to be parsed by ssh.

DarkBtf
  • 78
  • 1
  • 7

1 Answers1

4

If it fails when run by GilabCI, it means the user used by GitLab CI is not the same as the one used when you ssh in the running instance.

See for instance "AWS SSH connection error: Permission denied (publickey)"

Another thing to check is PermitRootLogin and AllowUsers in /etc/ssh/sshd_config.

This debug1: key_parse_private2: missing begin marker appears even after successful key authorization if your user access restricted.

Check after a manually ssh on the remote machine:

tail -f -n 80 /var/log/auth.log

The OP DarkBtf adds in the comments:

When echo-ing a multiline environment variable, quotes are needed.
So basically every line of the key ends with ^M, which shows correctly in gitlab's console but actually unable to be parsed by ssh.

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • When using gitlab CI docker executor, the user is root. I didn't put the key into .ssh, so I specified the identity as "-i app-key.pem" in ssh command. Are there any problems for doing so? – DarkBtf Nov 12 '16 at 12:43
  • @DarkBtf the only problem is: " where is app-key.pem"? Could you try with an absolute path (and add a command which makes sure that absolute path is visible, like `cat /full/path/to/app-key.pem`) – VonC Nov 12 '16 at 12:46
  • I've tried using absolute path (main post edited), still permission denied. – DarkBtf Nov 12 '16 at 13:10
  • @DarkBtf is there the public key in that same file? Or as as `/builds/foobar/bar/sshkey/app-key.pem` (or `app-key.pem.pub`) – VonC Nov 12 '16 at 13:13
  • @DarkBtf Also, ssh manually on the remote and do a `tail -f -n 80 /var/log/auth.log`, add http://stackoverflow.com/a/31817833/6309 – VonC Nov 12 '16 at 13:17
  • thanks, I have used ssh-keyscan to get public key in advance, store it in both ~/.ssh/known_hosts and the same folder as /builds/foobar/bar/sshkey/app-key.pem.pub. Still getting permission denied :( – DarkBtf Nov 12 '16 at 13:23
  • @DarkBtf going on the remote and see what `/var/log/auth.log` is still the best way to see if there are any of the issues mentioned in http://stackoverflow.com/q/23392763/6309. – VonC Nov 12 '16 at 13:31
  • 1
    I have found where the actual bug is... really dumb... when echo-ing a multiline environment variable, quotes are needed – DarkBtf Nov 12 '16 at 13:47
  • so basically every line of the key ends with ^M, which shows correctly in gitlab's console but actually unable to be parsed by ssh. – DarkBtf Nov 12 '16 at 13:49
  • Anyway, thanks for your help so I can eliminate all possible pitfalls to get it :D – DarkBtf Nov 12 '16 at 13:49
  • @DarkBtf Great! I have included your comment in the answer for more visibility. – VonC Nov 12 '16 at 14:14