SSH connection refused from Gitlab CI

Question

I am using Gitlab CI to build a static generated website and send it to my server. I am using a SSH keys pair to establish the connection from rsync, but my server is refusing the connection.

I tried several things: started with ED25519 key, changed to RSA key, all public keys are in authorised_keys in the server, changed permissions in files and folders by gitlab-ci.yml, used ssh-keyscan to put the servers in know_hosts... nothing help.

The gitlab-ci.yml is the following:

image: ruby:2

pages:
  stage: deploy
  before_script:
  - apt-get update
  - apt-get install rsync -y
  - eval $(ssh-agent -s)
  - chmod 400 "$SSH_PRIVATE_KEY"
  - ssh-add "$SSH_PRIVATE_KEY"
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - ssh-keyscan $SERVER >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts
  script:
  - bundle install
  - bundle exec jekyll build -d $LOCALDIR
  after_script:
  - rsync -paz $LOCALDIR $USER@$SERVER:$REMOTEDIR
  artifacts:
    paths:
    - $LOCALDIR
  only:
  - master
  - dev

Part of log output is the following:

$ rsync -paz $LOCALDIR $USER@$SERVER:$REMOTEDIR
Permission denied, please try again.
Permission denied, please try again.
$SERVER: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]

Well, any help or tip in this topic?

EDIT 1 - SSH logs

Thank you @Andrew. I used the ssh -vvvvvv ... option as suggested and this is an extract of the output:

debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug3: no such identity: /root/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug3: no such identity: /root/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /root/.ssh/id_xmss
debug3: no such identity: /root/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method

So, maybe the command ssh-add "$SSH_PRIVATE_KEY" in gitlab-ci.yml is not working as expected.

EDIT 2 - the answer

Thank you to @VonC the tip to copy the SSH private key fixed the problem. Following the final gitlab-ci.yml working as expected:

image: ruby:2

pages:
  stage: deploy
  before_script:
  - apt-get update
  - apt-get install rsync -y
  - eval $(ssh-agent -s)
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - cp "$SSH_PRIVATE_KEY" ~/.ssh/id_ed25519
  - chmod 600 ~/.ssh/id_ed25519
  - ssh-keyscan $SERVER >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts
  script:
  - bundle install
  - bundle exec jekyll build -d $LOCALDIR
  after_script:
  - rsync -auz --delete --omit-dir-times $LOCALDIR $USER@$SERVER:$REMOTEDIR
  artifacts:
    paths:
    - $LOCALDIR
  only:
  - master
  - dev

Can you add `ssh -vvvvvv $USER@$SERVER exit 0`, assuming your using account is not restricted to rsync, and show relevant logs? Also, do you have acces to server in question to check sshd logs? `journalctl -u sshd` should help — Andrew, Mar 29 '23 at 11:12
@user I have added a workaround in [my answer](https://stackoverflow.com/a/75877529/6309) — VonC, Mar 29 '23 at 13:19

VonC · Accepted Answer · 2023-03-29T13:18:47.877

First, make sure to use a private key without passphrase, which means you do not need the eval $(ssh-agent -s).
Pipelines normally do not interact well with passphrase to be set during an ssh connection.

Second, double-check the corresponding public key is properly added to the remote server ~$USER/.ssh/authorized_keys.

Using ssh -Tvv $USER@$SERVER can help on the pipeline side to check if your key is actually used.
And you can start a sshd in debug mode on the remote server side to check you are actually receiving SSH queries.

maybe the command ssh-add "$SSH_PRIVATE_KEY" in gitlab-ci.yml is not working as expected.

As an alternative, copy your key with a default name:

cp "$SSH_PRIVATE_KEY" ~/.ssh/id_ed25519

That way, it will be selected, without needing the ssh-agent.

Thank you @VonC, the tip to copy the private key fixed the problem. Now rsync is working as expected. I will add my final gitlab-ci.yml in the question. All the best! — user2540800, Mar 29 '23 at 14:07

Bencho Naut · Answer 2 · 2023-03-29T17:53:44.183

only one step is necessary ..
( the examples below are with rsa also in filenames , you could use ed25519 as well )

since you will have you key in a variable SSH_PRIVATE_KEY it "should" work like this ( but read below)

after_script:
- apt-get update && apt-get -y --no-install-recommends bash rsync openssh-client && ( echo "$SSH_PRIVATE_KEY" > /tmp/id_rsa && chmod 0600 /tmp/id_rsa )  &&  ssh-agent bash -c "test -e ~/.ssh || mkdir ~/.ssh; ssh-keyscan $SERVER >> ~/.ssh/known_hosts;ssh-add ~/tmp/id_rsa; rsync -paz $PWD $USER@$SERVER:$REMOTEDIR "

this might fail(multiline string), so i recommend to base64-encode your key ( base64 -w0 ~/.ssh/id_rsa ;echo )

  after_script:
  - apt-get update && apt-get -y --no-install-recommends bash rsync openssh-client && ( echo "$SSH_PRIVATE_KEY" |base64 -d > /tmp/id_rsa && chmod 0600 /tmp/id_rsa )  &&  ssh-agent bash -c "test -e ~/.ssh || mkdir ~/.ssh; ssh-keyscan $SERVER >> ~/.ssh/known_hosts;ssh-add ~/tmp/id_rsa; rsync -paz $PWD $USER@$SERVER:$REMOTEDIR "

(

in fact you could also leave out the ssh-agent when you put the key under ~/.ssh/id_rsa BUT it needs to have the 0600 permissions
you can also try head -c 120 /tmp/id_rsa to see if the content starts with the proper values )

SSH connection refused from Gitlab CI

2 Answers2