5

This did work previously!

My deployment step in my pipeline SSH's onto a DO box & pulls the code from a docker registry. As mentioned, this worked previously & this was my deploy step in my .gitlab-ci.yml back then which worked fine inspiration from here under Using SSH:

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - eval "$(ssh-agent -S)"
    - ssh-add ~/.ssh/id_rsa
    - ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
      "docker stop ${CI_PROJECT_NAME};
      docker rm ${CI_PROJECT_NAME};
      docker container prune -f;
      docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
      docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
      docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
      docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"

Once I just attempted to run the deploy step & failed. Coming back with this error:

...
 $ mkdir -p ~/.ssh
 $ echo "${DEPLOYMENT_SERVER_PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa
 $ chmod 600 ~/.ssh/id_rsa
 $ eval "$(ssh-agent -s)"
 Agent pid 22
 $ ssh-add ~/.ssh/id_rsa
 Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
 $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh gitlab@${DEPLOYMENT_SERVER_IP} "docker stop ${CI_PROJECT_NAME}; docker rm ${CI_PROJECT_NAME}; docker container prune -f; docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH}; docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}; docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest; docker run -d -p ${PORT}:${PORT} --restart always -m 2g --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
Running after_script
00:02
Uploading artifacts for failed job
00:01
 ERROR: Job failed: exit code 255

Steps I took to set this up originally

  • Run ssh-keygen -t rsa -b 2048 on DO box (with no password)
  • Added public key into authorized_keys on the DO box
  • Copy the private key into a CI variable DEPLOYMENT_SERVER_PRIVATE_KEY

I know the port is open for SSH as I am able to SSH from my local machine into gitlab user. I have now changed my deployment step (based on comments from here, this article, & this one) to:

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - echo ${CI_REGISTRY_USER}
    - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
      "docker stop ${CI_PROJECT_NAME};
      docker rm ${CI_PROJECT_NAME};
      docker container prune -f;
      docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
      docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
      docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
      docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"

Still to no avail! The verbosing logging of ssh spit out:

...
 $ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
 /usr/bin/ssh-agent
 $ eval $(ssh-agent -s)
 Agent pid 18
 $ echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
 Identity added: (stdin) ((stdin))
 $ mkdir -p ~/.ssh
 $ chmod 700 ~/.ssh
 $ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
 $ cat ~/.ssh/config
 Host *
    StrictHostKeyChecking no
 $ echo ${CI_REGISTRY_USER}
 gitlab-ci-token
 $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
 OpenSSH_8.3p1, OpenSSL 1.1.1g  21 Apr 2020
 debug1: Reading configuration data /root/.ssh/config
 debug1: /root/.ssh/config line 1: Applying options for *
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug2: resolve_canonicalize: hostname 134.xxx.xxx.xxx is address
 Pseudo-terminal will not be allocated because stdin is not a terminal.
 debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
 debug2: ssh_connect_direct
 debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
 debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection refused
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
 ERROR: Job failed: exit code 255

I also added the -T option suggested here to disable pseudo-tty allocation but all that did was remove the pseudo line from the logs.

EDIT

Looking at the logs on the DO box (/var/log/auth.log), I've got the error:

Jun 22 15:53:37 exchange-apis sshd[16159]: Connection closed by 35.190.162.232 port 49750 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16160]: Connection closed by 35.190.162.232 port 49754 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16162]: Connection closed by 35.190.162.232 port 49752 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16163]: Unable to negotiate with 35.190.162.232 port 49756: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth]
Jun 22 15:53:38 exchange-apis sshd[16161]: Unable to negotiate with 35.190.162.232 port 49758: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth]

Googling this error, common cause seems to be due to OpenSSH dropping support for DSA keys. However, not sure why this would effect me as I generated an RSA key pair. Anyway, running dpkg --list | grep openssh spits out:

ii  openssh-client                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) client, for secure access to remote machines
ii  openssh-server                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) server, for secure access from remote machines
ii  openssh-sftp-server                    1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines

& sshd -v spits out:

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017

Nevertheless, worked of the answers; here & here so my deploy stage is now:

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\tHostkeyAlgorithms +ssh-dss\n\tPubkeyAcceptedKeyTypes +ssh-dss\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  script:
    - ssh -oHostKeyAlgorithms=+ssh-dss gitlab@${DEPLOYMENT_SERVER_IP} ls

Still got no look with that & I get the same error in the output of the runner & the log on the DO box. Any ideas?

wmash
  • 4,032
  • 3
  • 31
  • 69

1 Answers1

2

Ideally, if you can log on to the DO box, you would stop the ssh service, and launch /usr/bin/sshd -de, in order to establish a debug session on the SSH daemon side, with logs written on stderr (instead of system messages)

But if you cannot, at least try and generate an rsa key without passphrase, for testing. That means you don't need the ssh-agent.
And try a ssh -Tv gitlab@${DEPLOYMENT_SERVER_IP} ls to see what log is produced there.

Try with a classic PEM format

ssh-keygen -t rsa -P "" -m PEM

after editing the pipeline a bit more, I've noticed that it is actually this line that is causing the issue: ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts

It can be the case if it leads to a badly formatted ~/.ssh/known_hosts, especially if the ${DEPLOYMENT_SERVER_IP} is not correctly set.
Try and add a echo "DEPLOYMENT_SERVER_IP='${DEPLOYMENT_SERVER_IP}'", and a cat ~/.ssh/known_hosts commands to the before_script section, to know more.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • in debug mode was the same output in the CI & the same output in the server. My SSH key pair already has no passphrase. Please look at my edit for more information I have found out when looking at logs – wmash Jun 22 '20 at 17:44
  • 1
    @wmash Try for testing a `ssh-keygen -t rsa -P "" -m PEM`, and see if that old format works better. – VonC Jun 22 '20 at 19:41
  • I've just tried that & it succeeds! Can you explain why that would be the case – wmash Jun 22 '20 at 20:43
  • @wmash Either because of the -b 2048 (not always accepted), or the OpenSSH format (-m PEM switches to the old format: https://stackoverflow.com/a/53645530/6309). I have edited the answer accordingly. – VonC Jun 22 '20 at 21:42
  • VonC after editing the pipeline a bit more, I've noticed that it is actually this line that is causing the issue: `ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts`. When I remove it, it succeeds just fine. Do you know why this would be? If you do, can you put it in your answer & I'll accept – wmash Jun 22 '20 at 22:51