11

I have the weirdest error in GitHub Actions that I have been trying to resolve for multiple hours now and I am all out of ideas.

I currently use a very simple GitHub Action. The end goal is to run specific bash commands via ssh in other workflows.

Dockerfile:

FROM ubuntu:latest

COPY entrypoint.sh /entrypoint.sh

RUN apt update && apt install openssh-client -y
RUN chmod +x entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh:

#!/bin/sh

mkdir -p ~/.ssh/
echo "$1" > ~/.ssh/private.key
chmod 600 ~/.ssh/private.key
echo "$2" > ~/.ssh/known_hosts

echo "ssh-keygen"
ssh-keygen -y -e -f ~/.ssh/private.key
echo "ssh-keyscan"
ssh-keyscan <IP>

ssh -i ~/.ssh/private.key -tt <USER>@<IP> "echo test > testfile1"
echo "known hosts"
cat ~/.ssh/known_hosts
wc -m ~/.ssh/known_hosts

action.yml

name: "SSH Runner"
description: "Runs bash commands in remote server via SSH"
inputs:
  ssh_key:
    description: 'SSH Key'
  known_hosts:
    description: 'Known Hosts'
runs:
  using: 'docker'
  image: 'Dockerfile'
  args:
    - ${{ inputs.ssh_key }}
    - ${{ inputs.known_hosts }}

current workflow file in the same repo:

on: [push]

jobs:
  try-ssh-commands:
    runs-on: ubuntu-latest
    name: SSH MY_TEST
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: test_ssh
        uses: ./
        with:
          ssh_key: ${{secrets.SSH_PRIVATE_KEY}}
          known_hosts: ${{secrets.SSH_KNOWN_HOSTS}}

In the github action online console I get the following output:

ssh-keygen
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "2048-bit RSA, converted by root@844d5e361d21 from OpenSSH"
AAAAB3NzaC1yc2EAAAADAQABAAABAQDaj/9Guq4M9V/jEdMWFrnUOzArj2AhneV3I97R6y
<...>
9f/7rCMTJwae65z5fTvfecjIaUEzpE3aen7fR5Umk4MS925/1amm0GKKSa2OOEQnWg2Enp
Od9V75pph54v0+cYfJcbab
---- END SSH2 PUBLIC KEY ----
ssh-keyscan
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
# <IP>:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
<IP> ssh-ed25519 AAAAC3NzaC1lZD<...>9r5SNohBUitk
<IP> ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRNWiDWO65SKQnYZafcnkVhWKyxxi5r+/uUS2zgYdXvuZ9UIREw5sumR95kbNY1V90<...>
qWXryZYaMqMiWlTi6ffIC5ZoPcgGHjwJRXVmz+jdOmdx8eg2llYatRQbH7vGDYr4zSztXGM77G4o4pJsaMA/
***
Host key verification failed.
known hosts
***
175 /github/home/.ssh/known_hosts

As far as I understand *** is used to replace GitHub secrets which in my case is the key of the known host. Getting *** as a result for the ssh-keyscan and the cat known_host should mean, that the known_hosts file is correct and a connection should be possible. Because in both cases the console output is successfully censored by GitHub. And since the file contains 175 characters I can assume it contains the actual key. But as one can see the script fails with Host key verification failed.

When I do the same steps manually in another workflow with the exact same input data I succeed. Same goes for ssh from my local computer with the same private_key and known_host files.

This for example works with the exact same secrets

- name: Create SSH key
        run: |
          mkdir -p ~/.ssh/
          echo "$SSH_PRIVATE_KEY" > ../private.key
          sudo chmod 600 ../private.key
          echo "$SSH_KNOWN_HOSTS_PROD" > ~/.ssh/known_hosts
        shell: bash
        env:
          SSH_PRIVATE_KEY: ${{secrets.SSH_PRIVATE_KEY}}
          SSH_KNOWN_HOSTS: ${{secrets.SSH_KNOWN_HOSTS}}
- name: SSH into DO and run
        run: >
          ssh -i ../private.key -tt ${SSH_USERNAME}@${SERVER_IP}
          "
            < commands >
          "

Using the -o "StrictHostKeyChecking no" flag on the ssh command in the entrypoint.sh also works. But I would like to avoid this for security reasons.

I have been trying to solve this issue for hours, but I seem to miss a critical detail. Has someone encountered a similar issue or knows what I am doing wrong?

jns_ai_unr
  • 753
  • 1
  • 8
  • 19
  • How did you determine the value in ${{secrets.SSH_KNOWN_HOSTS}}? – Matthew Setter Oct 04 '21 at 20:07
  • What do you mean by that? Generally Github censors your secrets. So if you do not know what your secret is (for example in case you lost it), but you want to retrieve the value, you could try to write it to a file and create a github artifact from it. Then you should be able to download it via the the web interface. If this is not what you meant, please elaborate your question. – jns_ai_unr Oct 04 '21 at 20:45

3 Answers3

10

So after hours of searching I found out what the issue was. When force accepting all host keys with the -o "StrictHostKeyChecking no" option no ~/.ssh/known_hosts file is created. Meaning that the openssh-client I installed in the container does not seem to read from that file. So telling the ssh command where to look for the file solved the issue:

ssh -i ~/.ssh/private.key -o UserKnownHostsFile=/github/home/.ssh/known_hosts -tt <USER>@<IP> "echo test > testfile1"

Apparently one can also change the location of the known_hosts file within the ssh_config permanently (see here).

Hope this helps someone at some point.

jns_ai_unr
  • 753
  • 1
  • 8
  • 19
2

For the GitHub runner in 2023, the .ssh dir doesn't exist, so you have to create it and known_hosts first:

mkdir -p ~/.ssh/ && touch ~/.ssh/known_hosts

Then add the public SSH host key:

ssh-keyscan [hostname] >> ~/.ssh/known_hosts

And the private key:

eval $(ssh-agent)
ssh-add - <<< "${{ secrets.SSH_PRIVATE_KEY }}"
Jannik
  • 357
  • 2
  • 7
  • 15
1

First, add a chmod 600 ~/.ssh/known_hosts as well in your entrypoint.

For testing, I would check if options around ssh-keyscan make any difference:

ssh-keyscan -H <IP>
# or
ssh-keyscan -t rsa -H <IP>

Check that your key is generated using the default rsa public-key cryptosystems.
The HostKeyAlgorithms used might be set differently, in which case:

ssh-keyscan -H -t ecdsa-sha2-nistp256 <IP>
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • I have added the `chmod 600` but unfortunately that did not change anything. The v-server has 3 keys when I scan it using ssh-keyscan. `ssh-rsa`, `ssh-ed25519` and the one I am using (`ecdsa-sha2-nistp256`) which is censored in the GitHub output. I have tried adding each one of these to the `known_hosts` file and I have also tried adding all three at once. All v-servers (digital ocean) use `ecdsa-sha2-nistp256` and I always use these within the `known_hosts` file. Like I said above, I think the issue stems from using a bash script. Because using the same key in a GitHub workflow.yml works. – jns_ai_unr Jul 22 '20 at 08:03
  • @j.unruh OK. Using `-H` did not change anything eight? – VonC Jul 22 '20 at 08:10
  • No sadly not, but I did find what the problem was. See below. – jns_ai_unr Jul 22 '20 at 08:38