Not able to connect private github repository in apache ariflow helm chart in Kubernetes

Question

I am new working with Airflow and Kubernetes. I am trying to use apache Airflow in Kubernetes.

To deploy it I used this chart: https://github.com/apache/airflow/tree/master/chart.

I want to upload my dags in my github repository so:

gitSync:
    enabled: true
    # git repo clone url
    # ssh examples ssh://git@github.com/apache/airflow.git
    # git@github.com:apache/airflow.git
    # https example: https://github.com/apache/airflow.git
    repo: https://github.com/mygithubrepository.git
    branch: master
    rev: HEAD
    root: "/git"
    dest: "repo"
    depth: 1
    # the number of consecutive failures allowed before aborting
    maxFailures: 0
    # subpath within the repo where dags are located
    # should be "" if dags are at repo root
    subPath: ""

Then I see that to use a private github repository I have to create a secret as is specified in the value.yml file:

# if your repo needs a user name password
    # you can load them to a k8s secret like the one below
    #   ---
    #   apiVersion: v1
    #   kind: Secret
    #   metadata:
    #     name: git-credentials
    #   data:
    #     GIT_SYNC_USERNAME: <base64_encoded_git_username>
    #     GIT_SYNC_PASSWORD: <base64_encoded_git_password>
    # and specify the name of the secret below
    #credentialsSecret: git-credentials

I am creating the secret:

apiVersion: v1
data:
  GIT_SYNC_USERNAME: bXluYW1l
  GIT_SYNC_PASSWORD: bXl0b2tlbg==
kind: Secret
metadata:
  name: git-credentials
  namespace: default

Then I use the secret name in the value.yml file:

repo: https://github.com/mygithubrepository.git
    branch: master
    rev: HEAD
    root: "/git"
    dest: "repo"
    depth: 1
    # the number of consecutive failures allowed before aborting
    maxFailures: 0
    # subpath within the repo where dags are located
    # should be "" if dags are at repo root
    subPath: ""
    # if your repo needs a user name password
    # you can load them to a k8s secret like the one below
    #   ---
    #   apiVersion: v1
    #   kind: Secret
    #   metadata:
    #     name: git-credentials
    #   data:
    #     GIT_SYNC_USERNAME: <base64_encoded_git_username>
    #     GIT_SYNC_PASSWORD: <base64_encoded_git_password>
    # and specify the name of the secret below
    credentialsSecret: git-credentials

but seems not bee working.

What is the exact error message you get ? – mario Oct 09 '20 at 15:14 — mario, Oct 09 '20 at 15:14

score 3 · Accepted Answer · answered Oct 15 '20 at 21:43

I see that you are connecting to your github repo via https.

Try to use:

ssh://git@github.com/mygithubrepository.git

or simply

git@github.com/mygithubrepository.git

You can experience issues with connecting via https especially if you have two-factor authentication enabled on your github account. It's described more in detail in this article.

Also take a look at VonC's answer where he mentions:

As noted in Oliver's answer, an HTTPS URL would not use username/password if two-factor authentication (2FA) is activated.

In that case, the password should be a PAT (personal access token) as seen in "Using a token on the command line".

That applies only for HTTPS URLS, SSH is not affected by this limitation.

As well as at this one provided by rc0r.

But as I said, simply using ssh instead of https should resolve your problem easily.

Agreed: using a token is the bast practice (instead of using your account password): https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token#using-a-token-on-the-command-line — VonC, Oct 15 '20 at 22:41

score 0 · Answer 2 · answered Sep 07 '21 at 11:26

I ran into this exact issue. I was just about to use ssh instead, but then just persisted. How I got https to work (using a PAT) is:

Fill in your repo URL

repo: "https://github.com/USERNAME/REPOSITORY.git"

Add the name of your secret

httpSecret: "my-airflow-secret"

Fill in this part too

httpSecretUsernameKey: "your-username-key"
httpSecretPasswordKey: "your-PAT-key"

Now, the key is, when creating your base64 encoded strings to put in your secret, use:

echo -n "your-user-name" | base64

The -n means echo doesn't append a newline. After that It worked correctly. I have many other applications that didn't mind base64 encoded strings without -n so I guess I just got a bit complacent. It took me a while to realise the issue though.

Not able to connect private github repository in apache ariflow helm chart in Kubernetes

2 Answers2