2

I have an Airflow environment (v2.4.3) on Kubernetes and I want to sync it with a private git repo so that any changes I make to DAGs in my master branch get automatically picked up by my Airflow environment.

According to Airflow documentation, I can use Git-sync sidecar along with an SSH key added to my private git repo and Airflow env to make it work.

However, given that I am constantly creating new private repos and Airflow environments, I am wondering if there is a more simple way of connecting my private git repos to their respective Airflow environment.

If I have a webapp managing my Airflow environments and have access to an OAuth token from Github after signing into my account (or any other git service), could I use that to connect my an Airflow environement and sync changes to any git repo of my choice under my account?

jorgeavelar98
  • 75
  • 1
  • 9

2 Answers2

1

I was able to figure it out.

One can use personal access tokens as passwords provided by whatever git service the private repo is in along with the repo's username.

I just stored the personal access token as an Opaque secret in my Airflow K8s cluster and referenced that in my git-sync sidecar container yaml definition which I included in my Airflow yaml deployment definition.

      containers:
        - name: git-sync
          image: registry.k8s.io/git-sync/git-sync:v3.6.5
          args:
            - "-wait=60"
            - "-repo=<repo>"
            - "-branch=master"
            - "-root=/opt/airflow/dags"
            - "-username=<username>"
            - "-password-file=/etc/git-secret/token"
          volumeMounts:
            - name: git-secret
              mountPath: /etc/git-secret
              readOnly: true
            - name: dags-data
              mountPath: /opt/airflow/dags
      volumes:
        - name: dags-data
          emptyDir: {}
        - name: git-secret
          secret:
            secretName: github-token
jorgeavelar98
  • 75
  • 1
  • 9
0

Based on the values file comments, you can use a personal access token to authenticate:

 if your repo needs a user name password
 you can load them to a k8s secret like the one below
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: git-credentials
   data:
     GIT_SYNC_USERNAME: <base64_encoded_git_username>
     GIT_SYNC_PASSWORD: <base64_encoded_git_password>
 and specify the name of the secret below

Then you can provide the secret name to dags.gitSync.credentialsSecret:

dags:
  gitSync:
    ...
    credentialsSecret: git-credentials
Hussein Awala
  • 4,285
  • 2
  • 9
  • 23