Airflow k8s operator xcom - Handshake status 403 Forbidden

Question

When I run a docker image using KubernetesPodOperator in Airflow version 1.10

Once the pod finishes the task successfullly, airflow tries to get the xcom value by making a connection to the pod via k8s stream client.

Following is the error which I encountered:

[2018-12-18 05:29:02,209] {{models.py:1760}} ERROR - (0)
Reason: Handshake status 403 Forbidden
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 249, in websocket_call
    client = WSClient(configuration, get_websocket_url(url), headers)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 72, in __init__
    self.sock.connect(url, header=header)
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 223, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/usr/local/lib/python3.6/site-packages/websocket/_handshake.py", line 79, in handshake
    status, resp = _get_resp_headers(sock)
  File "/usr/local/lib/python3.6/site-packages/websocket/_handshake.py", line 152, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message)
websocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden

I'm using K8s service account for this

DAG configs

xcom=true,

get_logs=True,

in_cluster=true

Hi, we are experiencing the same issue. Did you find the solution? — Yoav Gaudin, Jan 30 '19 at 13:17

score 7 · Answer 1 · answered Mar 07 '19 at 02:26

7

So we also hit this problem, we had to modify our rbac rules, in particular we had to add the resource "pods/exec" with the verbs "create" and "get"

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: airflow-runner
rules:
- apiGroups: [""]
  resources: ["deployments", "pods", "pods/log", "pods/exec", "persistentvolumeclaims"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["singleuser-image-credentials"]
  verbs: ["read","list","watch","create","get"]

answered Mar 07 '19 at 02:26

Timothy Griffiths

71
1
4

1

Did you also create the `ServiceAccount` and `ClusterRoleBinding`? – alltej Feb 23 '20 at 04:27
Yes, It's required to have the ClusterRoleBinding and ServiceAccount to the binding. – Damon Liao Sep 02 '22 at 09:09

score 0 · Answer 2 · answered Oct 18 '21 at 13:07

In my case, I run a pod with python script inside kubernetes cluster with another pods running inside. The script try to perform actions over other pods such as list, get and exec command inside the pods. With exec, the following ClusterRole and ClusterRoleBinding works.

ClusterRole

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-exec
rules:
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["*"]

ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: pod-exec
subjects:
- kind: ServiceAccount
  name: default
  namespace: couchdb
roleRef:
  kind: ClusterRole
  name: pod-exec
  apiGroup: rbac.authorization.k8s.io

Remember to change the name of your ServiceAccount and namespace.

Airflow k8s operator xcom - Handshake status 403 Forbidden

2 Answers2

Linked