1

I'm running a KubernetesPodOperator with a custome docker image that was imported with microk8s. When running the DAG in airflow I see the following log until I get the time out message AirflowException('Pod Launching failed: {error}'.format(error=ex))

[2020-04-14 23:06:39,875] {logging_mixin.py:95} INFO - [[34m2020-04-14 23:06:39,875[0m] {[34mpod_launcher.py:[0m141} INFO[0m - Event: [1mk8s-pod-ml-3a541375[0m had an event of type [1mPending[0m[0m

Searching for the problem I ran microk8s kubectl get pods on ubuntus terminal and got: enter image description here

as you can see the name of the pod is almost exactly the same in the logs (except fro the 1m at the begining) but the status is ErrImageNeverPull. I couldn't find anything in the internte regarding that status or the error except for this that says 'The default timeout for KubernetesPodOperator is 120 seconds, which can result in timeouts occurring before larger images download.' I change the default timeout to 600 seconds but did not work.

Alejandro Andrade
  • 2,196
  • 21
  • 40
  • Does this answer your question? [Getting "ErrImageNeverPull" in pods](https://stackoverflow.com/questions/56392041/getting-errimageneverpull-in-pods) – mdaniel Apr 15 '20 at 02:54
  • @mdaniel I saw that before but not because im not running in a VM, i'm not using minikube and `docker-env` is not a subcomand of microk8s – Alejandro Andrade Apr 15 '20 at 12:48
  • Just try adding `imagePullPolicy: Never` to your pod spec and let us know if it works. – Matt Apr 15 '20 at 14:03
  • @HelloWorld honestly don't know where to add it the config file of a microk8s is different than the one shown in that thread – Alejandro Andrade Apr 15 '20 at 14:11

1 Answers1

0

The problem

Microk8s was not having access to docker. I found out about it by running microk8s.inspect and getting the 

WARNING:  Docker is installed. 
File "/etc/docker/daemon.json" does not exist. 
You should create it and add the following lines: 
{
    "insecure-registries" : ["localhost:32000"] 
}

and then restart docker with: sudo systemctl restart docker
Building the report tarball
  Report tarball is at /var/snap/microk8s/1320/inspection-report-20200415_141500.tar.gz

After doing as the warning said airflow manage to deploy the image in the pod.

Alejandro Andrade
  • 2,196
  • 21
  • 40