3

Definitely missing something, and could use some quick assistance!

Simply, how do you deploy a Docker image to an Airflow DAG for running jobs? Does anyone have a simple example of deploying a Google container and running it via Airflow/Composer?

curious_kills
  • 31
  • 1
  • 2
  • However, see the trouble I am having here https://stackoverflow.com/questions/50253082/broken-dag-no-module-named-docker - it seems Docker python module is not installed – MarkeD Jun 04 '18 at 08:29
  • There's an alternative solution here https://stackoverflow.com/questions/51185485/running-docker-operator-from-google-cloud-composer - but again, it's not optimal – danodonovan Oct 08 '19 at 09:12

1 Answers1

2

You can use the Docker Operator, included in the core Airflow repository.

If pulling an image from a private registry, you'll need to set a connection config with the relevant credentials and pass it to the docker_conn_id param.

Ben Gregory
  • 934
  • 6
  • 5
  • Thanks so much! Any advice on getting beyond this issue? Unfortunately, I had to setup a custom plugin to the newest github push( See: https://github.com/apache/incubator-airflow/commit/49556917d9a3e9192d9043afd6cf3ca7939a2403) Basically, took the most up-to-date docker operator and created it as a custom plugin. This is google composer specific, in some way, but I am getting the following error: `bind address needs a port: {0}".format(addr)` Do you think i need to do the aforementioned, setup `docker_conn_id`? How do I set this up with composer?? Thanks in advance!!! – curious_kills Jun 05 '18 at 05:57
  • 1
    Even if i roll back packaging i still get an error with binding the port, i must be missing something again, because it is not clear to me how to generate this information for Composer, looking at the hook code: ` extra_options = conn.extra_dejson self.__base_url = base_url self.__version = version self.__tls = tls self.__registry = conn.host self.__username = conn.login self.__password = conn.password self.__email = extra_options.get('email') self.__reauth = False if extra_options.get('reauth') == 'no' else True` – curious_kills Jun 05 '18 at 08:02
  • Sorry I don't have much experience using Google Composer. If this doesn't get resolved and is a blocker for you, I might suggest spinning up your own Airflow or using a managed service. https://github.com/jghoman/awesome-apache-airflow#airflow-deployment-solutions – Ben Gregory Jun 05 '18 at 21:08
  • Yeah, i think you are right. Ugh, put so much effort into this and so close. – curious_kills Jun 05 '18 at 21:20