How to make a docker image/container on Google Cloud Composer and deploy it?

Question

I have airflow dags running on Google Cloud Composer that trains machine learning models on some training data and stores the model with the best accuracy. I want to make a docker container/image that has the best model and deploy it directly to Google Cloud or download the image to my local machine.

I looked at StackOverflow answers, Google Cloud Composer documentation and tutorials but they generally deal with running airflow inside docker or running commands inside a docker container created from an existing docker image. I want to be able to create a docker image and then download/deploy it.

I already have Dockerfile and other setup for creating docker images on my local machine. I do not know how to create a docker image on cloud composer using airflow and then download the image.

I have a task that builds a docker image.

def build_docker(ti, **context):

    import docker
    import os
    import subprocess

    # client = docker.from_env() ..........................................(1)

    docker_folder = ti.xcom_pull(
        task_ids="setup",
        key="docker_folder",
    )
    model_id = ti.xcom_pull(
        task_ids="setup",
        key="model_id",
    )
    model_path = ti.xcom_pull(
        task_ids="setup",
        key="model_path",
    )
    model_type = ti.xcom_pull(task_ids="setup", key="model_type")

    docker_image_name = f"{model_type}:{model_id}"

    os.chdir(docker_folder)
    os.system(f"cp {model_path} {os.path.join(docker_folder,'best_model')}")
    
    print(os.getcwd())

    # client.images.build(path=".", tag=docker_image_name) ................(2)
    output = subprocess.run(
        f"docker build -t {docker_image_name} .",
        shell=True,
        capture_output=True,
        encoding="utf-8",
    )
    print(output)

If I run this task on local, I can see that a docker image is made and I can create containers and run them. I cannot do the same in google cloud composer. I get the error command "docker" not found.

To bypass this, I installed docker pypi package and then uncommented line (1) and (2) but then I get the error

sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

Your `Dockerfile` launchs a Python program for your ML logic ? — Mazlum Tosun, Oct 11 '22 at 13:23
Yes. Airflow dag trains a model and I need to make a docker image of the python program with the model. — shubh gupta, Oct 12 '22 at 05:27

score 2 · Answer 1 · answered Oct 12 '22 at 08:37

2

You can also use a PythonOperator in your Airflow task :

Your PythonOperator can invoke your Python ML program :

PythonOperator(
        task_id="train_ml_model",
        op_kwargs={
            'my_param1': 'my_param1_value',
            'my_param2': 'my_param2_value'
        },
        python_callable=train_ml_model
    )

def train_ml_model(my_param1, my_param2):
    # Your ML program

In PyPackages from Cloud Composer you can add all the needed Python packages for your ML program :

answered Oct 12 '22 at 08:37

Mazlum Tosun

5,761
1
9
23

I already have DAGs that trains my model and stores them. I want to write a DAG that can make a docker image of trained model + deployment code. I have edited my question to reflect what I have done so far for making docker. – shubh gupta Oct 12 '22 at 10:09

score 1 · Accepted Answer · answered Oct 30 '22 at 04:28

Building docker images within docker (dind) is not recommended and you don't have access to the configuration of the containers in Composer that would allow you to run dind.

I would recommend exporting your ML model to Cloud Storage and then running a GCP Cloud Build job to push the Docker image

How to make a docker image/container on Google Cloud Composer and deploy it?

2 Answers2