gcloud.dataflow.flex-template.run unrecognized arguments: --temp-location

Question

I'm providing both a staging and temp location via the gcloud dataflow flex-template run CLI. These are both valid optional flags (as per docs) and there isn't any mention that you should or shouldn't provide both.

Why else would this error come up?

ERROR: (gcloud.dataflow.flex-template.run) unrecognized arguments: --temp-location (did you mean '--staging-location'?) gs://gcs-bucket-name

Edit: adding context as requested

The process is executing within a Buildkite CI/CD pipeline, so generally speaking a Buildkite agent/step calls a gcloud Docker container which runs a bash script. I'm also able to run this command and container combo locally just fine -- the error is only present while running in the pipeline

Dockerfile

FROM gcr.io/google.com/cloudsdktool/cloud-sdk

COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh

WORKDIR /app

# RUN sudo apt-get install google-cloud-sdk <-- threw error

ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"

gcloud-deploy-flex-template.sh

gcloud dataflow flex-template run ${JOB_NAME} \
    --template-file-gcs-location ${TEMPLATE_PATH}.json \
    --region us-central1 \
    --staging-location ${GCS_PATH}/staging/${JOB_NAME} \
    --temp-location ${GCS_PATH}/temp \
        --parameters requirements_file=requirements.txt \
        --parameters input_subscription=${INPUT_SUBSCRIPTION} \
        --parameters output_table=${OUTPUT_TABLE} \
        --parameters subject=${SUBJECT} \
        --parameters schema_registry_url=${SCHEMA_REGISTRY_URL} \
    --subnetwork=${SUBNETWORK} \
    --service-account-email=${SERVICE_ACCOUNT_EMAIL}

Can you share the complete command with fake values please ? — Mazlum Tosun, Jan 11 '23 at 23:40
Is your gcloud CLI updated? I tried to specify both `--temp-location` and `--staging-location` and it works fine. — Bruno Volpato, Jan 12 '23 at 01:00
Thanks for the responses @MazlumTosun @BrunoVolpato I've added context and will double check the gcloud version (this is a good suggestion). It shouldn't be an issue because the `flex-template` arg is part of the new beta version, and would error out differently if it were an old version — willwrighteng, Jan 12 '23 at 02:27
Why you don't directly use the official `google cloud sdk` image : `google/cloud-sdk:412.0.0` ? Can you give use the exact `gcloud sdk` your image returns please ? — Mazlum Tosun, Jan 12 '23 at 11:37
adding the image tag did the trick! feel free to convert your comments to a response @MazlumTosun and I'll accept as the answer: `FROM gcr.io/google.com/cloudsdktool/cloud-sdk:408.0.1` — willwrighteng, Jan 12 '23 at 21:38

Mazlum Tosun · Accepted Answer · 2023-01-12T22:48:02.090

As I proposed in my comment, you can use the offical google sdk image with the latest version :

FROM google/cloud-sdk:412.0.0

COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh

WORKDIR /app

# RUN sudo apt-get install google-cloud-sdk <-- threw error

ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"

or

FROM gcr.io/google.com/cloudsdktool/cloud-sdk:408.0.1

COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh

WORKDIR /app

# RUN sudo apt-get install google-cloud-sdk <-- threw error

ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"

gcloud.dataflow.flex-template.run unrecognized arguments: --temp-location

1 Answers1