I'm trying to develop an automated data pipeline using Google Cloud Dataflow. The service, which is running through Cloud Run, has been designed to receive HTTP requests with the pipeline arguments and initialize the Apache Beam pipeline, with the same arguments, using the Google Dataflow Runner. The module builds the Beam pipeline and starts it without waiting until complete. There's above the example of PipelineOptions:
options = PipelineOptions(
project=project,
region='europe-west4',
temp_location=f'gs://{bucket}/temp',
staging_location=f'gs://{bucket}/staging',
runner='dataflow',
job_name=job_id,
sdk_container_image=f'us-central1-docker.pkg.dev/{project}/data-pipeline/base:latest',
sdk_location='container',
service_account_email=service_account
)
The problem is that Dataflow is raising the following exception in some times:
The zone 'projects/{project}/zones/europe-west4-a' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
Is there a way to check the resource availability in the GCP zones?
Initially I was trying to run the Apache Beam pipeline in us-central1 zone, but the problem was becoming more frequent. I modified the zone from us-central1 to europe-west4 and the problem became less frequent, but it didn't stop completely.
