Triggering an alert when multiple dataflow jobs run in parallel in GCP

Question

I am using google cloud dataflow to execute some resource intensive dataflow jobs. And at a given time, my system must execute no more than 2 jobs in parallel.

Since each job is quite resource intensive, I am looking for a way to trigger an alert when more than 2 dataflow jobs are running.

I tried implementing a custom_count which increments after the start of each job. But custom_couter only display after the job has executed. And it might be too late to trigger an alert by then.

Does the [Cloud Monitoring alerts](https://cloud.google.com/dataflow/docs/guides/using-monitoring-intf#create-alerts) help you? — Prajna Rai T, Sep 29 '22 at 15:15

score 0 · Answer 1 · answered Sep 28 '22 at 13:16

0

You could modify the quota dataflow.googleapis.com/job_count of the project to be limited to 1, and no two jobs could run parallel in that project. The quota is at the project level, it would not affect other projects.

answered Sep 28 '22 at 13:16

Israel Herraiz

611
3
8

I am looking to develop an alerting mechanism rather than restricting the jobs – Tharindu Kumara Sep 28 '22 at 14:09
In that case, you could use the [value of the quota to implement an alert in Cloud Monitoring](https://cloud.google.com/monitoring/alerts/using-quota-metrics). – Israel Herraiz Sep 28 '22 at 16:40

score 0 · Answer 2 · answered Sep 28 '22 at 17:12

Another option is to use an GCP monitoring system that is observing the running Dataflow jobs. You can e.g. use Elastic Cloud (available via Marketplace) to load all relevant Metrics and Logs. Elastic can visualize and alert on every state you are interested in. I found this terraform project very helpful in order to get started with that approach.

Triggering an alert when multiple dataflow jobs run in parallel in GCP

2 Answers2