4

I am looking for a way to find DBU cost for DLT clusters, does it get stored anywhere I have been looking into event_logs but did not find information related to cost. it does have cluster resource utilization details.

here is what I found, could somebody assist here ?enter image description here

I'm looking to monitor cost for the compute used by DLT pipelines

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
Chhaya Vishwakarma
  • 1,407
  • 9
  • 44
  • 72

2 Answers2

1

The DLT Event table doesn't contain the cost information. The cost information is available via Azure Cost Management console where you can filter information by the tags, but you will need to add cluster tags explicitly in the pipeline settings. Otherwise you will need to go through each existing run and extract the cluster ID tag that was used for a specific run of DLT.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
0

If the workspace is Unity Enabled, the system tables may help you. If it is non-unity enabled, Databricks Overwatch can help here. It is an additional implementation and data crunching. But it will help us to gather more granular details about the pipes, cost, ingress, and egress data.

Jaganadh Gopinadhan
  • 460
  • 1
  • 7
  • 19