1

I have a databricks cluster and I want to store my cluster logs in an ADLS Gen 2 account. For that I followed this answer-

How to re-direct logs from Azure Databricks to another destination?

Also, this article talks about creating a cluster and sending the logs to ADLS but writes a CURL command to create a cluster.

But this didn't help. I changed the mount script in logging like this-

enter image description here

But this also didn't help.

PeakyBlinder
  • 1,059
  • 1
  • 14
  • 35

1 Answers1

2

Right now, Azure Databricks doesn't support writing the logs directly into ADLS (in contrast to AWS & GCP that allow to write directly). The solution would be to mount your storage container into Databrikcs (using dbutils.fs.mount command) and then point cluster configuration to that mounted location.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132