Databricks Utilities (dbutils) works with object storage to chain and parameterize notebooks and work with secrets.
Documentation: Databricks Utilities | Databricks on AWS
Questions tagged [dbutils]
44 questions
27
votes
6 answers
How to delete all files from folder with Databricks dbutils
Can someone let me know how to use the databricks dbutils to delete all files from a folder.
I have tried the following but unfortunately, Databricks doesn't support…
Carltonp
- 1,166
- 5
- 19
- 39
13
votes
2 answers
How to properly access dbutils in Scala when using Databricks Connect
I'm using Databricks Connect to run code in my Azure Databricks cluster locally from IntelliJ IDEA (Scala).
Everything works fine. I can connect, debug, inspect locally in the IDE.
I created a Databricks Job to run my custom app JAR, but it fails…
empz
- 11,509
- 16
- 65
- 106
6
votes
2 answers
How do we access databricks job parameters inside the attached notebook?
In Databricks if I have a job request json as:
{
"job_id": 1,
"notebook_params": {
"name": "john doe",
"age": "35"
}
}
How do I access the notebook_params inside the job attached notebook?
Sannix19
- 75
- 1
- 6
5
votes
1 answer
Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
Trying to read delta log file in databricks community edition cluster. (databricks-7.2 version)
df=spark.range(100).toDF("id")
df.show()
df.repartition(1).write.mode("append").format("delta").save("/user/delta_test")
with…
ashley
- 53
- 1
- 6
5
votes
1 answer
ModuleNotFoundError: No module named 'pyspark.dbutils'
I am running pyspark from an Azure Machine Learning notebook. I am trying to move a file using the dbutil module.
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
def get_dbutils(spark):
try:
…
Jeanne Lane
- 495
- 1
- 9
- 26
5
votes
2 answers
Check if the path exists in Databricks
I try to check if the path exists in Databricks using Python:
try:
dirs = dbutils.fs.ls ("/my/path")
pass
except IOError:
print("The path does not exist")
If the path does not exist, I expect that the except statement executes.
However,…
Fluxy
- 2,838
- 6
- 34
- 63
4
votes
2 answers
Can I iterate through the widgets in a databricks notebook?
Can I iterate through the widgets in a databricks notebook? Something like this pseudocode?
# NB - not valid
inputs = {widget.name: widget.value for widget in dbutils.widgets}
Michael Grazebrook
- 41
- 5
4
votes
1 answer
How do I copy a local file to Azure Databricks DBFS filestore
I am using the below command in Azure Databricks to try and copy the file test.csv from the local C: drive to the Databricks dbfs location as shown.
dbutils.fs.cp("C:/BoltQA/test.csv", "dbfs:/tmp/test_files/test.csv")
I am getting this…
ibexy
- 609
- 3
- 16
- 34
3
votes
0 answers
Disable dbutils.fs.put() write to console "Wrote x bytes"
does anyone know how to suppress the output of dbutils.fs.put() ?
write dbutils.fs.put('abfs://some_address_to_some_abfs_location_you_have/helloworld.txt','Hello World',True) and notice that it outputs "Wrote X bytes." to the console stdout. I don't…
Kaniz Fatma
- 78
- 7
2
votes
0 answers
Apache DBUtils: Standard way to handle different datatype returned from SQL (INT/NULL)?
Is there a standard proper java code to properly handle the following datatype which the result may possible returned as integer or null? I'm using common-dbutils to perform query execution.
--MSSQL
QUERY ="select ID from Users where Name='test'"…
lancelot _sin
- 25
- 4
2
votes
2 answers
Spark use dbutils.fs.ls().toDF in .jar file
I'm trying to package my jar based off of code in a databricks notebook.
I have the following line that works in databricks but is throwing an error in the scala code:
import com.databricks.dbutils_v1.DBUtilsHolder.dbutils
val spark = SparkSession
…
steven hurwitt
- 183
- 2
- 15
2
votes
1 answer
dbutils.notebook.run does not pick run notebook saved on DBFS location
I export my databricks workspace directory (/Users/xyz/) contents which has several python notebooks and scripts onto a databricks specific location for e.g. /dbfs/tmp and then try to call the following code to run a python notebook named xyz.py…
Pankaj Verma
- 51
- 4
2
votes
0 answers
ModuleNotFoundError: No module named 'pyspark.dbutils' while running multiple.py file/notebook on job clusters in databricks
I am working in TravisCI, MlFlow and Databricks environment where .tavis.yml sits at git master branch and detects any change in .py file and whenever it gets updated, It will run mlflow command to run .py file in databricks environment.
my…
Sachin Sharma
- 352
- 5
- 18
1
vote
0 answers
How can I Convert 'pyspark.dbutils.DBUtils' to 'dbruntime.dbutils.DBUtils' in Databricks
I am working on a project where we have some helper functions that uses dbutils and they were initially used as notebook but now they got converted to python modules. Now I cannot access those methods as they cannot find dbutils.
I searched for ways…
Nikunj Kakadiya
- 2,689
- 2
- 20
- 35
1
vote
1 answer
Using databricks dbutils in spark submit (Scala) - Null pointer exception
Im trying to use dbutils in scala spark. Im submitting this job on databricks using spark submit. But, Im getting null pointer exception.
import com.databricks.dbutils_v1.DBUtilsHolder.dbutils
try{
val s3_ls =…
sks
- 145
- 1
- 1
- 9