Stack Overflow
Stack Overflow

Questions tagged [databricks-community-edition]

85 questions
7
votes
3 answers

Execute multiple notebooks in parallel in pyspark databricks

Question is simple: master_dim.py calls dim_1.py and dim_2.py to execute in parallel. Is this possible in databricks pyspark? Below image is explaning what am trying to do, it errors for some reason, am i missing something here?
Chandra
  • 371
  • 3
  • 10
7
votes
2 answers

How can I import data downloaded from Kaggle to DBFS using Databricks Community Edition?

I managed to download datasets from Kaggle using Kaggle API. And the data was stored under the directory of /databricks/driver. %sh pip install kaggle %sh export KAGGLE_USERNAME=my_name export KAGGLE_KEY=my_key kaggle competitions download -c…
Memphis Meng
  • 1,267
  • 2
  • 13
  • 34
5
votes
1 answer

Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:

Trying to read delta log file in databricks community edition cluster. (databricks-7.2 version) df=spark.range(100).toDF("id") df.show() df.repartition(1).write.mode("append").format("delta").save("/user/delta_test") with…
ashley
  • 53
  • 1
  • 6
4
votes
1 answer

Entering a proper path to files on DBFS

I uploaded files to DBFS: /FileStore/shared_uploads/name_surname@xxx.xxx/file_name.csv I tried to access them by pandas and I always receive information that such files don't exist. I tried to use the following…
QbS
  • 425
  • 1
  • 4
  • 17
4
votes
3 answers

How can I connect Databricks Community Edition cluster from PyCharm

I want to work on some small exercise projects, I wish to use databricks cluster. Can this be done. I am hoping there is some way to connect databricks cluster through databricks-connect utility. Just need some steps. Thanks in advance.
Manish
  • 1,144
  • 8
  • 12
3
votes
1 answer

How to Save Great_Expectations suite locally on Databricks (Community Edition)

I'm able to save a Great_Expectations suite to the tmp folder on my Databricks Community Edition as follows: ge_partdf.save_expectation_suite('/tmp/myexpectation_suite.json',discard_failed_expectations=False) But the problem is, when I restart the…
Patterson
  • 1,927
  • 1
  • 19
  • 56
3
votes
2 answers

Databricks cannot save stream checkpoint

I'm trying to set up the stream to begin processing incoming files. Looks like Databricks is unable to save a checkpoint. I tried location in ADLS Gen2 and DBFS with the same result. Databricks creates needed folder with some scructure but cannot…
Vik Muzychko
  • 51
  • 6
3
votes
1 answer

How to create Personal Access Token in databricks?

I am new to Databricks. I have a community edition account in databricks. However, I am not able to find the Access token tab when I click User Settings. Am I missing something over here? Please help
Pritish
  • 658
  • 3
  • 16
  • 38
3
votes
3 answers

Can't Access /dbfs/FileStore using shell commands in databricks runtime version 7

In databricks runtime version 6.6 I am able to successfully run a shell command like the following: %sh ls /dbfs/FileStore/tables However, in runtime version 7, this no longer works. Is there any way to directly access /dbfs/FileStore in runtime…
Willard
  • 502
  • 8
  • 21
2
votes
2 answers

How to get the whole cluster information in azure databricks at the runtime?

The below code was working for the older version and the version has changed the code is not working in databricks. Latest Version :12.0 (includes Apache Spark 3.3.1, Scala…
Venkatesh
  • 91
  • 1
  • 9
2
votes
1 answer

Databricks notebook: use magic commands for several lines

I may be missing the obvious, but: I am using the Databricks community edition notebook. I am trying to use several %fs lines within the same cell Is this possible... ? I tried this, as cell content: %fs rm /FileStore/tables/file.txt %fs ls…
horace_vr
  • 3,026
  • 6
  • 26
  • 48
2
votes
1 answer

Unable to mount Azure ADLS Gen 2 on from Community Edition of Databricks : com.databricks.rpc.UnknownRemoteException: Remote exception occurred

I am trying to mount ADLS Gen 2 from my databricks Community Edition, but when I run the following code: test = spark.read.csv("/mnt/lake/RAW/csds.csv", inferSchema=True, header=True) I get the error: com.databricks.rpc.UnknownRemoteException:…
Patterson
  • 1,927
  • 1
  • 19
  • 56
2
votes
1 answer

Databricks dbfs file read issue

I am trying to open a file that i uploaded to the dbfs location. However, I get error while trying to open the file but I can see the file when I do a ls. Also there is no issue while reading the file to a RDD. Can someone explain the behavior of…
restrepo
  • 33
  • 1
  • 4
2
votes
1 answer

Data Bricks Community Edition Cannot Read Uploaded Table (Was Working Before)

So when I was trying to work with some ideas on DataBricks Community Edition today, I suddenly find out the python pandas can no longer read a existing uploaded table. The directories were working before, but none of my previously uploaded tables…
stan12
  • 31
  • 4
2
votes
1 answer

Databricks Notebooks - how to know which kernel

I am using Databricks community edition for running spark workloads . I understand it uses kernels to run the notebooks Is there anyway to identify which kernel the notebooks uses to run ? How exactly the notebook will run behind the scenes( Very…
Nag
  • 1,818
  • 3
  • 24
  • 41
1
2 3 4 5 6