Stack Overflow
Stack Overflow

Questions tagged [gcp-databricks]

For questions about the usage of Databricks Lakehouse Platform on Google Cloud Platform

Databricks on Google Cloud Platform (GCP) has some specifics that should be covered by this tag

15 questions
4
votes
1 answer

Why does Databricks only plot 1000 rows?

Is there any way in Databricks to plot more than 1000 rows with the built in visualization? I tried using limit() function, but it still shows only the first 1000.
JAdel
  • 1,309
  • 1
  • 7
  • 24
3
votes
3 answers

Databricks - How to access Workspace Files in init scripts

Hope everyone is doing well... We are exploring to see if it will be possible to organize a few of our jars as part of a folder in Workspace and have it moved around as part of the init scripts. For example, in the workspace we have the following…
rainingdistros
  • 450
  • 3
  • 11
3
votes
1 answer

How to use Azure DataBricks Api to submit job?

I am a beginner in Azure Databricks and I want to use APIs to create cluster and submit job in python. I am stuck as I am unable to do so. Also if I have an existing cluster how will the code look like? I got job id after running this code but…
Coderash
  • 47
  • 1
  • 9
3
votes
1 answer

Can't read directly from pandas on GCP Databricks

Usually on Databricks on Azure/AWS, to read files stored on Azure Blob/S3, I would mount the bucket or blob storage and then do the following: If using Spark df = spark.read.format('csv').load('/mnt/my_bucket/my_file.csv', header="true") If using…
uimarer
  • 33
  • 3
2
votes
1 answer

Does Dataproc support Delta Lake format?

Is the Databricks Delta format available with Google's GCP DataProc? For AWS and AZURE it is clear that this is so. However, when perusing, researching the internet, I am unsure that this is the case. Databricks docs less clear as well. I am…
thebluephantom
  • 16,458
  • 8
  • 40
  • 83
2
votes
1 answer

How to specify a python version on databricks?

I am using GCP to run Databricks and using pip to install guidedlda. It seems like guidedlda uses atleast python version 3.7. How can I mention the python version in databricks configured for GCP?. I can see an option on Azure and AWS but not on…
Amit Pal
  • 10,604
  • 26
  • 80
  • 160
1
vote
1 answer

While mounting S3 Bucket on the DataBricks whic is on GCP environment getting error "Key based mount points are not supported"

Here is the code: access_key = dbutils.secrets.get(scope = "dll-gcp", key = "aws-access-key") secret_key = dbutils.secrets.get(scope = "dll-gcp", key = "aws-secret-key") encoded_secret_key = secret_key.replace("/", "%2F") aws_bucket_name =…
Shahid Haider
  • 43
  • 5
1
vote
0 answers

ClassNotFoundException when connecting from Databricks

I get a ClassNotFoundException when I try connecting from Databricks on GCP: Py4JJavaError: An error occurred while calling o1808.jdbc. : com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host localhost, port 1433 has…
kelloti
  • 8,705
  • 5
  • 46
  • 82
1
vote
0 answers

Databricks cluster failing to start

Cluster terminated.Reason:Cloud provider launch failure A cloud provider error was encountered while launching worker nodes. See the Databricks guide for more information. GCP error message: Compute Quota Exceeded for databricksharish2022 in region…
Harish Srivastava
  • 11
  • 2
1
vote
0 answers

BAD_REQUEST: GCP workspace creation request validation failed

I'm trying to connect my databricks to GCP and I have given following permissions on CLI: compute, container, deploymentmanager, iam using the command "gcloud services enable [servicename]" I've attached a screenshot for the error I got BAD_REQUEST:…
sehaj
  • 11
  • 1
0
votes
0 answers

GCP hosted Databricks - DBFS temp files - Not Found

Hieveryone, I've been working on obtaining DDL at the schema level in Hive Metastore within GCP-hosted Databricks. I've implemented a Python code that generates SQL files in the dbfs/temp directory. However, when running the code, I'm encountering a…
Hazal
  • 13
  • 3
0
votes
1 answer

save each line of string as a file in spark

I am working on a use case wherein I need to save each line of text in RDD as a separate file in Google Cloud Storage. The run platform is Databricks with spark version 3.2.x and language is Scala. Can you please point me to relevant document that…
Aishwary Shukla
  • 450
  • 1
  • 7
  • 21
0
votes
0 answers

Not able to create cluster on Databricks, GKE deleted without my permission

I started a free trial with Databricks and everything was running perfectly. The trial ended on the 28th April and I am assuming I was simply transferred to the normal premium paid plan. I last used my general cluster on the 2nd May. Since coming…
ojp
  • 973
  • 1
  • 11
  • 26
0
votes
1 answer

How to load data from Azure Databricks SQL to GCP Databricks SQL

Is there an easy way to load data from Azure Databricks Spark DB to GCP Databricks Spark DB?
user3402692
  • 47
  • 1
  • 4
0
votes
1 answer

Is it possible to mount GCP storage bucket as a mount point in DataBricks which is also in the same project?

"I have a requirement where I need to mount the GCP storage bucket on the GCP-Databrciks cluster but I can't find any appropriate documents on it; my intention is to use that bucket as an external table location"
Shahid Haider
  • 43
  • 5