Stack Overflow
Stack Overflow

Questions tagged [databricks-sql]

Questions about Databricks SQL

For questions about Databricks SQL - a serverless data warehouse on the Databricks Lakehouse Platform that lets you run all your SQL and BI applications at scale with improved performance, a unified governance model, open formats and APIs, and your tools of choice

357 questions
6
votes
1 answer

spark 3.2.1 Apache spark table incompatible data type with parquet

Situation: I get a parquet file generated for me every X amount of time. Can't change the column type of the file, nor parquet schema. Can't modify and rewrite the parquet to a new file location because it has to be picked up from there. …
Jose R
  • 738
  • 2
  • 10
  • 26
5
votes
2 answers

How to use OPTIMIZE ZORDER BY in Databricks

I have two dataframes(from a delta lake table) that do a left join via an id column. sd1, sd2 %sql select a.columnA, b.columnB, from sd1 a left outer join sd2 b on a.id = b.id The problem is that my query takes a long time, looking for…
Ana Isael Morales
  • 110
  • 1
  • 5
5
votes
4 answers

How to get the cluster's JDBC/ODBC parameters programmatically?

Databricks documentation shows how get the cluster's hostname, port, HTTP path, and JDBC URL parameters from the JDBC/ODBC tab in the UI. See image: (source: databricks.com) Is there a way to get the same information programmatically? I mean using…
Emer
  • 3,734
  • 2
  • 33
  • 47
4
votes
2 answers

DROP Multiple Tables in Database using Databricks

I am trying to drop multiple tables in databrick scala using the following command select 'DROP TABLE ' + tableName from ABC where tableName LIKE 'in%' Error results saying Error in SQL statement: AnalysisException: Table or view not found:…
user3418170
  • 93
  • 6
4
votes
1 answer

Delta Live Tables for Batch Incremental Processing

Is it possible to use Delta Live Tables to perform incremental batch processing? Now, I believe that this code will always load all of the data available in the directory when a pipeline is run, CREATE LIVE TABLE lendingclub_raw COMMENT "The raw…
Minura Punchihewa
  • 1,498
  • 1
  • 12
  • 35
4
votes
1 answer

Difference between CREATE TEMPORARY VIEW vs Createorreplacetempview in spark databricks

In databricks, what's the difference between the two methods. %sql CREATE TEMPORARY VIEW diamonds USING CSV OPTIONS (path "/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv", header…
Shubham
  • 41
  • 1
  • 1
  • 2
3
votes
1 answer

Adding custom column to calendar table containing the year number of the following year on and after 2nd Sunday of December of each year

I have created the following calendar table: WITH dates AS ( SELECT EXPLODE(SEQUENCE(TO_DATE('1970-01-01'), TO_DATE('2100-12-31'), INTERVAL 1 DAY)) AS calendar_date ), calendar_table AS ( SELECT YEAR(calendar_date) * 10000 +…
QueryingQuail
  • 45
  • 6
3
votes
1 answer

Compute statistics on a delta table in SQL in Databricks

I'm trying to compute statistics on a delta table in SQL from Databricks. I do : ANALYZE TABLE '/my_dir/my_table' COMPUTE STATISTICS and I get an error message : Error in SQL statement: ParseException: no viable alternative at input 'ANALYZE TABLE…
TOMC
  • 123
  • 10
3
votes
2 answers

Optimize a Delta table used to make queries that use GROUP BY

Working in a Delta table. When it comes to optimising my delta table, I have learned a few things. Partition by: Is beneficial when the column in which the partition is made is used in a "where" condition. i.e In a previous step. Let's say this is…
Ana Isael Morales
  • 110
  • 1
  • 5
3
votes
2 answers

New Databricks JDBC driver version doesn't recognize JDBC URL

I had been using the Databricks JDBC driver version 2.6.22 and tried to upgrade to 2.6.27. However, after upgrading I get messages saying my JDBC URLs are invalid when trying to connect. These JDBC URLs work fine with the old version of the driver…
hamdog
  • 991
  • 2
  • 10
  • 24
3
votes
1 answer

Databricks DELTA CTAS with LOCATION using %sql

DELTA does not have CREATE TABLE LIKE. It does have CTAS. I want to copy the definition of a table only, but also specify the LOCATION. E.g. this does not work: CREATE TABLE IF NOT EXISTS NEW_CUSTOMER_FEED AS SELECT * from NEW_CUSTOMER_FEED WHERE 1…
thebluephantom
  • 16,458
  • 8
  • 40
  • 83
3
votes
2 answers

How to retrieve a column value from DESCRIBE DETAIL

I would like to use the "Last modified" value from the description of my table in databricks. I know how to get all columns from the table by using "DESCRIBE DETAIL table_name", but I wish to simply get the last modified value since I need to use it…
Pontus Johansson
  • 61
  • 1
  • 4
3
votes
2 answers

Databricks SQL analytics

am trying to do this tutorial about databricks sql analytics (https://learn.microsoft.com/en-us/azure/databricks/sql/get-started/admin-quickstart) but when i create my databricks workspace i do not have the icon at the bottom of the sidebar to acces…
Mbula Guy Marcel
  • 81
  • 1
  • 4
  • 11
2
votes
1 answer

Databricks SQL restful API to query delta table

As of today, can we use any Databricks SQL Restful API to query Delta Tables stored in ADLS from any external UI?There is some information mentioned over this link https://docs.databricks.com/sql/api/index.html? but not sure how to use…
krishcoolster
  • 53
  • 5
2
votes
2 answers

How to get the whole cluster information in azure databricks at the runtime?

The below code was working for the older version and the version has changed the code is not working in databricks. Latest Version :12.0 (includes Apache Spark 3.3.1, Scala…
Venkatesh
  • 91
  • 1
  • 9
1
2 3
…
23 24