Stack Overflow
Stack Overflow

Questions tagged [delta-lake]

Delta Lake is an open source project that supports ACID on top of Apache Spark. It provides ACID transactions, scalable metadata handling, time travel, unified batch and streaming source and sink, and is fully compatible with Apache Spark APIs.

From https://delta.io/:

Delta Lake is an open source project that brings ACID transactions to Apache Spark™ and big data workloads. Key Features:

  • ACID Transactions: Data lakes typically have multiple data pipelines reading and writing data concurrently, and data engineers have to go through a tedious process to ensure data integrity, due to the lack of transactions. Delta Lake brings ACID transactions to your data lakes. It provides serializability, the strongest level of isolation level.
  • Scalable Metadata Handling: In big data, even the metadata itself can be "big data". Delta Lake treats metadata just like data, leveraging Spark's distributed processing power to handle all its metadata. As a result, Delta Lake can handle petabyte-scale tables with billions of partitions and files at ease.
  • Time Travel (data versioning): Delta Lake provides snapshots of data enabling developers to access and revert to earlier versions of data for audits, rollbacks or to reproduce experiments.
  • Open Format: All data in Delta Lake is stored in Apache Parquet format enabling Delta Lake to leverage the efficient compression and encoding schemes that are native to Parquet.
  • Unified Batch and Streaming Source and Sink: A table in Delta Lake is both a batch table, as well as a streaming source and sink. Streaming data ingest, batch historic backfill, and interactive queries all just work out of the box.
  • Schema Enforcement: Delta Lake provides the ability to specify your schema and enforce it. This helps ensure that the data types are correct and required columns are present, preventing bad data from causing data corruption.
  • Schema Evolution: Big data is continuously changing. Delta Lake enables you to make changes to a table schema that can be applied automatically, without the need for cumbersome DDL.
  • Audit History: Delta Lake transaction log records details about every change made to data providing a full audit trail of the changes.
  • Updates and Deletes: Delta Lake supports Scala / Java APIs to merge, update and delete datasets. This allows you to easily comply with GDPR and CCPA and also simplifies use cases like change data capture.
  • 100% Compatible with Apache Spark API: Developers can use Delta Lake with their existing data pipelines with minimal change as it is fully compatible with Spark, the commonly used big data processing engine.
1226 questions
25
votes
3 answers

Parquet vs Delta format in Azure Data Lake Gen 2 store

I am importing fact and dimension tables from SQL Server to Azure Data Lake Gen 2. Should I save the data as "Parquet" or "Delta" if I am going to wrangle the tables to create a dataset useful for running ML models on Azure Databricks ? What is the…
learner
  • 833
  • 3
  • 13
  • 24
23
votes
8 answers

Databricks drop a delta table?

How can I drop a Delta Table in Databricks? I can't find any information in the docs... maybe the only solution is to delete the files inside the folder 'delta' with the magic command or dbutils: %fs rm -r delta/mytable? EDIT: For clarification, I…
Joanteixi
  • 427
  • 1
  • 4
  • 10
23
votes
2 answers

Apache Spark + Delta Lake concepts

I have many doubts related to Spark + Delta. 1) Databricks propose 3 layers (bronze, silver, gold), but in which layer is recommendable to use for Machine Learning and why? I suppose they propose to have the data clean and ready in the gold…
Eric Gabriel Bellet Locker
  • 313
  • 2
  • 8
20
votes
7 answers

How to drop a column from a Databricks Delta table?

I have recently started discovering Databricks and faced a situation where I need to drop a certain column of a delta table. When I worked with PostgreSQL it was as easy as ALTER TABLE main.metrics_table DROP COLUMN metric_1; I was looking…
samba
  • 2,821
  • 6
  • 30
  • 85
17
votes
3 answers

Databricks - How to change a partition of an existing Delta table?

I have a table in Databricks delta which is partitioned by transaction_date. I want to change the partition column to view_date. I tried to drop the table and then create it with a new partition column using PARTITIONED BY (view_date). However my…
samba
  • 2,821
  • 6
  • 30
  • 85
13
votes
3 answers

Databricks - is not empty but it's not a Delta table

I run a query on Databricks: DROP TABLE IF EXISTS dublicates_hotels; CREATE TABLE IF NOT EXISTS dublicates_hotels ... I'm trying to understand why I receive the following error: Error in SQL statement: AnalysisException: Cannot create table…
QbS
  • 425
  • 1
  • 4
  • 17
13
votes
1 answer

lakeFS, Hudi, Delta Lake merge and merge conflicts

I'm reading documentation about lakeFS and right now don't clearly understand what is a merge or even merge conflict in terms of lakeFS. Let's say I use Apache Hudi for ACID support over a single table. I'd like to introduce multi-table ACID support…
alexanoid
  • 24,051
  • 54
  • 210
  • 410
13
votes
3 answers

How to rename a column in Databricks

How do you rename a column in Databricks? The following does not work: ALTER TABLE mySchema.myTable change COLUMN old_name new_name int It returns the error: ALTER TABLE CHANGE COLUMN is not supported for changing column 'old_name' with type…
David Maddox
  • 1,884
  • 3
  • 21
  • 32
13
votes
5 answers

Delta Lake rollback

Need an elegant way to rollback Delta Lake to a previous version. My current approach is listed below: import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, testFolder) spark.read.format("delta") .option("versionAsOf", 0) …
Fang Zhang
  • 1,597
  • 18
  • 18
11
votes
2 answers

What are the major differences between S3 lake formation governed tables and databricks delta tables?

What are the major differences between S3 lake formation governed tables and databricks delta tables? they look pretty similar.
MGomez
  • 123
  • 1
  • 5
11
votes
3 answers

How to CREATE TABLE USING delta with Spark 2.4.4?

This is Spark 2.4.4 and Delta Lake 0.5.0. I'm trying to create a table using delta data source and seems I'm missing something. Although the CREATE TABLE USING delta command worked fine neither the table directory is created nor insertInto…
Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
11
votes
6 answers

What is the correct way to install the delta module in python?

What is the correct way to install the delta module in python?? In the example they import the module from delta.tables import * but i did not find the correct way to install the module in my virtual env Currently i am using this spark param…
ofriman
  • 198
  • 1
  • 1
  • 9
10
votes
2 answers

How to write to delta table/delta format in Python without using Pyspark?

I am looking for a way to write back to a delta table in python without using pyspark. I know there is a library called deltalake/delta-lake-reader that can be used to read delta tables and convert them to pandas dataframes. The goal is to write…
FRITTENPIET
  • 101
  • 1
  • 4
10
votes
2 answers

check if delta table exists on a path or not in databricks

I need to delete certain data from a delta-lake table before I load it. I am able to delete the data from delta table if it exists but it fails when the table does not exist. Databricks scala code below // create delete statement val del_ID =…
VNK
  • 125
  • 1
  • 1
  • 8
10
votes
4 answers

How to add a new column to a Delta Lake table?

I'm trying to add a new column to data stored as a Delta Table in Azure Blob Storage. Most of the actions being done on the data are upserts, with many updates and few new inserts. My code to write data currently looks like…
Comrade_Question
  • 305
  • 1
  • 2
  • 10
1
2 3
…
81 82