What is the correct way to install the delta module in python?

Question

What is the correct way to install the delta module in python??

In the example they import the module

from delta.tables import *

but i did not find the correct way to install the module in my virtual env

Currently i am using this spark param -

"spark.jars.packages": "io.delta:delta-core_2.11:0.5.0"

Answer 1

As the correct answer is hidden in the comments of the accepted solution, I thought I'd add it here.

You need to create your spark context with some extra settings and then you can import delta:

spark_session = SparkSession.builder \
    .master("local") \
    .config("spark.jars.packages", "io.delta:delta-core_2.12:0.8.0") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

from delta.tables import *

Annoyingly, your IDE will of course shout at you about this as the package isn't installed and you will also be operating without autocomplete and type hints. I'm sure there's a work around and I will update if I come accross it.

The package itself is on their github here and the readme suggests you can pip install but that doesn't work. In theory you could clone it and install manually.

Answer 2

Because Delta's Python codes are stored inside a jar and loaded by Spark, delta module cannot be imported until SparkSession/SparkContext is created.

Answer 3

To run Delta locally with PySpark, you need to follow the official documentation.

This works for me but only when executing directly the script (python <script_file>), not with pytest or unittest.

To solve this problem, you need to add this environment variable:

PYSPARK_SUBMIT_ARGS='--packages io.delta:delta-core_2.12:1.0.0 pyspark-shell'

Use Scala and Delta version that match your case. With this environment variable, I can run pytest or unittest via cli without any problem

from unittest import TestCase

from delta import configure_spark_with_delta_pip
from pyspark.sql import SparkSession


class TestClass(TestCase):
    
    builder = SparkSession.builder.appName("MyApp") \
        .master("local[*]")
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    
    spark = configure_spark_with_delta_pip(builder).getOrCreate()

    def test_create_delta_table(self):
            self.spark.sql("""CREATE IF NOT EXISTS TABLE <tableName> (
                              <field1> <type1>)
                              USING DELTA""")

The function configure_spark_with_delta_pip appends a config option in builder object

.config("io.delta:delta-core_<scala_version>:<delta_version>")

Answer 4

Here's how you can install Delta Lake & PySpark with conda.

• Make sure you have Java installed (I use SDKMAN to manage multiple Java versions)
• Install Miniconda
• Pick Delta Lake & PySpark versions that are compatible. For example, Delta Lake 1.2 is compatible with PySpark 3.2.
• Create a YAML file with the required dependencies, here is an example from the delta-examples repo I created.
• Create the environment with a command like conda env create envs/mr-delta.yml
• Activate the conda environment with conda activate mr-delta
• Here is an example notebook. Note that it starts with the following code:

import pyspark
from delta import *

builder = pyspark.sql.SparkSession.builder.appName("MyApp") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

spark = configure_spark_with_delta_pip(builder).getOrCreate()

Answer 5

If you are facing issues with Jupyter notebook add the below environment variable

from pyspark.sql import SparkSession
import os
from delta import *

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages  org.apache.spark:spark-avro_2.12:3.4.1,io.delta:delta-core_2.12:2.4.0 pyspark-shell'
# RUN spark-shell --packages org.apache.spark:spark-avro_2.12:3.4.1
# RUN spark-shell --packages io.delta:delta-core_2.12:2.4.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

builder = SparkSession.builder.appName("SampleSpark") \
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

spark = builder.getOrCreate()

Answer 6

In my case the issue was I had a Cluster running on a Databricks Runtime lower than 6.1

https://docs.databricks.com/delta/delta-update.html

The Python API is available in Databricks Runtime 6.1 and above.

After changing the Databricks Runtime to 6.4 problem disappeared.

To do that: Click clusters -> Pick the one you are using -> Edit -> Pick Databricks Runtime 6.1 and above