0
Attempted to update an empty pipeline. 
This error usually means that no tables were discovered in your specified Notebook libraries. 
Please verify that your Notebook libraries include table definitions.

DLT notebook code

%python

import dlt
@dlt.table
def kafka_raw():
    TOPIC = "<event_hub>"
    BOOTSTRAP_SERVERS = "<event_hub>.servicebus.windows.net:9093"
    EH_SASL = 'kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://<event_hub>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<event_hub shared_access_policy primary_key>;";'

    return (
        spark.readStream.format("kafka")
        .option("subscribe", TOPIC)
        .option("kafka.bootstrap.servers", BOOTSTRAP_SERVERS)
        .option("kafka.sasl.mechanism", "PLAIN")
        .option("kafka.security.protocol", "SASL_SSL")
        .option("kafka.sasl.jaas.config", EH_SASL)
        .option("kafka.request.timeout.ms", "60000")
        .option("kafka.session.timeout.ms", "60000")
        .option("failOnDataLoss", "false")
        .option("startingOffsets", "earliest")
        .load()
    )
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Aseem
  • 5,848
  • 7
  • 45
  • 69
  • Hmm, strange - do you see any warnings in the DLT event log in UI? Did you check that pipeline points to the correct path or branch in repo? – Alex Ott Aug 19 '23 at 10:50
  • Refer to https://stackoverflow.com/a/76948856/3842788 – Aseem Aug 21 '23 at 21:40
  • your source code there doesn't match what you posted here. For example, `@dlt.create_table` doesn't exist – Alex Ott Aug 22 '23 at 06:46

1 Answers1

0

As @Alex Ott said check the source code path, whether it is taking the notebook having dlt definitions.

Below, is the normal notebook.

enter image description here

Then, I have given this notebook source code path to dlt pipeline.

enter image description here

After, running i got same error as you. So, in Source code option give the correct notebook path having your @dlt.table decorator.

I changed to original notebook.

enter image description here

Then got output as expected.

Output: enter image description here

JayashankarGS
  • 1,501
  • 2
  • 2
  • 6