2

This happens when some files are deleted from the data source that Autoloader stream is reading from.

try:
    raw_df = spark.readStream.format("cloudFiles") \
            .option("cloudFiles.format","csv") \
            .option("cloudFiles.includeExistingFiles", "true") \
            .option("cloudFiles.allowOverwrites", "true") \
            .option("cloudFiles.schemaLocation", 
                opPath.outputPath +"/checkpoints/" + storageAccountInfo.adlsContainerName) \
            .option("delimiter","\t")\
            .load(source)

  except Exception as f:
    print(f)

Error Image

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
SWATHI.M SHETTY
  • 21
  • 1

1 Answers1

0

I realized this was happening because some files were deleted from the blob storage which was the source of cloudfiles stream.

SWATHI.M SHETTY
  • 21
  • 1