Databricks spark.readstream format differences

Question

I am having confusion on the difference of the following code in Databricks

spark.readStream.format('json')

vs

spark.readStream.format('cloudfiles').option('cloudFiles.format', 'json')

I know cloudfiles as the format would be regarded as Databricks Autoloader . In performance/function comparison , which one is better ? Anyone has some experience on that?

Thanks

Alex Ott · Answer 1 · 2022-01-22T22:13:29.403

4

There are multiple differences between these two. When you use Auto Loader you get at least, there are more things (see doc for all details):

Better performance, scalability and cost efficiency when discovering new files. You can either use file notification mode (when you get notified about new files using cloud-native integration) or optimized file listing mode that uses native cloud APIs to list files and directories. Spark's file streaming is relying on the Hadoop APIs that are much slower, especially if you have a lot of nested directories and a lot of files
Support for schema inference and evolution. With Auto Loader you can detect changes in the schema for JSON/CSV/Avro, and adjust it to process new fields.

edited Jan 22 '22 at 22:13

answered Jan 22 '22 at 10:35

Alex Ott

80,552
8
87
132

Just for clarity, both of these queries achieve the same function, i.e. reads new files that become available in the data source, yes? However, with Auto Loader, one can enjoy the benefits that you have mentioned, such as better performance, scalability and so on? – Minura Punchihewa Jun 29 '22 at 17:19
1

Yes. they achieve the same functionality, Autloader is just more optimized – Alex Ott Jun 29 '22 at 17:56

Databricks spark.readstream format differences

1 Answers1