2

At the moment we have code running on spark streaming and we want to migrate it to structured streaming. As far as I can see StreamingListener is only for spark streaming, what should I use when I am using structured streaming?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
Funzo
  • 1,190
  • 2
  • 14
  • 25

1 Answers1

2

The equivalent is StreamingQueryListener. It is documented in the official programming guide, under Reporting Metrics programmatically using Asynchronous APIs.

You can also asynchronously monitor all queries associated with a SparkSession by attaching a StreamingQueryListener (...) Once you attach your custom StreamingQueryListener object with sparkSession.streams.attachListener(), you will get callbacks when a query is started and stopped and when there is progress made in an active query. Here is an example,

The official API is available on for Scala and Java, but you can find examples of its usage in Python, in another thread on Stack Overflow.

user12531688
  • 58
  • 4
  • 1
    Isn't `StreamingQueryListener` and `StreamingListener` two different API for different uses ? one being global and the other being specific to a query ? – Mehdi LAMRANI Dec 27 '19 at 11:02
  • I am not sure which one would be which @MehdiLAMRANI if you put things this way. There are of course two different API's - because structured streaming model, even in batch mode, is significantly different than legacy streaming. – user12531688 Dec 27 '19 at 11:10
  • Imho it all depends on what you are trying to achieve. I personnally use both, with a clear preference for the newest API as it is clearly more flexible – Mehdi LAMRANI Dec 27 '19 at 11:11