0

When I have multiple stages of aggregations, each stage use output of last stage as input, I know its not supported yet as far as spark 2.3.2, what`s your work around?

dunlu_98k
  • 209
  • 2
  • 3
  • 11
  • I know one way is to store output of last stage into Kafka, then readstream from Kafka and do the next stage, but what a huge waste of Serializations/Deserializations – dunlu_98k Oct 14 '18 at 08:02
  • 1
    Possible duplicate of [Multiple aggregations in Spark Structured Streaming](https://stackoverflow.com/questions/41011002/multiple-aggregations-in-spark-structured-streaming) – Harichandan Pulagam Jun 13 '19 at 15:19

0 Answers0