I understand generally spark streaming is used to process the data in streams, but can I use it to process requests? This question is similar to this question, but with lesser volume most of the time.
My argument is that instead of having an intermediate service to read the message from the queue and then start the processing using spark, can I leverage Spark Streaming to read from the queue?
Additional info:
- The system will see larger requests on certain times of the day and very little requests for the rest of the day.
- Each request will have few parameters that determine what data the spark job should read and how to process it.
