We have a few thousand IoT devices that send us their temperature every second. The input source can be MQTT or JSON (or a queue if needed).
Our goal is to near continuously process data for each of these devices and calculate the average for the last hour, etc. Lastly, we want it to send a notification if it exceeds a certain level.
What are some good alternatives stream processing tools that are flexible and scalable? Is Kafka the way to go, or are there alternatives?
Have you looked at IBM's IoT foundation?
https://www.ibm.com/cloud/internet-of-things
IBM also offers kafka based Event Streams. https://www.ibm.com/cloud/event-streams
You could use a new platform called Quix. https://quix.ai/
They have an MQTT connector - hiveMQ I believe. They also have connectors for slack or twillio to send your notification. You just have to write your logic for whatever calculation you want. although they do provide some pre built tansformations for thing slike rolling average etc.
I believe its running on Kafka but you dont have to know the ins and outs of setting all of that up.
I built something similar on there and it works really well.
You're absolutely right that Kafka is a great solution for this use case.
One of Kafka's main use-cases is high throughput environments like you're describing. It is exceptional for enabling analytics on real time data (e.g. finding average temperature from thousands of devices) and connecting multiple services. You can, of course, host it yourself. Alternatively (as in other suggestions) IBM provides it as a cloud service called Event Streams - and have a page on using it for Real Time Data Streams.
An alternative to the suggestions above would be Redpanda as the streaming patform and Bytewax as the processing solution for calculating the average and notifying.
