kafka or redis for realtime BI

Question

I am working on a project for real time business intelligence and i am using the elastic stack spark streaming and kafka ? but I am wondering if I may use redis instead of kafka because it appears that redis in an in memory beast that can forward data in real time ,I'm quite confused can anyone help me for choosing the right technologie,I have posted before a question here which architecture for real time BI and they adviced me the next architecture: LOgstash=>kafka=>spark streaming=>elasticsearch=>kibana thanks for your answers

Answer 1

In short , If data are large and data should be reused, you should first consider Kafka otherwise when dealing with real-time messages processing with a minimal latency, you should first try Redis .

You can try Redis since Redis can support a production ELK (Elasticsearch, Logstash, Kibana) Stack. Redis is an in-memory store. This means that it uses its primary memory for storage and processing which makes it much faster than the disk-based Kafka. The only problem with Redis’ in-memory store is that we can’t store large amounts of data for long periods of time. Since the primary in-memory is smaller than a disk, we have to clear it regularly by automatically moving data from in-memory to disks and making room for new data. In addition, Redis doesn’t have the concept of parallelism like Kafka does, where multiple processes can consume the data at the same time.

Architecture with Redis -: logstash(shipper) ->redis <- logstash(indexer) ->elastic search -> kibana . Redis has buffer facility also in case logstash(indexer) is down.

To know ,how u can use redis with logstash and elastic , here is good explaination :- https://ianunruh.com/2014/05/monitor-everything-part-2.html

For collecting for knowledge on redis vs kafka ,u can check this link :- https://logz.io/blog/kafka-vs-redis/ https://www.quora.com/What-are-the-advantages-of-Kafka-over-Redis-for-pub-sub