3

I am writing a Trident topology to process stream of data from Kafka and feed in to Redis and Cassandra. I am able to write the data in to Cassandra. Now I would like to write the same data in to Redis.

Is there a way to duplicate the tuples and branch it in to 2 flow where one goes in to Redis and another goes in to Cassandra?

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Karthick Selvaraj
  • 53
  • 4

1 Answers1

3

For Trident you can go with smth like this:

TridentTopology topology = new TridentTopology();
Stream stream = topology.newStream("MySpout", spout);
stream.partitionPersist(...); // to Redis
stream.partitionPersist(...); // to Cassandra

So it will be saving the data from your stream to both databases in parallel.

However I would also think if such parallel thing should be done inside a single topology or if having two different topologies reading from the same topic is a better idea. Imagine Cassandra cluster goes down. In case of two topologies you'll still be able to continue saving the data to Redis. But if there's just a single topology, every tuple failed to go to Cassandra most likely will result in a FailedException to trigger replaying and every subsequent replay of a tuple will involve saving the tuple to Redis once again unnecessarily.

bopcat
  • 378
  • 4
  • 13