1

I am pulling messages from rabbit and aggregating them. I have noticed that even with autoAck=false the messages are ackd as soon as they have been aggregated. This means that if my process crashes the messages will not be re-queued for rety.

I have plugged in the leveldb java db as the aggregation repo which solves the problem, as I presume it will only ack once it has aggregated and stored in repo? To be honest level-db doesn't make me feel comfortable based on its activity and issues with the C version, guess I can switch it to a SQL repo.

So. Question is, is there something I can do so Camel will only Ack the messages of exchanges once the aggregated message has completed its journey? After aggregation my next step is to simple write to disk.

Skynet5
  • 63
  • 7

1 Answers1

1

That is by design, then aggregator is a stateful EIP. So the incoming message is handed over to the aggregator, and then continued, and thus the rabbit consumer that processed the message is completed and its acked.

You need to plugin a persistent aggregation repository as you say.

Since you last step is to write to disk, I would re-design so you take each message from rabbit and write to disk first, and then do another from disk -> aggregate -> disk.

Claus Ibsen
  • 56,060
  • 7
  • 50
  • 65
  • re: disk approach. surely I'll have the same problem. it'll move the file once in the aggregate and if it fails then I'll have a problem. I can't lose any messages. It sounds like I need to use the persistent store...but leveldb and hawtdb don't really look serious enough. shame really...as comsuming from a queue and only acking once batched should be trival and is fairly easy to write manually. Any other thoughts? – Skynet5 Jul 07 '17 at 09:27
  • Using plain files is better than a leveldb store, as plain files are easier to manage. And as you say you need to store in files anyway afterwards. – Claus Ibsen Jul 07 '17 at 10:48
  • so would I not be using a repository at all in this case? rmq -> disk1 -> aggregator -> disk2. I guess I'm trying to understand that at disk1 stage the consumer will move the file when it hits the aggregator...so I'm still in the same position. If the system crashes...those being aggregated will be lost, and the files will be moved to archive and I'll have to work out whats what. – Skynet5 Jul 07 '17 at 11:25
  • Avoid the agg repo and just use the files. You have the message on disk are safe, and its much easier to move files back to a folder than having to re-submit messages from rabbitmq. – Claus Ibsen Jul 07 '17 at 11:57
  • so actually I think my understanding is that this doesn't give me what I need for fault tolerance. I'm happy to write to disk from RMQ first. But my aggregation requirement (20k limit files you answered previously) needs to be reliable and self recoverable. It sounds like as you say, the ack ro RMQ on submitting to the aggregator, whilst by design, does kill support for my usecase. Had it not ackd un till aggregation was complete and the message written, the system could have gone down and rabbit would have automatically made those messages available again. – Skynet5 Jul 07 '17 at 12:47