2

I'm facing a weird situation were my sagas sometimes just stop invoking next steps while no errors are written to the logs. so for example consider this (pseudo)code:

public async Task Handle(Step1 step1)
{
  //... do something
  await _bus.Send(new Step2 { Id = step1.Id });
}

public async Task Handle(Step2 step2)
{
  //... do something else
  // go to next step      
}

In this case step1 is handled fine but there are just no signs of Handle(Step2 step2) ever being invoked with the saga data just sitting in the SQL table with revision field being equal to 1 and no messages in main or error queues of the RabbitMq. Is at-least-once delivery guarantee applicable here? Any suggestions of how this is possible or which debug information I could add? I'm using MySQL persistence and RabbitMq transport with this config:

services.AddRebus(configure => configure
    .Logging(l => l.MicrosoftExtensionsLogging(loggerFactory))
    .Transport(t => t.UseRabbitMq(rabbitMqConnection, "rebusqueue"))
    .Sagas(s => 
          {
              s.StoreInMySql(dbConnection, "Rebus_Saga_Data", "Rebus_Saga_Data_Index");
              s.EnforceExclusiveAccess();
          })
     .Timeouts(t => t.StoreInMySql(dbConnection, "Rebus_Saga_Timeouts"))
     .Routing(r => r.TypeBased().MapAssemblyOf<MessageBase>("rebusqueue")));
Oleg Golovkov
  • 303
  • 2
  • 7
  • Could something else by any chance be consuming messages from the queue "rebusqueue"? – mookid8000 Aug 31 '20 at 17:43
  • @mookid8000 I can't think of any. Do I understand it correctly that the message is reported as "processed" to rabbitmq only after the `Handle` method completed invocation without exceptions? So I can be sure that `await _bus.Send(new Step2 { Id = step1.Id })` from my question was successfully invoked, the `Step2` message was successfully persisted in RabbitMq and only after that the `Step1` message was marked as complete and removed from RabbitMq? – Oleg Golovkov Sep 01 '20 at 10:18
  • Rebus ACKs the message with RabbitMQ as the _very last thing_ after having handled it. The order is like this: 1) fetch saga matching the incoming message, 2) invoke your handler code, 3) update saga data in database, 4) send outgoing messages, 5) ACK the message – mookid8000 Sep 01 '20 at 10:20
  • @mookid8000 ok thanks, I see, is there an easy way to configure logging to see all sent/received/seen/failed etc messages (maybe just IDs or full bodies serialized to JSON)? – Oleg Golovkov Sep 01 '20 at 10:47
  • not really, no - if you're interested, I can debug your problem if you can reproduce it and post the code in a repository somewhere – mookid8000 Sep 01 '20 at 10:48
  • @mookid8000 the issue is that it's flaky and I can't see a pattern here. I also see sometimes my sagas are started several times for the same message ID, my logic handles those but still curious why it happens - I don't have any retries when doing `await _bus.Send(..)`. Anyway I added manual logging around each `_bus` call and will continue monitoring, thanks for your replies I'll post my results if I manage to figure out the issue – Oleg Golovkov Sep 01 '20 at 10:57

0 Answers0