exceptions in my service stack service not moving messages to dead letter queue

Question

I have a service stack service with the standard service stack RabbitMQ abstraction. Message queues are automatically created for my type MyRequest, and I have a service method which I have set up to process requests from the MyRequest.In Queue

I was expecting that if I threw an exception within this method that the messages would be placed on the dead letter queue. However they are just being removed from the In queue, and do not go to the dead letter queue

public class MyOtherService : AsmServiceBase
{
    public void Any(MyRequest request)
    {
        try
        {
            throw new InvalidOperationException("this is an invalid operation");
        }
        catch (Exception ex)
        {
            Console.Write("exceptions");
            throw;
        }
    }
}

[Route("/api/myrequest", "POST")]
public class MyRequest : HeliosRequestBase<MyResponse>
{
    public string Content { get; set; }
}

public class MyResponse : HeliosResponseBase
{

}

and this is the code in the AppHost that routes the MyRequest messages through to my service method:

RabbitMqServer mqServer = RabbitMqServerFactory
                         .GetRabbitMqServer(m_ServiceDiscovery).Result;
mqServer.RegisterHandler<MyRequest>(ExecuteMessage);
mqServer.Start();

Could someone please explain what I am doing wrong?

updates Images of the queues before

and after (note that the service method is called and the exception thrown for each method)

If I change the RegisterHandler line to be like so

 mqServer.RegisterHandler<MyRequest>(
            x =>
            {
                try
                {
                    object resp =  ExecuteMessage(x);
                    //if I throw an exception here it will go to the DLQ
                    return resp;
                }
                catch (Exception ex)
                {
                    throw;
                }
            }
            );

I can see that the exception is contained in the object that is returned from ExecuteMessage. I guess maybe I need to rethrow it? Or am I doing something wrong here?

update 2 - how the message is published is the issue With help from mythz examples I have been able to determine that the issue is caused by how the message is being published to the queue.

We call teh publish method like so:

 public void PublishOneWay<TRequest>(TRequest request, string queueName, int timeoutInMilliseconds)
    {
        if (m_Disposed)
            throw new ObjectDisposedException("The service client has been disposed and cannot be used.");
        Message message = MakeMessage(request);


        //Updated to just send one-way messages - why bother enquing if we are never going to capture the callback/block on this thread?
        //MessagesToProcess.Add(new Tuple<string, Message, MessageDirection>(queueName, message, MessageDirection.OneWay));
        MqClient.SendOneWay(queueName, message); //will work if I send request instead of message

    }

And the problem seems to be in this code where we construct a new Message object to send.

 private Message MakeMessage<TRequest>(TRequest request)
    {
        Guid messageId = Guid.NewGuid();
        Message newMessage = new Message<TRequest>(request) { ReplyTo = ResponseQueueName, Id = messageId };

        newMessage.Meta = new Dictionary<string, string>();

        newMessage.Meta.Add(ServiceContextConstants.TrackingIdentifier, AsmServiceContext.TrackingIdentifierData?.Value);
        newMessage.Meta.Add(ServiceContextConstants.SessionContext, AsmServiceContext.SessionContextData?.Value);

        return newMessage;
    }

If I send the original request object, then the exceptions are sent to the DLQ as you would expect. It also works if I don't set the ReplyTo property of the message object.

I wonder, is there any property that I need to set on the Message object that would create this behaviour? I may be able to not set the ReplyTo property, but am not sure how much of our code will change as a result.

Answer 1

Edit

If you're using an explicit ReplyTo address any Errors will be sent to that ReplyTo address instead of the DLQ.

If your Response DTO has a ResponseStatus property the Exception will be populated in the ResponseStatus of the Response DTO, otherwise you can read the Exception Info using the generic ErrorResponse DTO, e.g:

var requestMsg = new Message<ThrowVoid>(request)
{
    ReplyTo = $"mq:{request.GetType().Name}.replyto"
};
mqProducer.Publish(requestMsg);

var msg = mqClient.Get<ErrorResponse>(requestMsg.ReplyTo, null);
mqClient.Ack(msg);

msg.GetBody().ResponseStatus.ErrorCode //= InvalidOperationException

---

I'm unable to repro this issue using just normal ServiceStack classes as seen with this commit which works as expected in all MQ Servers.

I've extracted the code for using ServiceStack's RabbitMQ Server below:

public class ThrowVoid
{
    public string Content { get; set; }
}

public class TestMqService : Service
{
    public void Any(ThrowVoid request)
    {
        throw new InvalidOperationException("this is an invalid operation");
    }
}

public class AppHost : AppSelfHostBase
{
     public AppHost(Func<IMessageService> createMqServerFn)
        : base(nameof(TestMqService), typeof(TestMqService).Assembly) {}

    public override void Configure(Container container)
    {
        var mqServer = new RabbitMqServer { RetryCount = 1 };
        container.Register<IMessageService>(c => mqServer);
        mqServer.RegisterHandler<ThrowVoid>(ExecuteMessage);

        AfterInitCallbacks.Add(appHost => mqServer.Start());
    }
}

Where the message is sent with:

using (var mqFactory = appHost.TryResolve<IMessageFactory>())
{
    var request = new ThrowVoid { Content = "Test" };

    using (var mqProducer = mqFactory.CreateMessageProducer())
    using (var mqClient = mqFactory.CreateMessageQueueClient())
    {
        mqProducer.Publish(request);

        var msg = mqClient.Get<ThrowVoid>(QueueNames<ThrowVoid>.Dlq, null);
        mqClient.Ack(msg);

        Assert.That(msg.Error.ErrorCode, Is.EqualTo("InvalidOperationException"));
    }
}

It's impossible to tell what your issue is given a lot of the implementation is hidden behind your own custom classes, but I'd recommend starting with a small stand-alone example like above that does work and add slowly add your custom code to identify what's causing it.

Although I would recommend against using inheritance to hide properties in your DTOs, DTOs are declarative models used to define your Service Contracts, hiding their properties behind inheritance makes it harder for anyone reading it to know exactly what each Service accepts and returns. Inheritance can be useful to define reusable functionality, but DTOs are declarative and should have no implementation so you're basically adding unnecessary coupling to hide the explicit Service Contract making it harder to be able to infer what each Service does by looking at the Service Contracts.