NServiceBus Message Replay Archive Architecture

Question

I'm building an application that needs to preserve copies of the messages it is sending so that I can replay all the messages at a later stage. This is necessary because the processing of the messages will change dramatically over the course of development, but the data must be captured ASAP since it is realtime observation data. I can't seem to find any built-in functionality that directly addresses this and while I could write a custom tool to persist the data, that seems to contradict the purpose of using NServiceBus in the first place. Some options I'm considering:

Use the ForwardReceivedMessagesTo functionality of the Target bus to create an Archive queue, and build a simple application which uses this Archive queue as an input queue for simply forwarding messages onto the Target bus whenever the Replayer tool runs. This does clear the Archive queue, requiring it to be backed up first using the mqbkup utility, but this can be automated as part of the replay process. Alternatively, using two alternating Archive queues (one taking in new messages, and one for replaying) would solve this.
Use a publish/subscribe model and have an Archiver subscribe to the Target queue, placing the message in an Archive queue. A Replayer tool similar to the one above could use the Archive queue as an input queue and forward the messages to the Target. This would also clear the Archive queue, requiring one of the solutions above.
MassTransit people mention something called BusDriver that allows copying between queues but I can't find anything more about it.

My primary concern is to choose the approach that is least likely to lose data as once an observation is made it can never be made again outside of a narrow time window. This seems like it should be a common problem and yet I can't seem to find a straightforward solution to it. Suggestions?

Update I've decided to go with a journalled Target queue. I'll have an Archiver use the journal as an input and store messages to a database (could just be file-based), as well as allow for replaying messages from that database to the Target queue. While it would be possible to write a tool that copies messages from the journal queue to the target queue, the real problem - from a practical perspective - is that of managing the journal queue: it can't be backed up easily (mqbkup brings down the MSMQ service, which is unacceptable), and operating non-destructively on the queue requires me to write an MSMQ-based tool when I'd rather stick to the NServiceBus level of abstraction. Ultimately, MSMQ is a transport and not a store of messages so it needs to be treated as such.

score 1 · Answer 1 · answered Jun 06 '12 at 16:45

#3 BusDriver command-line reference:

BusDriver is a command-line utility used to administer queues, service bus instances and other things related to MassTransit.

Command-Line Reference

  busdriver.exe [verb] [-option:value] [--switch]

    help, --help        Displays help

    count               Counts the number of messages in the specified
                        queue

      -uri              The URI of the queue

    peek                Displays the body of messages in the queue without
                        removing the messages

      -uri              The URI of the queue
      -count            The number of messages to display

    move                Move messages from one queue to another

      -from             The URI of the source queue
      -to               The URI of the destination queue
      -count            The number of messages to move

    requeue             Requeue messages from one queue to another

      -uri              The URI of the queue
      -count            The number of messages to move

    save                Save messages from a queue to a set of files

      -uri              The URI of the source queue
      -file             The name of the file to write to (will have .1, .2
                        appended automatically for each message)
      -count            The number of messages to save
      --remove          If set, the messages will be removed from the queue

    load                Load messages from a set of files into a queue

      -uri              The URI of the destination queue
      -file             The name of the file to read from (will have .1, .2
                        appended automatically for each message)
      -count            The number of messages to load
      --remove          If set, the message file will be removed once the
                        message has been loaded

    trace               Request a trace of messages that have been received
                        by a service bus

      -uri              The URI of the control bus for the service bus
                        instance
      -count            The number of messages to request

        status          Request a status probe of the bus at the endpoint

          -uri          The URI of the control bus for the service bus instance

    exit, quit          Exit the interactive console (run without arguments
                        to start the interactive console)

Examples:

    count -uri:msmq://localhost/mt_server
        Returns the number of messages that are present in the local
        MSMQ private queue named "mt_server"

    peek -uri:msmq://localhost/mt_client
        Displays the body of the first message present in the local
        MSMQ private queue named "mt_client"

    trace -uri:msmq://localhost/mt_subscriptions
        Requests and displays a trace of the last 100 messages received
        by the mt_subscriptions (the default queue name used by the
        subscription service, which is part of the RuntimeServices)

    move -from:msmq://localhost/mt_server_error -to:msmq://localhost/mt_server
        Moves one message from the mt_server_error queue to the mt_server
        queue (typically done to reprocess a message that was previously
        moved to the error queue due to a processing error, etc.)

score 1 · Answer 2 · answered Mar 12 '11 at 13:54

1

The first option seems viable. I would add that NSB comes with a tool named ReturnToSourceQueue.exe that will replay messages for you. You could turn on journalling to keep messages, but something would have to clean that up since it doesn't roll. We have NetApp and we've used its SnapMirror to backup queue data, I'm sure there are similar things out there.

answered Mar 12 '11 at 13:54

Adam Fyles

6,030
1
23
29

As I understand it, ReturnToSourceQueue will remove the messages from the archive queue, which will clear out the archive. Further research shows that using mqbkup to backup the archive queue will bring down the MSMQ service, which is unacceptable. Thanks for bringing Journaling to my attention; it is better than ForwardReceivedMessagesTo because it ensures a copy of the message exists regardless of whether the subscriber queue ever accepts it. There is still the problem of replaying messages non-destructively. I've updated my question with my current conclusions for future reference. – Ryan Shaw Mar 14 '11 at 14:41

NServiceBus Message Replay Archive Architecture

2 Answers2