18

Now with Azure Service fabric, would there be a use-case for also using a separate queue solution such as Windows Service Bus? The downsides would probably be a new single point of failure, but are there upsides? Queues can add some buffering, but on the other hand, Service Fabric should be able to scale really well and offer stateful capabilities, so no queue buffers should be needed?

user1340582
  • 19,151
  • 35
  • 115
  • 171

2 Answers2

16

Sure, the upside is that services like Azure Service Bus and Azure Storage Queues offer features that are not included out-of-the-box in Service Fabric. So the question to ask yourself is: do you add an external service dependency to get that functionality, or do you stay self-contained by building it yourself on Service Fabric? A self-contained application on Service Fabric is good, but re-inventing existing functionality is bad, so you have to decide where the most value is for you and lean in that direction.

For example, think about..

  • Portability. An application that is self contained on Service Fabric can be hosted wherever Service Fabric can run, which is pretty much anywhere (Azure, other public clouds, your own machines or data center, etc.).
  • No external dependencies means fewer points of failure, a single toolset, and a unified development, deployment, upgrade, and maintenance process.

On the other hand..

  • Services like Service Bus offer a rich set of features. Is it worth spending the time building and maintaining the features you need yourself on Service Fabric?
Vaclav Turecek
  • 9,020
  • 24
  • 29
  • 1
    Vaclav, understood, question for you though, do you know of any open source implementation of a SF Message queue? The thought being if we could get that out into the community then we wouldn't have to recreate the wheel. It doesn't seem like message queue functionality would change very often, once the project was written it would be pretty static. – gperrego Feb 16 '19 at 18:15
7

Good question! I'm noodling around this as well. In my case I'm using a RabbitMQ cluster for queuing. I wanted to avoid it and was hoping to have Stateful services using Reliable Queues. I exposed a method to add the message to the service and use the RunAsync method to dequeue the messages as they arrive. I'm not impressed with the performance using this approach, compared to a stateless service connecting to RabbitMQ. But before I give up, I'm planning to partition the Stateful service across a 5 nodes and see if there are any performance improvements, using the Stateful services a queue consuming workers.

code5
  • 4,434
  • 2
  • 31
  • 24
  • Really interested on your comments about performance of RabbitMQ vs Reliable Queues. What did you end up with? – kenchilada Jan 06 '17 at 15:13
  • 3
    Our app needs to process a very large number of conversations. I started off the with 5 Stateful services each with their own partitioned, using Reliable Qs. The partitions were used to help distribute the load of messages. Going with this route, processing 1000 msgs took 45 secs. Using RMQ, across 3 partitioned Stateless services, processing 1000 msgs only took 0.350 secs!!! A huge significant improvement not only in processing but the need to have Stateful replicas was no longer required. – code5 Jan 07 '17 at 21:19
  • 2
    To further amend what I was saying above, the downside as @Vaclav Turecek mentioned is the application (s) parts and pieces are more scattered not self contained. One of our engineers is currently investigating how to cluster RMQ using Service Fabric Cluster. No updates on this yet. I was hoping to use Azure Service Bus, but the business doesn't want to be tied to Azure. – code5 Jan 07 '17 at 21:28
  • 2
    Hi @code5 - Out of interest what did you end up doing with this? Have been looking at similar approaches and very interested in your performance metrics above. Did you look at RabbitMQ as a guest executable or running in a container? – Rob Bird Sep 25 '17 at 20:47
  • 4
    Hi @RobBird, actually our initially thinking was wrong from how to host RabbitMQ. You see, SFX is an App Cluster to support our Microservices. The same holds true for a Database Cluster. So we decided to just spawn a few Linux servers (you can use Windows as well) and host our RMQueue Cluster. Because we need very high throughput, we went with Rabbit. Message Service Bus offers the maintenance aspect which is nice. Price wise they are comparable. If it wasn't for the high performance I would choose MSB. There's always a trade off. – code5 Sep 26 '17 at 21:57
  • Hi @code5 , I'm looking at creating stateless services as workers to consume and process messages from RabbitMQ. In your stateless service did you just override RunAsync? I'd be interested to see what way you did that. – user195166 Mar 02 '18 at 11:01
  • @user195166 - Sure. I can provide the consumer portion of the code base that I used. I used dependency inject as well using Autofac. In my code sample, I use Plato.NET to help with the complexities of dealing with RMQ. Is there an email where I can seen you a code sample. It will be a project that I need to strip down. – code5 Mar 02 '18 at 16:19
  • 2
    @code5 a stripped down Github Gist would be beneficial to the community – FunksMaName Sep 10 '18 at 08:34
  • @code5, from your comment: ```One of our engineers is currently investigating how to cluster RMQ using Service Fabric Cluster. No updates on this yet. I was hoping to use Azure Service Bus, but the business doesn't want to be tied to Azure.``` We are in the same position. We would start investigating the same. Is it possibile to share the progresses of your engineer? m (dot) monducci (at) crif (dot) com – MonDeveloper May 03 '19 at 07:29
  • @MonDeveloper - Hi Mon. We initially went with creating a SFX cluster with only 3 nodes. We used Guest services i.e. Linux docker containers. It works quite well. However, there's talks now to move our RMQ containers over to Kubernetes. The price point is more or less the same however, they are thinking of using managed Kubernetes AKS. Going the container route is the best option. The question is, which cluster technology to chose? SFX Mesh was an option as well but our engineers are worried about performance and maintenance. – code5 May 05 '19 at 20:35