0

I have a scenario I think could be a fit for Service Fabric. I'm working on premises.

I read messages from a queue. Each message contains details of a file that needs to be downloaded. Files are downloaded and added to a zip archive.

There are 40,000 messages so that's 40,000 downloads and files. I will add these to 40 zip archives, so that's 1000 files per archive.

Would Service Fabric be a good fit for this workload?

I plan to create a service that takes a message off the queue, downloads the file and saves it somewhere. I'd then scale that service to have 100 instances.

Once all the files have been downloaded I'd kick of a different process to add the files to a zip archive. Bonus if you can tell me a way to incorporate adding the files to a zip archive as part of the service

Cœur
  • 37,241
  • 25
  • 195
  • 267
user195166
  • 417
  • 5
  • 16
  • Did you consider using Azure Functions? They scale on demand, you only pay for way you need and you do not need to set up and manage a Service Fabric Cluster. How big are the files we are talking about? Imho using Service Fabric sounds like overkill for your scenario. And having 100 instances does not tell us much, what would be the size of your cluster? – Peter Bons May 04 '18 at 05:50
  • Thanks for your response. I'm working on premises so Azure Functions is not possible. Files range from 100KB to approx 800KB. As regards number of services and size of cluster - I'd like to complete the work within 30 minutes so whatever number of instances/size of cluster would achive that will be used. – user195166 May 04 '18 at 08:30

2 Answers2

1

Would Service Fabric be a good fit for this workload?

If the usage will be just for downloading and compressing the files, I think it will be overkill to setup a cluster and manage it to sustain an application that is very simple. I think you could find many alternatives where you don't have to setup an environment just to keep your application running and processing a message from the queue.

I'd then scale that service to have 100 instances.

The number of instances does not mean the download will be faster, you have also to consider the the network limit, otherwise you will just end up with servers with idle CPU and Memory, where the network might be the bottleneck.

I plan to create a service that takes a message off the queue, downloads the file and saves it somewhere.

If you want to stick with service fabric and the queue approach, I would suggest this answer I gave a while ago: Simulate 10,000 Azure IoT Hub Device connections from Azure Service Fabric cluster

The information is not exactly what you plan to do, but might give directions based on the scale you have and how to process a large number of messages from a queue(IoT Hub messaging is very similar to service bus).

For the other questions I would suggest create them on a separate topic.

Diego Mendes
  • 10,631
  • 2
  • 32
  • 36
0

Agree to Diego, using service fabric for this would be an overkill, and it wont be the best utilization of the resources, moreover this seems to be more of a disk extensive problem where you would need lots of storage to download those file and than compressing it in a zip. An idea can be to use azure functions as the computation seems minimal to me in your case. Download the file in azure file share and than upload to whatever storage you want (BLOB for example). This way you wont be using much of resources and you can scale the function and the azure file share according to your needs.

Kayani
  • 942
  • 7
  • 23