0

We have an application which is quite scalable as it is. Basically you have one or more stateless nodes that all do some independent work of files that are read and written to shared NFS share. This NFS can be bottleneck but with local deployment customers just buys big enough box to have sufficient performance. Now we are moving this to Azure and I would like to have a better more "cloudy" way of sharing data :) and running some Linux NFS server isn't ideal scenario if we need to manage them. Is the Azure Blob storage the right tool for this job (https://azure.microsoft.com/en-us/services/storage/blobs/)?

  • we need good scalability. e.g. up to 10k files writen in a minute

  • files are quite small, less than 50KB per file on average

  • files created and read, not changed

  • files are short lived, we purge them every day

    I am looking for more practical experience with this kind of storage and how good it really is.

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
Primoz
  • 4,079
  • 17
  • 56
  • 67

1 Answers1

1

There are two possible solutions to your request, either using Azure Storage Blobs (Recommended for your scenario) or Azure Files.

Azure Blobs has the following scaling targets: enter image description here

  • It doesn't support the fact of attaching it a server, such as a network share.
  • Blobs do not support a hierarchy file structure besides having containers (Virtual folders can be accessed, but the con is you can't delete a container if it contains blobs- for the point about purging- but there are methods to do purging using your own code.)

Azure Files: enter image description here

Links recommended: Comparison between Azure Files and Blobs:https://learn.microsoft.com/en-us/azure/storage/common/storage-decide-blobs-files-disks

Informative SO post here

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
Adam Smith - Microsoft Azure
  • 2,495
  • 2
  • 12
  • 24
  • I was already looking into Azure Files which is just a smb3 file share, managed by MS. It offers simplicity and works fine even from Linux. It's just that when you have more than few nodes they can overwhelm it and there is no way that I can see, to scale it. – Primoz Aug 06 '18 at 18:08
  • per the documentation for Azure Scale targets, it has a Max IOPS per share of 1000 IOPS, and 20,000 requests per second for files of any valid size. So it could be scalable, unless if you have to have larger requirements. more info about scalability can be found here: https://learn.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets – Adam Smith - Microsoft Azure Aug 06 '18 at 19:02
  • I know that, I have read the documentation and I have tried Azure Files but they didn't live up to expectations. In theory I could have tens or hundreds of shares to spread the load, but then this is no longer single namespace. – Primoz Aug 06 '18 at 20:22
  • So I am looking out if anyone has practical experience with blob storage.Since bandwidth and number of requests are per blob then with each file being a separate blob you could push really big numbers, correct ? – Primoz Aug 06 '18 at 20:28
  • @Primoz Blob would be the best option for you, as it's very scalable, and very persistent performance wise. I only added/metnioned Azure File mainly if you add to use an NFS similar system. Blobs would be the way to go. – Adam Smith - Microsoft Azure Aug 06 '18 at 21:30