6

I'm trying to figure out the best performing approach when writing thousands of small Blobs to Azure Storage. The application scenario is the following:

  • thousands of files are being created or overwritten by a constantly running windows service installed on a Windows Azure VM
  • Writing to the Temporary Storage available to the VM, the service can reach more than 9,000 file creations per second
  • file sizes range between 1 KB and 60 KB
  • on other VMs with same sw running, other files are being created with same rate and criteria
  • given the need to build and keep updated a central repository, another service running on each VM copies the newly created files from the Temporary Storage to Azure Blobs
  • other servers should then read the Azure Blobs in their more recent version

Please note that for many constraints that I'm not listing for shortness, it's not currently possible to modify the main service to directly create Blobs instead of files on Temporary file system. ...and from what I' currently seeing it would mean a slower rate of creation, not acceptable per original requirements.

This copy operation, that I'm testing in a tight loop on 10,000 files, seems to be limited at 200 blob creations per second. I've been able to reach this result after tweaking the sample code named "Windows Azure ImportExportBlob" found here: http://code.msdn.microsoft.com/windowsazure/Windows-Azure-ImportExportB-9d30ddd5 with the async suggestions found in this answer: Using Parallel.Foreach in a small azure instance

I obtained this apparent maximum of 200 blob creations per second on an extralarge VM with 8 cores and setting the "maxConcurrentThingsToProcess" Semaphore accordingly. The network utilization during the test is max 1% of the available 10Gb shown in task manager. This means roughly 100 Mb of the 800 Mb that should be available on that VM size.

I see that the total size copied during the elapsed time gives me around 10 MB/sec.

Is there some limitation on the Azure Storage traffic you can generate or should I use a different approach when writing so many and small files ?

Community
  • 1
  • 1
MarioCannistra
  • 275
  • 3
  • 12
  • Are the containers different for Azure Storage? Also do you have any worker queues set up that you can scale out to, so they can distribute the copy/serialize/writes? Also is your VM attached storage striped in a RAID? (there are hacks to make this faster BTW, but MSFT is going to probably release VMs similar to what AWS has for specific scenarios: high I/O in your case) – Bart Czernicki Oct 31 '12 at 14:09
  • my answers to your 3 questions below: 1) Are the containers different for Azure Storage? No, my test program is writing all 10,000 files to the same container. This reminds me that I did not state that all files are written in a folder structure that is reproduced inside the container. Do you mean that a consistent improvement could be obtained writing to different containers? Do you have any url to documentation about this aspect? – MarioCannistra Oct 31 '12 at 16:21
  • 2) Also do you have any worker queues set up that you can scale out to, so they can distribute the copy/serialize/writes? My first trial has been a parallel.for loop on the 10,000 files and I got just 3 files/sec. So I applied the async pattern described in the mentioned answer [link](http://stackoverflow.com/questions/8424690/using-parallel-foreach-in-a-small-azure-instance) and thus I'm writing with a concurrency of 8. This way I don't see cpu usage above 8% and I reached 200 files / sec with just 1% of the netowrk used. Increasing concurrency on the same VM does not increase the throughput – MarioCannistra Oct 31 '12 at 16:23
  • Seems that I'm hitting a limit in the underlying blob service and/or some contentione happens when "too many" threads are writing blobs. Please, consider as well that each of the VM writing those files is already the implementation of a scale out approach but I'm finding a bottleneck in the distributed storage usage. – MarioCannistra Oct 31 '12 at 16:24
  • 3) Also is your VM attached storage striped in a RAID? I'm not using a VM attached storage for the source files to be copied. I'm using the Temporary drive, local to the VM node, that is automatically attached when you create the VM. On the extralarge VM it is a 600 GB drive. I'm satisfied with the more then 9,000 file creations / sec achieved on that. The Temporary storage has been mentioned by Russinovich in the TechEd Europe 2012 session where he also stripes 16 drives of 1TB each. – MarioCannistra Oct 31 '12 at 16:24
  • I tested that interesting approach, but is not suitable for this design. The main problem that I'm trying to solve i that all the VMs i mentioned in the fourth bullet point should be able to write to the same storage and this is not feasible with a VM attached storage (just one could write, other N could read). – MarioCannistra Oct 31 '12 at 16:25
  • Here are the scalability targets: http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx If you enable logging on the storage account that may provide better logging. – Brian Reischl Nov 01 '12 at 14:52
  • @Zander have you looked at this (this is the reason why I asked about one or more storage accounts...its a hack BTW): http://tk.azurewebsites.net/2012/08/26/hacking-azure-for-more-disk-performance/ – Bart Czernicki Nov 02 '12 at 03:19
  • How is your caching of the VHD drive set up...cache or no cache? – Bart Czernicki Nov 02 '12 at 03:20
  • @breischl: Thank you for the scalability targets. After reading that post, I started searching for more target figures possibly prepared by Microsoft and found 4 posts with important info. See the answer that I'm going to write with more details. – MarioCannistra Nov 05 '12 at 13:16
  • @Bart Czernicki: for this design I can't use a VHD even if created with that hack. I need to write from many VMs and read from many other VMs. Per Azure dcoumentation this is not possible and this is why I'm using the Blob Service. – MarioCannistra Nov 05 '12 at 13:16
  • @Zander When was your storage account created? (that link above for scalability targets posted above is GREATLY outdated). Microsoft "with little announcement" recently updated their storage to 2nd generation and it uses SSDs and improves transactions greatly. Here is an updated scalability target just released: http://blogs.msdn.com/b/windowsazure/archive/2012/11/02/windows-azure-s-flat-network-storage-and-2012-scalability-targets.aspx I would check that out. – Bart Czernicki Nov 05 '12 at 13:21
  • @Zander BTW (I know this is not what you are looking for), but this is the splitting across multiple storage accounts "hack" I was talking about for extra IOPs: http://tk.azurewebsites.net/2012/08/26/hacking-azure-for-more-disk-performance/ – Bart Czernicki Nov 05 '12 at 13:24

1 Answers1

2

@breischl Thank you for the scalability targets. After reading that post, I started searching for more target figures possibly prepared by Microsoft and found 4 posts (too many for my "reputation" to be posted here, the other 3 are part 2, 3 and 4 of the same series):

http://blogs.microsoft.co.il/blogs/applisec/archive/2012/01/04/windows-azure-benchmarks-part-1-blobs-read-throughput.aspx

the first post contains an important hint: "You may have to increase the ServicePointManager.DefaultConnectionLimit for multiple threads to establish more than 2 concurrent connections with the storage."

I've set this to 300 , rerun the test and seen an important increase in the MB/s. As I previously wrote, I was thinking to be hitting a limit in the underlying blob service when "too many" threads are writing blobs. This is the confirmation of my worries. Thus, I removed all the changes made to the code to work with a semaphore and replaced it again with a parallel.for to start as many blob upload operations as possible. The result has been awesome: 61 MB/s writing blobs and 65 MB/s reading.

The scalability target is 60 MB/s and I'm finally happy with the result.
Thank you all again for your answers.

MarioCannistra
  • 275
  • 3
  • 12