Difference between Azure Data Lake Storage x Azure Blob Storage and Azure File Storage

Question

I have a question about the use cases of the different Azure storage services:

• Azure Data Lake Storage.
• Azure Blob Storage.
• Azure File Storage.

what is the difference between these services? and when to use them since they all provide the same functionality (storage) on Azure's cloud platform.

Answer 1

You can take a look at this article: https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-comparison-with-blob-storage

I'd say main differences between Data Lake and Azure Storage Blob is scale and permissions model.

It really makes no sense to paste the whole article here. But you might want to look at Data Lake v2, which (as MS claims) is a mesh or Data Lake v1 and Azure Storage Blob: https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-introduction

As for the Azure File storage its just an SMB share over HTTPS (and it not really fast due to being 1 "stream" only).

Answer 2

Major storage types are block storage, object storage and file storage.

• Block Storage

  • Examples are hard disk drives and SSD's where the operating systems, databases can run. The OS can access the disk block by block with this storage.
  • Equivalent cloud managed offerings are Azure disks, AWS EBS.

• Azure Blob Storage

  • It's an object storage with no hierarchy. Comparable to AWS s3.
  • An object is a bundle of file along with a identifier and metadata.
  • There is no concept of folders.
  • Usually, its accessed using HTTPS and costs very low compared to other types of storage.
  • Best used to store all types of structured, semi structured and unstructured binary data for long duration.

• Azure File Storage

  • It's a file storage that can interface using NFS, SMB. Similar to windows shares available in office/workstation environments.
  • Here there is concept of files and folders, but these are not performant.
  • Object storage is now more widely used instead of file storage.

Azure Data Lake Storage Gen2 - Azure's managed distributed fileystem. It's a distributed file system for Hadoop compute and storage with compatible semantics to HDFS.

Blob storage vs ADLS described here - https://stackoverflow.com/a/76038745/6563567

Although they might seem quite the same but analytical workload like databricks can in some cases work far more efficiently with ADLS. Apart from this, its 100% HDFS compatible and provide Linux like ACLs on files and folders.

real benefit of ADLS is that it's very efficient to move files, rename files, move folders, rename folders, etc. ADLS's efficient directory manipulation is beneficial for analytics workloads like databricks/spark which best operates on file systems.