How do I iterate CloudBlobDirectory to copy the data?

Question

I have written below code to iterate through the Gen2 storage blob

 CloudStorageAccount sourceAccount = CloudStorageAccount.Parse(sourceConnection);
 CloudStorageAccount destAccount = CloudStorageAccount.Parse(destConnection);
 CloudBlobClient sourceClient = sourceAccount.CreateCloudBlobClient();
 CloudBlobClient destClient = destAccount.CreateCloudBlobClient();
 CloudBlobContainer sourceBlobContainer = sourceClient.GetContainerReference(sourceContainer);

 // Find all blobs that haven't changed since the specified date and time
 IEnumerable<ICloudBlob> sourceBlobRefs = FindMatchingBlobsAsync(sourceBlobContainer, transferBlobsNotModifiedSince).Result;



    private static async Task<IEnumerable<ICloudBlob>> FindMatchingBlobsAsync(CloudBlobContainer blobContainer, DateTime transferBlobsNotModifiedSince)
    {
        List<ICloudBlob> blobList = new List<ICloudBlob>();
        BlobContinuationToken token = null;

        // Iterate through the blobs in the source container
        do
        {
            BlobResultSegment segment = await blobContainer.ListBlobsSegmentedAsync(prefix: "", currentToken: token);
            foreach (CloudBlobDirectory VARIABLE in segment.Results)
            {
                BlobResultSegment segment2 = await VARIABLE.ListBlobsSegmentedAsync(currentToken: token);
                foreach (CloudBlobDirectory VARIABLE2 in segment2.Results)//Bad coding
                {
                    //how do I get children count ?
                }
            }

        }while (token != null);
     }

This will iterate only 2 levels but not dynamically till the inner levels. I have blob in below hierarchy

  --Container
    --FolderA
      --FolderAA
        --FolderAA1
          --File1.txt
          --File2.txt              
        --FolderAA2
          --File1.txt
          --File2.txt
        --FolderAA3
     --FolderAB
       --File8.txt
     --FolderAC
       --File9.txt

This hierarchy is dynamic

How do I loop and copy the blob content.

Note: I do not want to use CLI commands to copy. Because I won't have any control once copy started.

Update

Found some samples here: https://csharp.hotexamples.com/examples/Microsoft.WindowsAzure.Storage.Blob/CloudBlobContainer/ListBlobsSegmented/php-cloudblobcontainer-listblobssegmented-method-examples.html

It's really quite old :). Let me provide a solution using SDK version 9.3.3. More or less they should be compatible. — Gaurav Mantri, May 14 '20 at 09:41
Actually I found an answer I wrote long time back. Closing this as a duplicate of that. Do take a look at that and ask to reopen if that does not solve your problem. HTH. — Gaurav Mantri, May 14 '20 at 09:53
it says "ListBlobsSegmented" method not found. I have latest WindowsAzure.Storage 9.3.3 — kudlatiger, May 14 '20 at 11:42
Could you provide a working method which iterates through all subfolders, so that I can improvise it to meet my requirement of copying data to target storage account? — kudlatiger, May 14 '20 at 12:00
Added sample code that would list all blobs in a blob container. — Gaurav Mantri, May 14 '20 at 12:06

score 1 · Accepted Answer · answered May 14 '20 at 12:05

Please see the sample code below:

class Program
{
    static void Main(string[] args)
    {
        var storageAccount = CloudStorageAccount.Parse("UseDevelopmentStorage=true");
        var client = storageAccount.CreateCloudBlobClient();
        var container = client.GetContainerReference("test");
        var blobs = FindMatchingBlobsAsync(container).GetAwaiter().GetResult();
        foreach (var blob in blobs)
        {
            Console.WriteLine(blob.Name);
        }
        Console.WriteLine("-------------------------------------");
        Console.WriteLine("List of all blobs fetched. Press any key to terminate the application.");
        Console.ReadKey();
    }

    private static async Task<IEnumerable<ICloudBlob>> FindMatchingBlobsAsync(CloudBlobContainer blobContainer)
    {
        List<ICloudBlob> blobList = new List<ICloudBlob>();
        BlobContinuationToken token = null;

        // Iterate through the blobs in the source container
        do
        {
            BlobResultSegment segment = await blobContainer.ListBlobsSegmentedAsync(prefix: "", useFlatBlobListing: true, BlobListingDetails.None, 5000, token, new BlobRequestOptions(), new OperationContext());
            token = segment.ContinuationToken;
            foreach(var item in segment.Results)
            {
                blobList.Add((ICloudBlob)item);
            }
        } while (token != null);
        return blobList;
    }
}

How do I iterate CloudBlobDirectory to copy the data?

1 Answers1

Linked