What is the best way to get Azure listBlobs in parts using Java

Question

Good day. I use com.azure:azure-storage-blob dependency with Java 8. And I have a question about the performance of the program. To get all blobs:

BlobServiceClient client = new BlobServiceClientBuilder()
        .connectionString("connectionString")
        .buildClient();
client.getBlobContainerClient("containerName")
        .listBlobs(new ListBlobsOptions().setPrefix("prefix"), null)
            .stream()...

The snippet works well, but if I need to set a time limit instead of null, then error starts to appear:

Exception in thread "main" reactor.core.Exceptions$ReactiveException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'flatMap' (and no fallback has been configured)

The bottom line is that I need to read groups of objects and send them for further processing without waiting for all. Because the waiting time of 10 objects will definitely be less than 10000. I know there is continuationToken, but calling it on the returned collection will throw the exception mentioned above. Is there any optimal solution? Thanks in advance

score 0 · Answer 1 · answered Jun 09 '22 at 05:50

To know the complete details about the error, try enabling Storage logging and check logs.

Make use of the below sample code from MsDoc:

ListBlobsOptions options = new ListBlobsOptions()
     .setPrefix("prefixToMatch")
     .setDetails(new BlobListDetails()
         .setRetrieveDeletedBlobs(true)
         .setRetrieveSnapshots(true));
 String continuationToken = "continuationToken";
 client.listBlobs(options, continuationToken, timeout).forEach(blob ->
     System.out.printf("Name: %s, Directory? %b, Deleted? %b, Snapshot ID: %s%n",
         blob.getName(),
         blob.isPrefix(),
         blob.isDeleted(),
         blob.getSnapshot()));

Otherwise, make use of Duration.ofMinutes() in your code like below:

Duration duration = Duration.ofMinutes(2);
containerClient.listBlobs(options, duration)

If still the error occurs, try changing the versions of Azure Java SDK.

Reference:

c# - Set timeout on Azure Storage operations - Stack Overflow

Azure/azure-sdk-for-java/issues/15215

What is the best way to get Azure listBlobs in parts using Java

1 Answers1