How to handle "The specified resource does not exist" exception while using Entity group transactions to purge WADLogs table

Question

We have a requirement to purge the Azure WADLogs table on a periodic basis. We are achieving this by using Entity group transactions to delete the records older than 15 days. The logic is like this.

bool recordDoesNotExistExceptionOccured = false;
CloudTable wadLogsTable = tableClient.GetTableReference(WADLogsTableName);

partitionKey = "0" + DateTime.UtcNow.AddDays(noOfDays).Ticks;
TableQuery<WadLogsEntity> buildQuery = new TableQuery<WadLogsEntity>().Where(
                            TableQuery.GenerateFilterCondition("PartitionKey",                
                            QueryComparisons.LessThanOrEqual, partitionKey));

while (!recordDoesNotExistExceptionOccured)
{
    IEnumerable<WadLogsEntity> result = wadLogsTable.ExecuteQuery(buildQuery).Take(1000);

    //// Batch entity delete.
    if (result != null && result.Count() > 0)
    {
         Dictionary<string, TableBatchOperation> batches = new Dictionary<string, TableBatchOperation>();
         foreach (var entity in result)
         {
             TableOperation tableOperation = TableOperation.Delete(entity);

             if (!batches.ContainsKey(entity.PartitionKey))
             {
                 batches.Add(entity.PartitionKey, new TableBatchOperation());
             }

             // A Batch Operation allows a maximum 100 entities in the batch which must share the same PartitionKey.                        
             if (batches[entity.PartitionKey].Count < 100)
             {
                 batches[entity.PartitionKey].Add(tableOperation);
             }
        }

        // Execute batches.
        foreach (var batch in batches.Values)
        {
            try
            {
                await wadLogsTable.ExecuteBatchAsync(batch);
            }
            catch (Exception exception)
            {
                // Log exception here.

                // Set flag.
                if (exception.Message.Contains(ResourceDoesNotExist))
                {
                    recordDoesNotExistExceptionOccured = true;
                }
                break;
            }
        }
    }
    else
    {                            
        break;
    }
}

My questions are:

1. Is this an efficient way to purge the WADLogs table? If not, what can make this better?
2. Is this the correct way to handle the "Specified resource does not exist exception"? If not, how can I make this better?
3. Would this logic fail in any particular case?
4. How would this approach change if this code is in a worker which has multiple instances deployed?

I have come up with this code by referencing the solution given here.

Answer 1

1. Batches are in general the best way to purge part of a table if you can't just delete the whole thing. However, there are a couple of problems with details in the code above that I'll outline below.

2. I'm not sure why you're getting this error for a table entity which was just returned in the query unless you're deleting things from this table on a different thread. However, with the scenario you've given you should not halt the whole process when you get an error. Say you had 1000 results to delete. The 2nd had been deleted by another thread and returned an error when you tried to delete it again - now you're missing deleting all of the other entities, including the other ones in that batch since batches are atomic! To fix this scenario, I'd suggest removing the failing entity from the batch (the error message will include its index), retrying that batch, and then continuing to delete the rest of the batches.

3. See 2. Additionally, if you have more than 100 entities with the same partition key you are not deleting the remainder right away but are instead ignoring them and deleting them in the next loop iteration. If there's many entries with the same partition key, this could result in a lot of extra queries so it might be more efficient to add them to a new batch.

4. Once you get more workers going, the issue in 2 will get significantly worse with the current design since you always query the first 1000. Now you'll have multiple threads downloading the same results and trying to delete the same things. With the current design this will cause a lot of errors since workers will be deleting the same results. The easiest solution I see would be finding a better way to divide the work that doesn't cause overlap. One idea might be to have each worker take a different day to delete which would also have the added benefit of optimizing the number of batches which have to be sent. Of course, there are many other valid strategies.