IOException when uploading to Blob Storage in published app

Question

I've developed a simple app, which just have to upload files from a folder to Azure Blob Storage, I runs fine when I run in from VS, but in the published app I get this error once in a while:

ved System.IO.__Error.WinIOError(Int32 errorCode, String, maybeFullPath) ved System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) ved System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access) ved Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.UploadFromFile(String path, FileMode mode, AccessCondition accessCondition, BlobRequestOptions options, OperationsContext operationContext) ved Program.MainWindow.Process(object sender, NotifyCollectionChangedEventArgs e)

My code for uploading looks like this:

private void Process(object sender, NotifyCollectionChangedEventArgs e)
{
    if (paths.Count > 0){
        var currentPath = paths.Dequeue();
        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(UserSettings.Instance.getConnectionString());
        CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
        CloudBlobContainer blobContainer = blobClient.GetContainerReference(UserSettings.Instance.getContainer());
        CloudBlockBlob b = blobContainer.GetBlockBlobReference(System.IO.Path.GetFileName(currentPath));
        try
        {
           b.UploadFromFile(currentPath, FileMode.Open);
        }
        catch (StorageException s)
        {
            throw new System.InvalidOperationException("Could not connect to the specified storage account. Please check the configuration.");
        }
        catch (IOException exc)
        {
            throw new System.InvalidOperationException(exc.StackTrace);
        }
    }
}

It is the IOException in the catch that gets hit once in a while, any idea how to fix this?

If I go through the docs, I just get informed that the exception occour if a storage service error occoured. Any idea how to investigate the further?

https://learn.microsoft.com/en-us/java/api/com.microsoft.azure.storage.blob._cloud_blob.uploadfromfile?view=azure-java-legacy#com_microsoft_azure_storage_blob__cloud_blob_uploadFromFile_final_String_

If it's only occasional, you can just add a [retry policy](https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.storage.blob.blobrequestoptions.retrypolicy?view=azure-dotnet) — Nathan, Oct 02 '19 at 22:26
@Nathan When it happens, the error is present for all the following uploads. — Recusiwe, Oct 02 '19 at 22:35
Could you tell me what is "published app"? And could you please provide me a sample of the path you use? — Jim Xu, Oct 03 '19 at 05:31
@JimXu Published is just when I've used the Publish-function i Visual Studio. The path is just a path on the HDD. — Recusiwe, Oct 03 '19 at 19:02
@Recusiwe Is that you use Azure function to upload files to Azure storage? — Jim Xu, Oct 06 '19 at 13:48
Did you enable monitoring on your storage account to see if the request failed in Azure or if it is something local. https://learn.microsoft.com/en-us/azure/storage/common/storage-monitor-storage-account — Francois, Oct 07 '19 at 00:47
@Recusiwe Sorry. I cannot understand. Could you tell me what is "I've used the Publish-function i Visual Studio" and "The path is just a path on the HDD.(Is it the path on your local computer?)"? Besides, which service do you use? — Jim Xu, Oct 07 '19 at 00:55
The exception handling in the snippet is broken, turning a simple mishap into an undiagnosable error. There are a lot of possible IOException reasons, you need to be able to see the Message property to know what exactly happened. If you need to rethrow then be sure to use [the constructor](https://learn.microsoft.com/en-us/dotnet/api/system.invalidoperationexception.-ctor?view=netframework-4.8#System_InvalidOperationException__ctor_System_String_System_Exception_) that lets you specify the InnerException so that detail doesn't get lost. — Hans Passant, Oct 07 '19 at 08:31
I observed that the error only occours when i copy the files into the monitored folder, if I drag them, it works just fine? — Recusiwe, Oct 07 '19 at 15:03
@JimXu There is a publish-function in VS, Right-click on your project -> Publish. Yes, a path on the computer where the application is running. Service, what do you mean? I use the CloudBlockBlob to upload my files. — Recusiwe, Oct 07 '19 at 23:54
@Recusiwe I want to know which service you use to host your application. Azure app service or Azure Function? — Jim Xu, Oct 08 '19 at 01:18

score 1 · Accepted Answer · answered Oct 10 '19 at 04:07

I observed that the error only occours when i copy the files into the monitored folder, if I drag them, it works just fine?

It looks like your app is trying to read local file(s) that are still being written to the disk (incomplete) or locked by other processes. Dragging the files are somehow "atomic" operations (ie very fast) thus considerably reducing the chance of getting this error.

Try implementing the method from this answer to test if the file isn't locked prior to calling UploadFromFile(). Now depending on your code logic, you will need to implement some form of "retries" if the file is locked. Here's an example:

    protected virtual bool IsFileLocked(FileInfo file)
    {
        FileStream stream = null;

        try
        {
            stream = file.Open(FileMode.Open, FileAccess.Read, FileShare.None);
        }
        catch (IOException)
        {
            //the file is unavailable because it is:
            //still being written to
            //or being processed by another thread
            //or does not exist (has already been processed)
            return true;
        }
        finally
        {
            if (stream != null)
                stream.Close();
        }

        //file is not locked
        return false;
    }

    private void Process(object sender, NotifyCollectionChangedEventArgs e)
    {
        if (paths.Count > 0)
        {
            var currentPath = paths.Dequeue();
            CloudStorageAccount storageAccount = CloudStorageAccount.Parse(UserSettings.Instance.getConnectionString());
            CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer blobContainer = blobClient.GetContainerReference(UserSettings.Instance.getContainer());
            CloudBlockBlob b = blobContainer.GetBlockBlobReference(System.IO.Path.GetFileName(currentPath));
            try
            {
                FileInfo fi = new FileInfo(currentPath);
                while (IsFileLocked(fi))
                    Thread.Sleep(5000); // Wait 5 seconds before checking again
                b.UploadFromFile(currentPath, FileMode.Open);
            }
            catch (StorageException s)
            {
                throw new System.InvalidOperationException("Could not connect to the specified storage account. Please check the configuration.");
            }
            catch (IOException exc)
            {
                throw new System.InvalidOperationException(exc.StackTrace);
            }
        }
    }

Note that in no ways this will guarantee the file won't be locked again by another process between the IsFileLocked() call and the b.UploadFromFile() call.

I'll try your suggestion and get back. I observe that the problem occurs when I paste a count of 50 files or higher at once. — Recusiwe, Oct 12 '19 at 18:49
This worked far better, but when I paste more than 50 files to the folder, it only takes the first file and then stops. — Recusiwe, Oct 13 '19 at 19:33
Is it stopping because it is stuck in the IsFileLocked() loop or it is throwing out an exception? — Megabit, Oct 17 '19 at 00:38

score -1 · Answer 2 · answered Oct 04 '19 at 22:32

This is generally caused when there is some file missing which DotNet is referencing. I had this error and when I drill through I got to know that the navigationlevel.cs is missing. A piece of code in the snippet shows the significance of the file.

It refers to the actual source file and therefore it has to be present. When the source isn't available or elsewhere, the snippet reference of that page fails and you'll get that error (which is also why it refers to a .cs file).

If this is not the same issue, it could be due to some other file which is missing. So, try re-installing and re-compiling it.

Where do you see, that it refers to a .cs file? – Recusiwe Oct 04 '19 at 23:12 — Recusiwe, Oct 04 '19 at 23:12

IOException when uploading to Blob Storage in published app

2 Answers2