1

The following code is meant to copy files asynchronously but it causes deadlock in my app. It uses a task combinator helper method called 'Interleaved(..)' found here to return tasks in the order they complete.

public static async Task<List<StorageFile>> CopyFiles_CAUSES_DEADLOCK(IEnumerable<StorageFile> sourceFiles, IProgress<int> progress, StorageFolder destinationFolder)
{
    List<StorageFile> copiedFiles = new List<StorageFile>();
    List<Task<StorageFile>> copyTasks = new List<Task<StorageFile>>();

    foreach (var file in sourceFiles)
    {
        // Create the copy tasks and add to list
        var copyTask = file.CopyAsync(destinationFolder, Guid.NewGuid().ToString()).AsTask();
        copyTasks.Add(copyTask);
    }

    // Serve up each task as it completes
    foreach (var bucket in Interleaved(copyTasks))
    {
        var copyTask = await bucket;
        var copiedFile = await copyTask;
        copiedFiles.Add(copiedFile);
        progress.Report((int)((double)copiedFiles.Count / sourceFiles.Count() * 100.0));
    }

    return copiedFiles;
}

I originally created a simpler 'CopyFiles(...)' which processes the tasks in the order they were supplied (as opposed to completed) and this works fine, but I can't figure out why this one deadlocks frequently. Particularly, when there are many files to process.

Here is the simpler 'CopyFiles' code that works:

public static async Task<List<StorageFile>> CopyFiles_RUNS_OK(IEnumerable<StorageFile> sourceFiles, IProgress<int> progress, StorageFolder destinationFolder)
{
    List<StorageFile> copiedFiles = new List<StorageFile>();
    int sourceFilesCount = sourceFiles.Count();

    List<Task<StorageFile>> tasks = new List<Task<StorageFile>>();
    foreach (var file in sourceFiles)
    {
        // Create the copy tasks and add to list
        var copiedFile = await file.CopyAsync(destinationFolder, Guid.NewGuid().ToString()).AsTask();
        copiedFiles.Add(copiedFile);
        progress.Report((int)((double)copiedFiles.Count / sourceFilesCount *100.0));
    }

    return copiedFiles;
}

EDIT:

In an attempt to find out what's going on I've changed the implementation of CopyFiles(...) to use TPL Dataflow. I am aware that this code will return items in the order they were supplied, which is not what I want, but it removes the Interleaved dependency as a start. Anyway, despite this the app still hangs. It seems as if it's not returning from the file.CopyAsync(..) call. There is of course the possibility I'm just doing something wrong here.

public static async Task<List<StorageFile>> CopyFiles_CAUSES_HANGING_ALSO(IEnumerable<StorageFile> sourceFiles, IProgress<int> progress, StorageFolder destinationFolder)
{
    int sourceFilesCount = sourceFiles.Count();
    List<StorageFile> copiedFiles = new List<StorageFile>();

    // Store for input files.
    BufferBlock<StorageFile> inputFiles = new BufferBlock<StorageFile>();
    // 
    Func<StorageFile, Task<StorageFile>> copyFunc = sf => sf.CopyAsync(destinationFolder, Guid.NewGuid().ToString()).AsTask();
    TransformBlock<StorageFile, Task<StorageFile>> copyFilesBlock = new TransformBlock<StorageFile, Task<StorageFile>>(copyFunc);

    inputFiles.LinkTo(copyFilesBlock, new DataflowLinkOptions() { PropagateCompletion = true });

    foreach (var file in sourceFiles)
    {
        inputFiles.Post(file);
    }
    inputFiles.Complete();

    while (await copyFilesBlock.OutputAvailableAsync())
    {
        Task<StorageFile> file = await copyFilesBlock.ReceiveAsync();
        copiedFiles.Add(await file);
        progress.Report((int)((double)copiedFiles.Count / sourceFilesCount * 100.0));
    }
    copyFilesBlock.Completion.Wait();

    return copiedFiles;
}

Many thanks in advance for any help.

sokkyoku
  • 2,161
  • 1
  • 20
  • 22
Cleve
  • 1,273
  • 1
  • 13
  • 26
  • Can you show the code where you call this method? – Dennis Schröer Sep 30 '16 at 12:35
  • What kind of application are you running this task in? – Lasse V. Karlsen Sep 30 '16 at 12:35
  • Hi @Zure the calling code is this private async Task DoCopyingAsync(IProgress progress, IEnumerable files) { await Utilities.CopyFiles_CAUSES_DEADLOCK(files, progress, await Utilities.GetSaveFolderAsync()); } – Cleve Sep 30 '16 at 12:41
  • Hi @LasseV.Karlsen , it's a UWP app – Cleve Sep 30 '16 at 12:42
  • Just a guess and can't explain why, but try this instead of the await: Task.Run(() => Utilities.CopyFiles_CAUSES_DEADLOCK(files, progress, await Utilities.GetSaveFolderAsync())); – Dennis Schröer Sep 30 '16 at 12:49
  • @Zure, thanks for the suggestion, but sadly the app still hangs with this change. – Cleve Sep 30 '16 at 12:54
  • If you remove progress.Report(), does it still hang? – sellotape Sep 30 '16 at 13:04
  • @sellotape, yes it still hangs after removing progress.Report(). – Cleve Sep 30 '16 at 13:19
  • Did you try out the `await copyTask.ConfigureAwait(false);`? And the same for the `await copyTask.ConfigureAwait(false);`? – VMAtm Sep 30 '16 at 14:57
  • Hi @VMAtm, just tried var copyTask = await bucket.ConfigureAwait(false); var copiedFile = await copyTask.ConfigureAwait(false); within the second foreach but it still hangs. – Cleve Sep 30 '16 at 15:39
  • Can you annotate the code (e.g. with Debug.WriteLine()s/tracing/logging) to find out exactly which statement is blocking? Or when it blocks, Debug > Break All, and looks at the stacks for each thread. – sellotape Sep 30 '16 at 17:36
  • Hi @sellotape, I've done Debug > Break All, and the Task Window shows one Task awaiting which relates to the Top Level event handler that sets the copy off. There is one Active task which looks to relate to the code method DoCopyingAsync(...) that I list above. The rest of the tasks have a status of Scheduled, and waiting to run. I'm not sure I know enough to do anything with this information. Does this help? – Cleve Sep 30 '16 at 18:33
  • ok, I've sprinkled the code with Debug,WriteLines(..). When the app hangs it seems to get to immediately before the first foreach () in the CopyFiles_CAUSES_DEADLOCK() method, but it doesn't come out of the loop. Like I said originally, this doesn't happen every time the app is run, but frequently, especially if I choose lots of files to copy. – Cleve Sep 30 '16 at 20:21
  • Right, I know this is going to sound weird but in an attempt to narrow down the problem I put more Debug.WriteLines within the first foreach loop of CopyFiles_CAUSES_DEADLOCK. Upon doing this it wouldn't hang??? Also, it wouldn't show progress on the progress bar but the copying appeared to get done. I tried to make it hang way more times than it takes without these WriteLines in place. When I remove them it hangs again. Is it a timing issue? By that I mean that while they are being executed this is allowing something else to finish? Or is the act of performing the WriteLines changing somethg – Cleve Oct 01 '16 at 11:08
  • What is the `IEnumerable` you're using as input? If it hangs before even the first foreach loop I can only think of the `GetEnumerator` failing. Put a `var tmpArr = sourceFiles.ToArray()` at the top and use that instead? I was also peeved by Report calling `Count()`, this reiterates the entire enum every iteration. Store that non-changing number. Also, the simpler `CopyFiles` isn't just removing `Interleaved` I think, as it would still hang at the first foreach. Post that please. – Wolfzoon Oct 01 '16 at 20:13
  • Hi @Wolfzoon, the IEnumerable is a List that I'm using as input. I've added the simpler CopyFiles version above as an edit to the original post. – Cleve Oct 02 '16 at 09:14
  • @Wolfzoon, Tried `var tmpArr = sourceFiles.ToArray()` suggestion but app still hangs. – Cleve Oct 02 '16 at 10:03
  • Deadlock is not very plausible here, more likely is that you just don't have enough patience to wait for the program to complete. It will be **excessively** slow. You need to read [this answer](http://stackoverflow.com/a/25950320/17034). Shows you how to debug deadlock. And talks about the real reason why this works so poorly, you have too many processor cores and not enough disk drives. Only the second snippet is correct. – Hans Passant Oct 02 '16 at 10:58
  • Hi @HansPassant, have been more than patient waiting for the program to complete. Tried many times, waiting 10 minutes or more before killing it, for something that should take seconds. I can get it to regularly hang while trying to copy 90 files, each 30kb in size on average. All I want to do is copy a selection of files asynchronously in the order they complete showing progress on the way. – Cleve Oct 02 '16 at 14:26
  • Why do you do `copyFilesBlock.Completion.Wait();` and not `await copyFilesBlock.Completion;`? Also, your last `while` block should really be replaced with a `ActionBlock` and linked to your transform block, then you can await the completion of that action block. – Scott Chamberlain Oct 03 '16 at 18:09
  • Hi @ScottChamberlain, thanks for the comments. My excuse is that I don't know TPL Dataflow very well! I quickly cobbled together the edit as an alternative, but your suggestions are great. Thanks. – Cleve Oct 03 '16 at 19:02

0 Answers0