How upload files to azure in background with Delphi and OmniThread?

Question

I have tried to upload +100 files to azure with Delphi. However, the calls block the main thread, so I want to do this with a async call or with a background thread.

This is what I do now (like explained here):

procedure TCloudManager.UploadTask(const input: TOmniValue;
  var output: TOmniValue);
var
  FileTask:TFileTask;
begin
  FileTask := input.AsRecord<TFileTask>;

  Upload(FileTask.BaseFolder, FileTask.LocalFile, FileTask.CloudFile);
end;

function TCloudManager.MassiveUpload(const BaseFolder: String;
  Files: TDictionary<String, String>): TStringList;
var
  pipeline: IOmniPipeline;
  FileInfo : TPair<String,String>;
  FileTask:TFileTask;
begin
  // set up pipeline
  pipeline := Parallel.Pipeline
    .Stage(UploadTask)
    .NumTasks(Environment.Process.Affinity.Count * 2)
    .Run;
  // insert URLs to be retrieved
  for FileInfo in Files do
  begin
    FileTask.LocalFile := FileInfo.Key;
    FileTask.CloudFile := FileInfo.Value;
    FileTask.BaseFolder := BaseFolder;

    pipeline.Input.Add(TOmniValue.FromRecord(FileTask));
  end;//for

  pipeline.Input.CompleteAdding;

  // wait for pipeline to complete
  pipeline.WaitFor(INFINITE);
end;

However this block too (why? I don't understand).

Answer 1

This blocks because you are calling WaitFor which waits for all pipeline stages to finish their work. During that wait, the GUI is blocked.

A proper way to do it is

1. Store interface returned from Parallel.Pipeline in a global storage (for example in a TCloudManager field).
2. Schedule work to the pipeline.
3. Don't WaitFor end but assign OnStop handler and do whatever cleanup you need here (don't forget to nil out the global storage holding the pipeline interface).

To do step 3 you'll need fresh OmniThreadLibrary from the SVN because I just added this functionality :)

procedure TCloudManager.MassiveUpload(const BaseFolder: String;
  Files: TDictionary<String, String>);
var
  FileInfo : TPair<String,String>;
  FileTask:TFileTask;
begin
  // set up pipeline
  FPipeline := Parallel.Pipeline
    .Stage(UploadTask)
      .NumTasks(Environment.Process.Affinity.Count * 2)
    .OnStop(
      procedure begin
        ShowMessage('All done');
        FPipeline := nil;
      end)
    .Run;
//   insert URLs to be retrieved
  for FileInfo in Files do
  begin
    FileTask.LocalFile := FileInfo.Key;
    FileTask.CloudFile := FileInfo.Value;
    FileTask.BaseFolder := BaseFolder;

    FPipeline.Input.Add(TOmniValue.FromRecord(FileTask));
  end;//for
  FPipeline.Input.CompleteAdding;
end;

Answer 2

Delphi has a .NET variant, right? Were you aware there's a .NET Managed API for the Azure Storage Service?

The CloudBlockBlob class has an async variant for upload/download, etc.

http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storageclient.cloudblockblob_methods.aspx