Downloading multiple files in batches in iOS

Question

I have an app that right now needs to download hundreds of small PDF's based on the users selection. The problem I am running into is that it is taking a significant amount of time because every time it has to open a new connection. I know that I could use GCD to do an async download, but how would I go about doing this in batches of like 10 files or so. Is there a framework that already does this, or is this something I will have to build my self?

Answer 1

This answer is now obsolete. Now that NSURLConnection is deprecated and NSURLSession is now available, that offers better mechanisms for downloading a series of files, avoiding much of the complexity of the solution contemplated here. See my other answer which discusses NSURLSession.

I'll keep this answer below, for historical purposes.

---

I'm sure there are lots of wonderful solutions for this, but I wrote a little downloader manager to handle this scenario, where you want to download a bunch of files. Just add the individual downloads to the download manager, and as one finishes, it will kick off the next queued one. You can specify how many you want it to do concurrently (which I default to four), so therefore there's no batching needed. If nothing else, this might provoke some ideas of how you might do this in your own implementation.

Note, this offers two advantages:

1. If your files are large, this never holds the entire file in memory, but rather streams it to persistent storage as it's being downloaded. This significantly reduces the memory footprint of the download process.

2. As the files are being downloaded, there are delegate protocols to inform you or the progress of the download.

I've attempted to describe the classes involved and proper operation on the main page at the Download Manager github page.

---

I should say, though, that this was designed to solve a particular problem, where I wanted to track the progress of downloads of large files as they're being downloaded and where I didn't want to ever hold the entire in memory at one time (e.g., if you're downloading a 100mb file, do you really want to hold that in RAM while downloading?).

While my solution solves those problem, if you don't need that, there are far simpler solutions using operation queues. In fact you even hint at this possibility:

I know that I could use GCD to do an async download, but how would I go about doing this in batches of like 10 files or so. ...

I have to say that doing an async download strikes me as the right solution, rather than trying to mitigate the download performance problem by downloading in batches.

You talk about using GCD queues. Personally, I'd just create an operation queue so that I could specify how many concurrent operations I wanted, and download the individual files using NSData method dataWithContentsOfURL followed by writeToFile:atomically:, making each download it's own operation.

So, for example, assuming you had an array of URLs of files to download it might be:

NSOperationQueue *queue = [[NSOperationQueue alloc] init];
queue.maxConcurrentOperationCount = 4;

for (NSURL* url in urlArray)
{
    [queue addOperationWithBlock:^{
        NSData *data = [NSData dataWithContentsOfURL:url];
        NSString *filename = [documentsPath stringByAppendingString:[url lastPathComponent]];
        [data writeToFile:filename atomically:YES];
    }];
}

Nice and simple. And by setting queue.maxConcurrentOperationCount you enjoy concurrency, while not crushing your app (or the server) with too many concurrent requests.

And if you need to be notified when the operations are done, you could do something like:

NSOperationQueue *queue = [[NSOperationQueue alloc] init];
queue.maxConcurrentOperationCount = 4;

NSBlockOperation *completionOperation = [NSBlockOperation blockOperationWithBlock:^{
    [[NSOperationQueue mainQueue] addOperationWithBlock:^{
        [self methodToCallOnCompletion];
    }];
}];

for (NSURL* url in urlArray)
{
    NSBlockOperation *operation = [NSBlockOperation blockOperationWithBlock:^{
        NSData *data = [NSData dataWithContentsOfURL:url];
        NSString *filename = [documentsPath stringByAppendingString:[url lastPathComponent]];
        [data writeToFile:filename atomically:YES];
    }];
    [completionOperation addDependency:operation];
}

[queue addOperations:completionOperation.dependencies waitUntilFinished:NO];
[queue addOperation:completionOperation];

This will do the same thing, except it will call methodToCallOnCompletion on the main queue when all the downloads are done.

Answer 2

By the way, iOS 7 (and Mac OS 10.9) offer URLSession and URLSessionDownloadTask, which handles this quite gracefully. If you just want to download a bunch of files, you can do something like:

NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];
NSURLSession              *session       = [NSURLSession sessionWithConfiguration:configuration];

NSString      *documentsPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];
NSFileManager *fileManager   = [NSFileManager defaultManager];

for (NSString *filename in self.filenames) {
    NSURL *url = [baseURL URLByAppendingPathComponent:filename];
    NSURLSessionTask *downloadTask = [session downloadTaskWithURL:url completionHandler:^(NSURL *location, NSURLResponse *response, NSError *error) {
        NSString *finalPath = [documentsPath stringByAppendingPathComponent:filename];

        BOOL success;
        NSError *fileManagerError;
        if ([fileManager fileExistsAtPath:finalPath]) {
            success = [fileManager removeItemAtPath:finalPath error:&fileManagerError];
            NSAssert(success, @"removeItemAtPath error: %@", fileManagerError);
        }

        success = [fileManager moveItemAtURL:location toURL:[NSURL fileURLWithPath:finalPath] error:&fileManagerError];
        NSAssert(success, @"moveItemAtURL error: %@", fileManagerError);

        NSLog(@"finished %@", filename);
    }];
    [downloadTask resume];
}

Perhaps, given that your downloads take a "significant amount of time", you might want them to continue downloading even after the app has gone into the background. If so, you can use backgroundSessionConfiguration rather than defaultSessionConfiguration (though you have to implement the NSURLSessionDownloadDelegate methods, rather than using the completionHandler block). These background sessions are slower, but then again, they happen even if the user has left your app. Thus:

- (void)startBackgroundDownloadsForBaseURL:(NSURL *)baseURL {
    NSURLSession *session = [self backgroundSession];

    for (NSString *filename in self.filenames) {
        NSURL *url = [baseURL URLByAppendingPathComponent:filename];
        NSURLSessionTask *downloadTask = [session downloadTaskWithURL:url];
        [downloadTask resume];
    }
}

- (NSURLSession *)backgroundSession {
    static NSURLSession *session = nil;

    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration backgroundSessionConfiguration:kBackgroundId];
        session = [NSURLSession sessionWithConfiguration:configuration delegate:self delegateQueue:[NSOperationQueue mainQueue]];
    });

    return session;
}

#pragma mark - NSURLSessionDownloadDelegate

- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didFinishDownloadingToURL:(NSURL *)location {
    NSString *documentsPath    = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];
    NSString *finalPath        = [documentsPath stringByAppendingPathComponent:[[[downloadTask originalRequest] URL] lastPathComponent]];
    NSFileManager *fileManager = [NSFileManager defaultManager];

    BOOL success;
    NSError *error;
    if ([fileManager fileExistsAtPath:finalPath]) {
        success = [fileManager removeItemAtPath:finalPath error:&error];
        NSAssert(success, @"removeItemAtPath error: %@", error);
    }

    success = [fileManager moveItemAtURL:location toURL:[NSURL fileURLWithPath:finalPath] error:&error];
    NSAssert(success, @"moveItemAtURL error: %@", error);
}

- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didResumeAtOffset:(int64_t)fileOffset expectedTotalBytes:(int64_t)expectedTotalBytes {
    // Update your UI if you want to
}

- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didWriteData:(int64_t)bytesWritten totalBytesWritten:(int64_t)totalBytesWritten totalBytesExpectedToWrite:(int64_t)totalBytesExpectedToWrite {
    // Update your UI if you want to
}

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didCompleteWithError:(NSError *)error {
    if (error)
        NSLog(@"%s: %@", __FUNCTION__, error);
}

#pragma mark - NSURLSessionDelegate

- (void)URLSession:(NSURLSession *)session didBecomeInvalidWithError:(NSError *)error {
    NSLog(@"%s: %@", __FUNCTION__, error);
}

- (void)URLSessionDidFinishEventsForBackgroundURLSession:(NSURLSession *)session {
    AppDelegate *appDelegate = (id)[[UIApplication sharedApplication] delegate];
    if (appDelegate.backgroundSessionCompletionHandler) {
        dispatch_async(dispatch_get_main_queue(), ^{
            appDelegate.backgroundSessionCompletionHandler();
            appDelegate.backgroundSessionCompletionHandler = nil;
        });
    }
}

By the way, this assumes your app delegate has a backgroundSessionCompletionHandler property:

@property (copy) void (^backgroundSessionCompletionHandler)();

And that the app delegate will set that property if the app was awaken to handle URLSession events:

- (void)application:(UIApplication *)application handleEventsForBackgroundURLSession:(NSString *)identifier completionHandler:(void (^)())completionHandler {
    self.backgroundSessionCompletionHandler = completionHandler;
}

For an Apple demonstration of the background NSURLSession see the Simple Background Transfer sample.

Answer 3

If all of the PDFs are coming from a server you control then one option would be to have a single request pass a list of files you want (as query parameters on the URL). Then your server could zip up the requested files into a single file.

This would cut down on the number of individual network requests you need to make. Of course you need to update your server to handle such a request and your app needs to unzip the returned file. But this is much more efficient than making lots of individual network requests.

Answer 4

Use an NSOperationQueue and make each download a separate NSOperation. Set the maximum concurrent operations property on your queue to however many downloads you want to be able to run simultaneously. I'd keep it in the 4-6 range personally.

Here's a good blog post that explains how to make concurrent operations. http://www.dribin.org/dave/blog/archives/2009/05/05/concurrent_operations/

Answer 5

What came as a big surprise is how slow dataWithContentsOfURL is when downloading multiple files!

To see it by yourself run the following example: (you don't need the downloadQueue for downloadTaskWithURL, its there just for easier comparison)

- (IBAction)downloadUrls:(id)sender {
    [[NSOperationQueue new] addOperationWithBlock:^{
        [self download:true];
        [self download:false];
    }];
}

-(void) download:(BOOL) slow
{
       double startTime = CACurrentMediaTime();
        NSURLSessionConfiguration* config = [NSURLSessionConfiguration defaultSessionConfiguration];
        static NSURLSession* urlSession;

        if(urlSession == nil)
            urlSession = [NSURLSession sessionWithConfiguration:config delegate:nil delegateQueue:nil];

       dispatch_group_t syncGroup = dispatch_group_create();
       NSOperationQueue* downloadQueue = [NSOperationQueue new];
       downloadQueue.maxConcurrentOperationCount = 10;

       NSString* baseUrl = @"https://via.placeholder.com/468x60?text="; 
       for(int i = 0;i < 100;i++) {
           NSString* urlString = [baseUrl stringByAppendingFormat:@"image%d", i];
           dispatch_group_enter(syncGroup);
           NSURL  *url = [NSURL URLWithString:urlString];
           [downloadQueue addOperationWithBlock:^{
               if(slow) {
                   NSData *urlData = [NSData dataWithContentsOfURL:url];
                   dispatch_group_leave(syncGroup);
                   //NSLog(@"downloaded: %@", urlString);
               }
               else {
                   NSURLSessionDownloadTask* task = [urlSession downloadTaskWithURL:url completionHandler:^(NSURL * _Nullable location, NSURLResponse * _Nullable response, NSError * _Nullable error) {
                   //NSLog(@"downloaded: %@", urlString);
                   dispatch_group_leave(syncGroup);
                   }];[task resume];
               }
           }];
       }

       dispatch_group_wait(syncGroup, DISPATCH_TIME_FOREVER);

       double endTime = CACurrentMediaTime();
       NSLog(@"Download time:%.2f", (endTime - startTime));
}

Answer 6

There is nothing to "build". Just loop through the next 10 files each time in 10 threads and get the next file when a thread finishes.