Download huge number of images in the most efficient and fast way

Question

In my app, I am needing to download about 6,000 images. I realize this is a lot, but it is needed.

Currently I am using doing the following:

NSArray *photos = @[hugeAmountOfPhotoObjects];
for (ZSSPhoto *photo in photos) {
    [self downloadImageWithURL:photo.mobileURL progress:^(double progress) {

    } completion:^(UIImage *image) {

        // Save the image

    } failure:^(NSError *error) {

    }];
}

...

- (void)downloadImageWithURL:(NSURL *)url progress:(void (^)(double progress))progress completion:(void (^)(UIImage *image))completion failure:(void (^)(NSError *error))failure {

    NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:url];
    [request setTimeoutInterval:600];
    self.operationQueue.maxConcurrentOperationCount = NSOperationQueueDefaultMaxConcurrentOperationCount;
    AFHTTPRequestOperation *requestOperation = [[AFHTTPRequestOperation alloc] initWithRequest:request];
    requestOperation.responseSerializer = [AFImageResponseSerializer serializer];
    [requestOperation setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
        completion(responseObject);
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
        [self processOperation:operation error:error failure:failure];
    }];
    [requestOperation setDownloadProgressBlock:^(NSUInteger bytesRead, long long totalBytesRead, long long totalBytesExpectedToRead) {
        double percentDone = (double)totalBytesRead / (double)totalBytesExpectedToRead;
        progress(percentDone);
    }];
    [self.operationQueue addOperation:requestOperation];

}

The problem here is that it takes forever to download using this method, and some of my users are reporting crashing because of high memory usage.

Is there a better method that I could be using to download such a large number of image files?

are you trying to download all photos, then put it in array, then only save it to disk? — Tj3n, Dec 07 '16 at 05:40
@Tj3n - No, I save them to disk when each image has been download and returned by the completion block. — Nic Hubbard, Dec 07 '16 at 05:47
i would put in some mechanism to download only a few at a time, maybe have an array of undownloaded images, then a for loop that takes like 3 of them out, downloads them, then when complete fetches 3 more etc — Fonix, Dec 07 '16 at 05:49
I hope you don't do all of this downloading over a user's cellular connection. — rmaddy, Dec 07 '16 at 05:52
@rmaddy - This is a user initiated/imported action. They are importing their own files that have links to images. — Nic Hubbard, Dec 07 '16 at 05:52

Fonix · Answer 1 · 2016-12-07T06:19:03.633

You could try this somewhat recursively

NSMutableArray *undownloaded;

- (void) startDownload {

    undownloaded = [photos mutableCopy]; //get a list of the undownloaded images

    for(int i = 0; i < 3;i++) //download 3 at a time
        [self downloadImage];
}


- (void) downloadImage {

    if(undownloaded.count > 0){

        ZSSPhoto *photo = undownloaded.firstObject;
        [undownloaded removeObjectAtIndex:0];

        [self downloadImageWithURL:photo.mobileURL progress:^(double progress) {

        } completion:^(UIImage *image) {

            // Save the image

            [self downloadImage];

        } failure:^(NSError *error) {

            [self downloadImage];
            //[undownloaded addObject:photo]; //insert photo back into the array maybe to retry? warning, could cause infinite loop without some extra logic, maybe the object can keep a fail count itself

        }];
    }
}

warning: untested code, may need some tweaking

See above, do not decompress the images into memory, just save the data directly to disk to avoid wasting all app memory. — MoDJ, Dec 12 '16 at 19:29

score 0 · Answer 2 · answered Dec 08 '16 at 11:46

The speed problem can be solved (the speed will increase, but it might still be slow) with multithreading, downloading all the images at the same time instead of one per time. However, the memory problem is a bit more complicated.

ARC will release all the images after everything is finished, but right before that you gonna have 6,000 images in the device memory. You could optimize the images, reduce their resolution or download them in steps, like Google Images do (you download the images that will be visible at first, then when the user scrolls down you load the images in the new visible area; downloading the images only when they are needed).

Considering that you are downloading enough images to give a memory problem, you will probably take a lots of space in your user's device if you download all of them, and the 'steps' solution may solve that as well.

Now, let's suppose you must download all of them at the same time and space isn't a problem: I suppose that if you put the downloadImageWithURL:progress: method inside a concurrent queue, the images are gonna be freed from memory just after saving (it's just a supposition). Add this to your code:

dispatch_queue_t defaultPriorityQueueWithName(const char* name)
{
    dispatch_queue_t dispatchQueue = dispatch_queue_create(name, DISPATCH_QUEUE_CONCURRENT);
    dispatch_queue_t priorityQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
    dispatch_set_target_queue(dispatchQueue, priorityQueue);
    return dispatchQueue;
}

And change your code to that:

dispatch_queue_t threadItemLoadImage = defaultPriorityQueueWithName("DownloadingImages");
NSArray *photos = @[hugeAmountOfPhotoObjects];
for (ZSSPhoto *photo in photos) 
{
    dispatch_async(threadItemLoadImage, ^
    {
        [self downloadImageWithURL:photo.mobileURL progress:^(double progress) {

        } completion:^(UIImage *image) {

            // Save the image

        } failure:^(NSError *error) {

        }];
    });
}

You will need to remove setDownloadProgressBlock: in case it updates some view, since they will be downloaded simultaneously. Also, a warning: totalBytesExpectedToRead not always will be correctly retrieved at first, containing 0, which might make your app crash for dividing by zero as well in some rare occasions. In future cases, when you need to use setDownloadProgressBlock:, check totalBytesExpectedToRead value before doing that division.

Are you kidding? Don't download the image and decompress into memory (the UIImage). Just download the files and save to disk, you are going to blow app memory out of the water. — MoDJ, Dec 12 '16 at 19:29
I just repeated @Nic Hubbard code adding the multi queue possibility since he's already using his `downloadImageWithURL:progress:` method and I don't know if he has a reason for using it, but surely, he can also try a solution like that one for downloading the file without consuming memory (however, that doesn't solves the speed problem): http://stackoverflow.com/questions/4002979/downloading-a-large-file-iphone-sdk — VitorMM, Dec 14 '16 at 12:24
You can only reduce download speed by reducing the size of the images being downloaded. The best way to do that is to use reduced colorspace and then shrink the image size with a more improved compression approach on the images. It is possible to shrink PNGs down with a 256 color table, but other newer approaches are also available to reduce image size. — MoDJ, Dec 14 '16 at 20:19
I guess the size of the images isn't the real speed problem cause. The fact is that he downloads 6,000 images, and the interval between each download also consumes too much time. — VitorMM, Dec 15 '16 at 15:43

Download huge number of images in the most efficient and fast way

2 Answers2