How can I determine when I'm done downloading files with AFNetworking

Question

I've read through most of the posts that have to do with the async nature of AFNetworking. But, my problem is a little unique and I could use some help.

I'm starting a new async thread from a modal view controller that is supposed to tell the user that I'm downloading content from a web server. The idea is to download a bunch of files (over 100) and it's working great. I've even put retry counts into the downloads so that it will retry to download any file if it fails (up to a maximum). Everything is downloading great. The problem is that I don't know when it's done.

And here's why: I am download a list of JSON files. Those JSON files define list of other JSON files which have lists of PDFs and other types of files. Because of the nature of what I'm downloading, I have to download it in sequence. For instance:

1. Download file1.json
2. Get list of JSON files from file1.json after it's downloaded
3. Load each of these secondary JSON files (subFile1.json, subFile2.json, etc.)
4. After a secondary JSON files downloads (subFile1.json), I get a list of files to download from that JSON file (subFile1.json)
5. Download each of the files specified in the JSON subfile (subFile1.json for example)

So my process looks hierarchical to insure parent files get downloaded before child files: 1. Initial Download Method calls SubDownloadMethod2 ON SUCCESS (within success block) 2. SubDownloadMethod2 gets list from downloaded file and calls SubSubDownloadMethod3 3. SubSubDownloadMethod3 download PDF files (and other files) ON SUCCESS (within success block)

So as you can see, I have a dynamic number of files to download. Hence I don't know how many files I'll be downloading up front. It's made more difficult based on the fact that it can go down multiple levels and come from multiple directories on the web server.

I also do an recursive callback to each method if the download fails (up to a max retry count) which also makes it more difficult.

Because each download starts it's own thread (I'm assuming that's what AFNetworking is doing), I'm not sure when all the downloads are done. I don't know if the enqueueBatchOfHTTPRequestOperations might help. I don't fully understand it and I'm downloading from more than one directory on the web server. I'd also need to batch operations based on each level of download since I don't know how far I'll go until I download and parse defined JSON files.

HELP!!!!

I think it may help to put the code in. It's a lot of code to look at but it's a difficult problem (well, for someone with my skills).

If you look at the code, every time I need to download another file, I call the method at the very bottom:

- (void)downloadLibraryFile:(NSString *)fileOnServer targetFile:(NSString *)targetFile retryCount:(int)retryCount completionBlock:(DownloadLibraryFileCompletionBlock)completionBlock
{
NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:fileOnServer]];
AFHTTPRequestOperation *operation = [[AFHTTPRequestOperation alloc] initWithRequest:request];

operation.outputStream = [NSOutputStream outputStreamToFileAtPath:targetFile append:NO];

[operation setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject)
    {
         //NSLog(@"Successfully downloaded file to %@", path);
         completionBlock(fileOnServer, targetFile, retryCount, nil);
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
         completionBlock(fileOnServer, targetFile, retryCount, error);
    }];

[operation start];
}

This starts up AFNetorking and starts the download. It calls my completion block when done but how do I know when they are all done?

Here is the rest of the code (including the above method)

- (void)downloadLibraryOnReset
{
// Find and copy the page defintion file to the documents directoy
// TODO localize the call to get the appropriate file based on language
dispatch_queue_t queue = dispatch_queue_create("Library Download Queue", NULL);
dispatch_async(queue, ^{
    NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
    NSString *serverLibraryURL = [defaults objectForKey:kRootURL];
    serverLibraryURL = [serverLibraryURL stringByAppendingPathComponent:kPageDefinitionsDirectory];

    // Save server root URL
    self.serverRootURL = serverLibraryURL;

    // Add last component onto download path
    serverLibraryURL = [serverLibraryURL stringByAppendingPathComponent:kPageDefinitions];

    // Get target location
    NSArray *dirPaths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);;
    NSString *targetFile = [dirPaths objectAtIndex:0];
    targetFile = [targetFile stringByAppendingPathComponent:@"en"]; // TODO this needs to be localized based on language
    targetFile = [targetFile stringByAppendingPathComponent:kPageDefinitionsDirectory];
    self.pageDefintiionDirectoryURL = targetFile;
    // Create the subdirectory off of the documents directory to contain the config files
    NSFileManager *filemgr = [NSFileManager defaultManager];
    if ([filemgr createDirectoryAtPath:targetFile withIntermediateDirectories:YES attributes:nil error: NULL] == NO)
    {
        // Failed to create directory
    }

    NSString *pageDefinitionsFileURL = [targetFile stringByAppendingPathComponent:kPageDefinitions];

    [self downloadPageDefinitionsFile:serverLibraryURL targetFile:pageDefinitionsFileURL retryCount:kDownloadRetryCount];

    // Reset the resetContent flag to false
    [defaults setBool:NO forKey:kResetContent];
});
}

- (void)downloadPageDefinitionsFile:(NSString *)fileOnServer targetFile:(NSString *)targetFile retryCount:(int)retryCount
{
[self downloadLibraryFile:fileOnServer targetFile:targetFile retryCount:retryCount completionBlock:^(NSString *fileOnServer, NSString *targetFile, int retryCount, NSError *error) {
    if(error)
    {
        retryCount--;
        if(retryCount)
        {
            NSLog(@"RETRY DONWLOAD (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), fileOnServer);
            [self downloadPageDefinitionsFile:fileOnServer targetFile:targetFile retryCount:retryCount];
        }
        else
        {
            NSLog(@"RETRY COUNT EXCEEDED (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount, NSStringFromSelector(_cmd), fileOnServer);
        }
    }
    else
    {
        // Check to see if this file was downloaded after an error
        if(retryCount < kDownloadRetryCount)
        {
            NSLog(@">>RETRY SUCCESSFUL (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), fileOnServer);
        }

        // Copy down all config files defined in the pagedefinitions.json file
        //code from other library
        NSError* err = nil;

        NSString *path = self.pageDefintiionDirectoryURL;
        path = [path stringByAppendingPathComponent:kPageDefinitions];

        NSData *data = [NSData dataWithContentsOfFile:path];

        if(data)
        {
            // Convert to JSON Directory
            NSMutableDictionary *pageDefinitionsDict = [NSJSONSerialization JSONObjectWithData:data options:kNilOptions error:&err];

            //fileVersion = pageDefinitionsDict[kFileVersion];
            NSMutableArray *pages = pageDefinitionsDict[kPages];
            for (NSMutableDictionary *page in pages)
            {
                MINPageDefinition *pageDef = [[MINPageDefinition alloc] initWithDictionary:page];
                NSString *targetDirectory = [targetFile stringByDeletingLastPathComponent];
                pageDef.pageURL = [targetDirectory stringByAppendingPathComponent:pageDef.pageConfigFileName];

                //NSString *imageURL = [pageDef.pageURL stringByDeletingLastPathComponent];
                pageDef.pageImageURL = [self.pageDefintiionDirectoryURL stringByAppendingPathComponent:pageDef.pageImageName];

                [[MINPageStore sharedInstance].pageArray addObject:pageDef];
            }
            // Write modified pagedefinitions.json to the appropriate directory in Documents
            [[MINPageStore sharedInstance] writePageDefinitionFile:path];

            // Continue downloading page images and other config files,
            for (MINPageDefinition *currPage in [[MINPageStore sharedInstance] pageArray])
            {
                [self downloadPageDefinitionImageFile:currPage retryCount:kDownloadRetryCount];
                [self downloadPageDefinitionJSONFile:currPage retryCount:kDownloadRetryCount];
            }
        }
    }
}];
}

- (void)downloadPageDefinitionJSONFile:(MINPageDefinition *)pageDef retryCount:(int)retryCount
{
NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
NSString *serverLibraryURL = [defaults objectForKey:kRootURL];
serverLibraryURL = [serverLibraryURL stringByAppendingPathComponent:kPageDefinitionsDirectory];
serverLibraryURL = [serverLibraryURL stringByAppendingPathComponent:pageDef.pageConfigFileName];

[self downloadLibraryFile:serverLibraryURL targetFile:pageDef.pageURL retryCount:retryCount completionBlock:^(NSString *fileOnServer, NSString *targetFile, int retryCount, NSError *error) {

    if(error)
    {
        retryCount--;
        if(retryCount)
        {
            NSLog(@"RETRY DONWLOAD (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), serverLibraryURL);
            [self downloadPageDefinitionJSONFile:pageDef retryCount:retryCount];
        }
        else
        {
            NSLog(@"RETRY COUNT EXCEEDED (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount, NSStringFromSelector(_cmd), serverLibraryURL);
        }
    }
    else
    {
        // Check to see if this file was downloaded after an error
        if(retryCount < kDownloadRetryCount)
        {
            NSLog(@">>RETRY SUCCESSFUL (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), serverLibraryURL);
        }

        // Copy down all config files defined in the pagedefinitions.json file
        if([pageDef.pageType isEqualToString:kGridView])
        {
            [self downloadGridViewContent:pageDef];
        }
        else
        {
            //NSLog(@">>>>FINISHED DOWNLOADING PAGE: %@", pageDef.pageName);
        }

    }
}];
}

- (void)downloadPageDefinitionImageFile:(MINPageDefinition *)pageDef retryCount:(int)retryCount
{
NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
NSString *serverLibraryURL = [defaults objectForKey:kRootURL];
serverLibraryURL = [serverLibraryURL stringByAppendingPathComponent:kPageDefinitionsDirectory];
serverLibraryURL = [serverLibraryURL stringByAppendingPathComponent:pageDef.pageImageName];

[self downloadLibraryFile:serverLibraryURL targetFile:pageDef.pageImageURL retryCount:retryCount completionBlock:^(NSString *fileOnServer, NSString *targetFile, int retryCount, NSError *error) {
    if(error)
    {
        retryCount--;
        if(retryCount)
        {
            NSLog(@"RETRY DONWLOAD (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), serverLibraryURL);
            [self downloadPageDefinitionImageFile:pageDef retryCount:retryCount];
        }
        else
        {
            NSLog(@"RETRY COUNT EXCEEDED (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount, NSStringFromSelector(_cmd), serverLibraryURL);
        }
    }
    else
    {
        // Check to see if this file was downloaded after an error
        if(retryCount < kDownloadRetryCount)
        {
            NSLog(@">>RETRY SUCCESSFUL (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), serverLibraryURL);
        }
        dispatch_async(dispatch_get_main_queue(), ^{
            [self.masterViewController.tableView reloadData];
        });
    }

}];
}

- (void)downloadGridViewContent:(MINPageDefinition *)pageDef
{
// Parse off the json extension
// Use this to create a subdirectory under the pagedefinitions directoy
NSString *newDirectoryForGridView = [pageDef.pageURL stringByDeletingPathExtension];
newDirectoryForGridView = [newDirectoryForGridView lastPathComponent];
NSString *newGridViewDirectoryURL = [pageDef.pageURL stringByDeletingPathExtension];

// Create the subdirectory off of the documents directory to contain the config files
NSFileManager *filemgr = [NSFileManager defaultManager];
if ([filemgr createDirectoryAtPath:newGridViewDirectoryURL withIntermediateDirectories:YES attributes:nil error: NULL] == NO)
{
    // Failed to create directory
}

// Load the grid view config file
[MINVolume loadAlbumItems:pageDef.pageURL completionBlock:^(NSString *fileName, MINVolume *newVolume, NSError *error) {
    if(!error)
    {
        if(newVolume && [newVolume.albumsArray count] > 0)
        {
            // Iterate through the albums and create directories for each album
            for(MINAlbum *album in newVolume.albumsArray)
            {
                NSString *localAlbumDirectory = [newGridViewDirectoryURL stringByAppendingPathComponent:album.albumURL];
                if ([filemgr createDirectoryAtPath:localAlbumDirectory withIntermediateDirectories:YES attributes:nil error: NULL] == NO)
                {
                    // Failed to create directory
                }
                // Copy down all album content
                for(MINAlbumItem *albumItem in album.albumItemsArray)
                {
                    // Create names for local file and thumbnail
                    NSString *localAlbumItemFileURL = [localAlbumDirectory stringByAppendingPathComponent:albumItem.itemFileName];
                    NSString *localAlbumItemFileThumbURL = [localAlbumDirectory stringByAppendingPathComponent:albumItem.itemThumbnailImageName];

                    // Define paths for file and thumbnail on server
                    NSString *serverAlbumItemFileURL = [self.serverRootURL stringByAppendingPathComponent:newDirectoryForGridView];
                    serverAlbumItemFileURL = [serverAlbumItemFileURL stringByAppendingPathComponent:album.albumURL];
                    serverAlbumItemFileURL = [serverAlbumItemFileURL stringByAppendingPathComponent:albumItem.itemFileName];
                    NSString *serverAlbumItemFileThumbURL = [self.serverRootURL stringByAppendingPathComponent:newDirectoryForGridView];
                    serverAlbumItemFileThumbURL = [serverAlbumItemFileThumbURL stringByAppendingPathComponent:album.albumURL];
                    serverAlbumItemFileThumbURL = [serverAlbumItemFileThumbURL stringByAppendingPathComponent:albumItem.itemThumbnailImageName];

                    // Copy album item file
                    BOOL bFileExists = [filemgr fileExistsAtPath:localAlbumItemFileURL];
                    if(!bFileExists)
                    {
                        [self downloadAlbumItem:albumItem isThumbnail:(BOOL)false fileOnServer:serverAlbumItemFileURL targetFile:localAlbumItemFileURL retryCount:kDownloadRetryCount];
                    }
                    else
                    {
                        albumItem.itemURL = localAlbumItemFileURL;
                    }

                    // Copy album item thumbnail
                    BOOL bFileThumbnailExists = [filemgr fileExistsAtPath:localAlbumItemFileThumbURL];
                    if(!bFileThumbnailExists)
                    {
                        [self downloadAlbumItem:albumItem isThumbnail:true fileOnServer:serverAlbumItemFileThumbURL targetFile:localAlbumItemFileThumbURL retryCount:kDownloadRetryCount];
                    }
                    else
                    {
                        albumItem.itemThumbnailURL = localAlbumItemFileThumbURL;
                    }
                }
            }
        }
        else
        {
            NSLog(@"No volume found for file: %@", pageDef.pageConfigFileName);
        }
    }
}];
}

- (void)downloadAlbumItem:(MINAlbumItem *)albumItem isThumbnail:(BOOL)isThumbnail fileOnServer:(NSString *)fileOnServer targetFile:(NSString *)targetFile retryCount:(int)retryCount
{
[self downloadLibraryFile:fileOnServer targetFile:targetFile retryCount:retryCount completionBlock:^(NSString *fileOnServer, NSString *targetFile, int retryCount, NSError *error) {
    if(error)
    {
        retryCount--;
        if(retryCount)
        {
            NSLog(@"RETRY DONWLOAD (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), fileOnServer);
            [self downloadAlbumItem:albumItem isThumbnail:isThumbnail fileOnServer:fileOnServer targetFile:targetFile retryCount:retryCount];
        }
        else
        {
            NSLog(@"RETRY COUNT EXCEEDED (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount, NSStringFromSelector(_cmd), fileOnServer);
        }
    }
    else
    {
        // Check to see if this file was downloaded after an error
        if(retryCount < kDownloadRetryCount)
        {
            NSLog(@">>RETRY SUCCESSFUL (Attempt: %d, Method: %@) > File: %@", kDownloadRetryCount - retryCount, NSStringFromSelector(_cmd), fileOnServer);
        }

        if(isThumbnail)
            albumItem.itemThumbnailURL = targetFile;
        else
            albumItem.itemURL = targetFile;
    }
}];
}

- (void)downloadLibraryFile:(NSString *)fileOnServer targetFile:(NSString *)targetFile retryCount:(int)retryCount completionBlock:(DownloadLibraryFileCompletionBlock)completionBlock
{
NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:fileOnServer]];
AFHTTPRequestOperation *operation = [[AFHTTPRequestOperation alloc] initWithRequest:request];

operation.outputStream = [NSOutputStream outputStreamToFileAtPath:targetFile append:NO];

[operation setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject)
    {
         //NSLog(@"Successfully downloaded file to %@", path);
         completionBlock(fileOnServer, targetFile, retryCount, nil);
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
         completionBlock(fileOnServer, targetFile, retryCount, error);
    }];

[operation start];
}

Answer 1

If you're doing an AFHTTPRequestOperation, you can use it's setCompletionBlockWithSuccess method to set a block, which will be called once the download (or even upload) finishes.

Example:

AFHTTPRequestOperation *httpreq = [[AFHTTPRequestOperation alloc] initWithRequest:[NSURLRequest requestWithURL:downloadURL]];
[httpreq setCompetionBlockWithSuccess^(AFHTTPRequestOperation *operation, id responseObject) {
NSLog(@"%@",@"Done!");
}];

Also, you don't need to start the operation in a different thread as it already runs asynchronously.

Answer 2

you add all operation in NSOperationQueue like this:

NSOperationQueue *queue = [[NSOperationQueue alloc] init];
[queue addOperation: operation]; //your operation here
[operation start] 
[operation waitUntilFinish]

How to maintain NSOperationQueue refer below links:

Refer ios-how-to-know-when-nsoperationqueue-finish-processing-a-few-operations link.

Refer ios-how-to-check-if-an-nsoperation-is-in-an-nsoperationqueue link.

Refer ios-nsoperationqueue-operations-all-run-when-added-and-dont-queue link.

Answer 3

I would consider using an AFHTTPClient to handle your downloads especially since you want to manage multiple downloads. Once you initialise the client, you would need to create all your operations and then add them to the clients operation queue using the method -[AFHTTPClient enqueueBatchOfHTTPRequestOperations: progressBlock: completionBlock:] This way you can also determine the progress of the download using the progress block. Furthermore, the completion block will allow you to perform your own custom code on completion.

So you would start by initialising your client:

AFHTTPClient *client =[AFHTTPClient clientWithBaseURL:baseURL];

Then you want to create your HTTP Request Operations:

NSMutableURLRequest *request = [client requestWithMethod:@"GET" path:@"/" parameters:nil];
AFHTTPRequestOperation *firstOperation = [client HTTPRequestOperationWithRequest:request success:nil failure:nil];
AFHTTPRequestOperation *secondOperation = [client HTTPRequestOperationWithRequest:request success:nil failure:nil];

Then you would simply add these operations to the clients operation queue:

[client enqueueBatchOfHTTPRequestOperations:@[firstOperation, secondOperation, thirdImageRequestOperation] 
                              progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations) {
                              //Handle progress here
                              } 
                            completionBlock:^(NSArray *operations) {
                             //Handle completion of all downloads here
}];

Hope this helps:)