How do I wait for a large file to be downloaded?

Question

I have an app that successfully uses the synchronous methods to download files (NSData's initWithContentsOfURL and NSURLConnection's sendSynchronousRequest), but now I need to support large files. This means I need to stream to disk bit by bit. Even though streaming to disk and becoming asynchronous should be completely orthoganal concepts, Apple's API forces me to go asynchronous in order to stream.

To be clear, I am tasked with allowing larger file downloads, not with re-architecting the whole app to be more asynchronous-friendly. I don't have the resources. But I acknowledge that the approaches that depend on re-architecting are valid and good.

So, if I do this:

NSURLConnection* connection = [ [ NSURLConnection alloc ] initWithRequest: request delegate: self startImmediately: YES ];

.. I eventually have didReceiveResponse and didReceiveData called on myself. Excellent. But, if I try to do this:

NSURLConnection* connection = [ [ NSURLConnection alloc ] initWithRequest: request delegate: self startImmediately: YES ];
while( !self.downloadComplete )
    [ NSThread sleepForTimeInterval: .25 ];

... didReceiveResponse and didReceiveData are never called. And I've figured out why. Weirdly, the asynchronous download happens in the same main thread that I'm using. So when I sleep the main thread, I'm also sleeping the thing doing the work. Anyway, I have tried several different ways to achieve what I want here, including telling the NSURLConnection to use a different NSOperationQueue, and even doing dispatch_async to create the connection and start it manually (I don't see how this couldn't work - I must not have done it right), but nothing seems to work. Edit: What I wasn't doing right was understanding how Run Loops work, and that you need to run them manually in secondary threads.

What is the best way to wait until the file is done downloading?

Edit 3, working code: The following code actually works, but let me know if there's a better way.

Code executing in the original thread that sets up the connection and waits for the download to complete:

dispatch_queue_t downloadQueue = dispatch_get_global_queue( DISPATCH_QUEUE_PRIORITY_DEFAULT, 0 );
dispatch_async(downloadQueue, ^{
    self.connection = [ [ NSURLConnection alloc ] initWithRequest: request delegate: self startImmediately: YES ];
    [ [ NSRunLoop currentRunLoop ] run ];
});

while( !self.downloadComplete )
    [ NSThread sleepForTimeInterval: .25 ];

Code executing in the new thread that responds to connection events:

-(void)connection:(NSURLConnection*) connection didReceiveData:(NSData *)data {
    NSUInteger remainingBytes = [ data length ];
    while( remainingBytes > 0 ) {
        NSUInteger bytesWritten = [ self.fileWritingStream write: [ data bytes ] maxLength: remainingBytes ];
        if( bytesWritten == -1 /*error*/ ) {
            self.downloadComplete = YES;
            self.successful = NO;
            NSLog( @"Stream error: %@", self.fileWritingStream.streamError );
            [ connection cancel ];
            return;
        }
        remainingBytes -= bytesWritten;
    }
}

-(void)connection:(NSURLConnection*) connection didFailWithError:(NSError *)error {
    self.downloadComplete = YES;
    [ self.fileWritingStream close ];
    self.successful = NO;
}

- (void)connectionDidFinishLoading:(NSURLConnection *)connection {
    self.downloadComplete = YES;
    [ self.fileWritingStream close ];
    self.successful = YES;
}

Answer 1

... didReceiveResponse and didReceiveData are never called. And I've figured out why. Weirdly, the asynchronous download happens in the same main thread that I'm using. It doesn't create a new thread. So when I sleep the main thread, I'm also sleeping the thing doing the work.

Exactly. The connection is driven by the run loop; if you sleep the thread, the run loop stops, and that prevents your connection from doing its thing.

So don't do anything special. Let the app sit there, with the run loop running. Maybe put a little spinner on the screen to entertain the user. Go about your business if you can. If at all possible, let the user continue to use the application. Your delegate method will be called when the connection is complete, and then you can do what you need to do with the data.

When you move your code to a background thread, you'll again need a run loop to drive the connection. So you'll start create a run loop, schedule your connection, and then just return. The run loop will keep running, and your delegate method will again be called when the connection completes. If the thread is done, you can then stop the run loop and let the thread exit. That's all there is to it.

Example: Let's put this in concrete terms. Let's say that you want to make a number of connections, one at a time. Stick the URL's in a mutable array. Create a method called (for example) startNextConnection that does the following things:

• grabs an URL from the array (removing it in the process)

• creates an URL request

• starts a NSURLConnection

• return

Also, implement the necessary NSURLConnectionDelegate methods, notably connectionDidFinishLoading:. Have that method do the following:

• stash the data somewhere (write it to a file, hand it to another thread for parsing, whatever)

• call startNextConnection

• return

If errors never happened, that'd be enough to retrieve the data for all the URLs in your list. (Of course, you'll want startNextConnection to be smart enough to just return when the list is empty.) But errors do happen, so you'll have to think about how to deal with them. If a connection fails, do you want to stop the entire process? If so, just have your connection:didFailWithError: method do something appropriate, but don't have it call startNextConnection. Do you want to skip to the next URL on the list if there's an error? Then have ...didFailWithError: call startNextRequest.

Alternative: If you really want to keep the sequential structure of your synchronous code, so that you've got something like:

[self downloadURLs];
[self waitForDownloadsToFinish];
[self processData];
...

then you'll have to do the downloading in a different thread so that you're free to block the current thread. If that's what you want, then set up the download thread with a run loop. Next, create the connection using -initWithRequest:delegate:startImmediately: as you've been doing, but pass NO in the last parameter. Use -scheduleInRunLoop:forMode: to add the connection to the download thread's run loop, and then start the connection with the -start method. This leaves you free to sleep the current thread. Have the connection delegate's completion routine set a flag such as the self.downloadComplete flag in your example.

Answer 2

I hesitate to provide this answer because the others are correct that you really should structure your app around the asynchronous model. Nevertheless:

NSURLConnection* connection = [[NSURLConnection alloc] initWithRequest:request delegate:self startImmediately:NO];
NSString* myPrivateMode = @"com.yourcompany.yourapp.DownloadMode";
[connection scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:myPrivateMode];
[connection start];
while (!self.downloadComplete)
    [[NSRunLoop currentRunLoop] runMode:myPrivateMode beforeDate:[NSDate distantFuture]];

Do not do this on the main thread. Your app is just as likely to be terminated for blocking the main thread as for downloading too big a file to memory.

By the way, given that you're downloading to a file instead of memory, you should consider switching from NSURLConnection to NSURLDownload.

Answer 3

I think your sleepForInterval is blocking the NSURLConnection's activity -

No run loop processing occurs while the thread is blocked.

From the NSThread documentation.

---

I think you might have to rethink how you're setting your downloadComplete variable. Consider using your connectionDidFinishLoading:connection delegate method to determine when the download is complete instead of your loop + sleep?

- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{        
    self.downloadComplete = YES;

    // release the connection, and the data object
    [connection release];
    [receivedData release];
}

From the NSURLConnection guide.

You can use the connection:connection didFailWithError:error delegate method to ensure you're dealing with situations where the download does not complete.