1

I need a hint on how to write multiple NSData chunks to single file. Downloading a file using NSURLConnection in chunks. Each chunk is downloaded in a separate NSOperation thread. As the chunks finish downloading they need to be written to a file so combined result is the file downloaded.

What would be the best way to manage the NSData that is returned and writing it to a single file?

MikeM
  • 59
  • 1
  • 7
  • 1
    possible duplicate of http://stackoverflow.com/questions/1255295/append-nsdata-to-a-file-in-objective-c – Martin Gordon Apr 27 '10 at 14:36
  • @Martin Gordon: I don't think this is a duplicate. It may just be worded so that I don't understand it, but it sounds to me that he is downloading discrete chunks on other threads and then has to recreate the file. If that is true, appending chunks as they come in would not work since the file would not be sequenced properly (as any chunk may complete before any other chunk). – Jason Coco Apr 27 '10 at 15:00
  • As Jason Cocoa said each chunk (arbitrary size) of file is downloaded in a separate thread (so scheduling can be an issue when writing to file). All of the data needs to be combined into a single file. But since the file might be of sizes exceeding 100MB it is important that the data is written and not kept in memory. – MikeM Apr 27 '10 at 16:31

3 Answers3

11

No need to eat up memory by creating a large NSMutableData object (not possible with a large download anyway as you would run out of memory), nor do you need to waste time creating tons of small files and concating them (could take a very long time with a large file, iDevice disk IO is not very fast).

Just create an NSFileHandle and use it to write each NSData object to the end of the file as they come in.

NSFileHandle *handle = [NSFileHandle fileHandleForUpdatingAtPath:filePath];
[handle seekToEndOfFile];
[handle writeData:dataPiece];
[handle closeFile];

You'll need to first create the file though so that you can open it with the NSFileHandle. To do that you can just write the first NSData piece using the following, then write the rest using the file handle.

[dataPiece writeToFile:filePath atomically:YES];

edit: Actually I just re-read the question and realized you're using separate threads to download the chunks so they may not finish in order, so my solution won't work. I must have been tired the day I answered and skipped right over it. But, hopefully my answer can at least help others that are downloading chunks in order or single threaded.

Ben Baron
  • 14,496
  • 12
  • 55
  • 65
1

Assuming that you know what the final data size will be, and you want to stick to Foundation classes/Objective-C, you could create an instance of NSMutableData that can be shared across these operations. When an operation completes its chunk, it should lock some shared mutex, write its completed download to the appropriate place in the NSMutableData object and then unlock the shared mutex.

Once all the operations are joined, you can simply write the mutable data to a file using the writeToFile: convenience methods on the NSData class. If you're more proficient in C/BSD, you could also create the file as an mmap and simply write to it. Since all the data are going to discrete seconds in the mapping, you can write without locking a mutex. Once all the operations are joined, you can remove the mmap and close the file.

Jason Coco
  • 77,985
  • 20
  • 184
  • 180
  • C/BSD approach might be one way of doing it but i'm looking for Objective-C answer to this. What you gave will work but i'm concerned with the file size and keeping it all in memory until finish downloading. The file size will exceed 100MB. – MikeM Apr 27 '10 at 15:14
  • @user326943: If you are worried about file size, I think that mmap is the way to go. The other option would be to simply have a file handle open and seek/write the file. This option would require locking obviously and there may be a lot of contention as threads are joined since writing to the flash device is waaaay slow. – Jason Coco Apr 27 '10 at 16:46
  • @user326943: Another all Cocoa option might to be to have the operation write a temporary file named with something meaningful like the chunk's sequence in the main file. Once all the operations are joined, you can combine all these temporary files into the final, large file, reading and writing them in chunks on a single thread. – Jason Coco Apr 27 '10 at 16:47
  • +1 for `mmap`, see http://stackoverflow.com/questions/9319874/writing-in-a-file-from-multiple-threads/9321834 for a simple implementation. – mvds Feb 19 '12 at 12:14
1

Write each chunk to a separate file. Then when the last chunk is downloaded, concatenate each file in the correct order into a single large file.

lucius
  • 8,665
  • 3
  • 34
  • 41
  • Can you give more details on how that can be done. How do i append everything to a single file after i have the multiple files? – MikeM Apr 28 '10 at 02:57
  • 3
    You can use a loop with NSInputStream and initWithFileAtPath: to read a small buffer (4 kb is ideal) at a time, and use +[NSOutputStream outputStreamToFileAtPath:append:] to write to the new file. – lucius Apr 28 '10 at 16:06