0

I need the ability to import some fairly large text files (100Mb+) into CoreData in an application targeted towards mobile devices where memory is constrained. Each file contains a large number of small records which will be processed before being added to the database. Looking through many sources the recommended method for reading in a text file seems to be:

NSString *stringFromFileAtPath = [[NSString alloc]initWithContentsOfURL:url encoding:NSUTF8StringEncoding error:&error];

At first glance this seems like a very memory intensive way of doing what I require, but given that there seems to be no other recommended way to read the file would I be right in guessing that Apple have taken this into account and do their own memory management - perhaps faulting in data from the file only when necessary?

If not would the best way to proceed be using NSStream and NSScanner to retrieve and process one line of text at a time?

If the recommended method does handle memory well then the next step is often:

NSArray *lines = [stringFromFileAtPath componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]];

If I use this method I'm assuming that it would need the complete text file in memory so again it would be memory intensive. To save memory would I be better off using NSScanner or, given the limited processing power of mobile devices (certainly some of the older ones) would it take forever to complete?

Thanks in advance for any help you can give me with this question.

Dave

Setanta
  • 56
  • 4

3 Answers3

6

Memory-mapped files can be a good way of examining the contents of a large file. And it looks like -[NSString initWithBytesNoCopy:length:encoding:freeWhenDone:] will let you create an NSString that uses a memory-mapped file's contents as the string's value directly.

I haven't used that particular function, I must admit, but my app does use memory-mapped files and they were dead easy to get working. So at the very least, you have an easy way of not needing to have 100MB of data loaded at once, but all the convenience of having 100MB of data appearing to be loaded at once.

Use the POSIX function mmap to map a file into memory. You'll need a POSIX file handle, which my code gets from open (after doing the usual song and dance to get a UTF8 copy of the right file name in the bundle), but which can probably be done using one of the NS facilities too.

  • That sounds very interesting and would certainly be a possibility. Time to get my head into the docs for NSString again! – Setanta Mar 27 '11 at 20:18
1

Have you tried this solutions?

How to read data from NSFileHandle line by line?

Community
  • 1
  • 1
Nik
  • 9,063
  • 7
  • 66
  • 81
  • Well dang, no I hadn't - I thought I'd searched everywhere but I guess I was using the wrong keywords. Those are a couple of useful bits of code which I'll be storing away very quickly. – Setanta Mar 27 '11 at 20:15
  • Having had a closer look at the code it really doesn't do what I need - there doesn't seem to be any significant memory saving involved - so I'm currently rolling my own with NSStream which is making a huge memory saving. – Setanta Mar 30 '11 at 21:22
1

I think the best way to get a lot of data into a database is to deliver a pre-populated database (*.sqlite-file).

Jojo.Lechelt
  • 1,259
  • 12
  • 15
  • This is the correct answer, don't write custom code to load huge volumes data. Just write a sqlite file from your data on the desktop and put it up at a URL. sqlite takes care of opening the file and accesing the data row by row and it is super fast. – MoDJ Jul 16 '13 at 22:01