2

I've got an application that stores products in a Core Data file. These pruducts include images as "Transformable" data. Now I tried adding some attributes using Lightweight migration. When I tested this with a small database it worked well but when I use a really large one with nearly 500 MB the application usually crashes because of low memory. Does anybody know how to solve this problem?

Thanks in advanced!

HackingOtter
  • 134
  • 6
  • I haven't used core data but my understanding is that it isn't suited to large databases and some other DB implementation might be a better idea. Perhaps MySQL? – Robert Feb 15 '13 at 16:53
  • 3
    @Robert - not true at all. There are situations where it isn't suitable, but size isn't one of them. Also, CoreData isn't a DB. HackingOtter might want to consider storing only the URL for the images and keep the images in the Documents directory. – sosborn Feb 15 '13 at 17:17
  • Ah, OK. Not sure where I got that idea from. – Robert Feb 15 '13 at 17:20

1 Answers1

6

You'll have to use one of the other migration options. The automatic lightweight migration process is really convenient to use. But it has the drawback that it loads the entire data store into memory at once. Two copies, really, one for before migration and one for after.

First, can any of this data be re-created or re-downloaded? If so, you might be able to use a custom mapping model from the old version to the new one. With a custom mapping model you can indicate that some attributes don't get migrated, which reduces memory issues by throwing out that data. Then when migration is complete, recreate or re-download that data.

If that's not the case... Apple suggests a multiple pass technique using multiple mapping models. If you have multiple entity types that contribute to the large data store size, it might help. Basically you end up migrating different entity types in different passes, so you avoid the overhead of loading everything at once.

If that is not the case then (e.g. the bloat is all from instances of the same entity type), well, it's time to write your own custom migration code. This will involve setting up two Core Data stacks, one with the existing data and one with the new model. Run through the existing data store, creating new objects in the new store. If you do this in batches you'll be able to keep memory under control. The general approach would be:

  1. Create new instances in the new model and copy attributes only. You can't set up relationships yet because related objects might not exist in the new data store. Keep a mutable dictionary mapping NSManagedObjectIDs from the old store to the new one, for use in the next step. To keep memory use low:
    • As soon as you have created a destination store object, free up the memory for the source object by using refreshObject:mergeChanges with NO for the second argument.
    • Every 10 instances (or 50, or whatever) save changes on the destination managed object context and then reset it. The interval is a balancing act-- do it too often and you'll slow down unnecessarily, do it too rarely and memory use rises.
  2. Do a second pass where you set up relationships in the destination store. For each source object,
    • Find the corresponding destination object, using the object ID map you created
    • Run through the source object's relationships. For each one, find the corresponding destination object, also using the object ID map.
    • Set the destination object's relationship based on the result.

While you are at it consider why your data store is so big. Are you storing a bunch of binary data blobs in the data store? If so, make sure you're using the "Allows external storage" option in the new model.

Tom Harrington
  • 69,312
  • 10
  • 146
  • 170
  • hThanks for this good answer. Does somebody know a good tutorial for this? – HackingOtter Feb 15 '13 at 21:03
  • This guy describes part of it, but he didn't have relationships to deal with: http://hamishrickerby.com/2012/06/04/core-data-migrations-and-large-data-sets/ – Tom Harrington Feb 15 '13 at 22:43
  • This is great insight for those of us unfortunate enough to realize the last strategy is the only appropriate one. But, I want to suggest that calling [srcContext refreshObject:srcEntity mergeChanges:NO] on the source entity only solves half of the memory issue. Wont you have just about as much memory floating around per the destination objects? – stephen Mar 26 '13 at 02:38
  • Somewhat, yes, but that's where doing it in batches comes in. At regular intervals, save changes and reset the destination context. That limits memory growth since you're clearing it out periodically. – Tom Harrington Mar 26 '13 at 03:17