So after experimentation and googling the conclusions I came to were as follows:
The Significant update change and in fact any process that launches the app behind the lock screen is the problem here. The file .plist the [NSUserDefaults defaultUser] loads is protected (NSFileProtectionCompleteUntilFirstUserAuthentication) so that it is not accessible until after the first unlock after the app is launched. So if a process launches your app in the background and your app tries to access the defaultUser user defaults it can't load the file and so gives you a new blank set of user defaults.
What happened here in my case is that the app then went into a state of waiting for the EULA and PP to be accepted since it read from the defaults (that couldn't be read) that they hadn't been accepted yet. After unlocking the phone and re-opening the app - which please note is already 'launched' - there are some processes that write to NSUserDefaults, some in my app and some in libraries my app uses. In most of those cases I was calling synchronise on the defaults therefore blasting away the old defaults that couldn't be read. I imagine this could be the case for a lot of people.
There are a few different solutions.
First I wrote a class equivalent to NSUserDefaults wrapping an NSMutableDictionary and saving the dictionary to a .plist in Library/Application Support. I changed the protection on the file to NSFileProtectionNone. Note this is not advisable if you store sensitive information in this file. Also note that you have to set the permissions on the file every time you write it. Something like:
NSError *error;
BOOL saved = [defaultsDic writeToURL:defaultsFileUrl atomically:YES];
[[NSFileManager defaultManager] setAttributes:[NSDictionary dictionaryWithObject:NSFileProtectionNone forKey:NSFileProtectionKey]ofItemAtPath:[defaultsFileUrl path] error:&error];
This method works fine but as it turns out I had another issue with data I was reading and writing from the keychain. See the link in my question above, its the same problem. The keychain values have the same protection up until the first unlock after the app is launched. I didn't want to remove the protection from the keychain and I actually wasn't super comfortable with removing the protection from my Custom user defaults either.
So the next solution is to actually solve the problem. Don't try to access protected data if the app is launched behind the lock screen! This means I have to detect that the app is launched behind the lock screen and then wait until the app is unlocked before i proceed to read my user defaults and keychain values.
The first requirement is to check on application launch if protected data is available so perhaps in applicationDidLaunch or somewhere else suitable.
[[UIApplication sharedApplication]isProtectedDataAvailable]
If this is not true when the app is launched then you are behind the lock screen. You should pause at this point and refrain from any operations that access NSUserDefaults or Keychain (or any protected file for that matter!). You then need to wait for a signal that the protected data has become available. The appDelegate receives the following when the user unlocks the lock screen:
-(void)applicationProtectedDataDidBecomeAvailable:(UIApplication *)application
Once you receive that you can carry on execution of your application.
In my case I control everything in a singleton class. When that class is created (which happens only at app launch) I check to see if protected data is available or not and subscribe to NSNotificationCenter for the same notification:
[[NSNotificationCenter defaultCenter]addObserver:self selector:@selector(applicationProtectedDataDidBecomeAvailable) name:UIApplicationProtectedDataDidBecomeAvailable object:nil];
So with this second method, the problem is solved and the data remains protected and everyone is happy.
