5

In my app I want to save user settings in a plist file for each user logs in, I write one class called CCUserSettings which has almost the same interface as NSUserDefaults and it reads and writes a plist file related to the current user id. It works but has poor performance. Every time user calls [[CCUserSettings sharedUserSettings] synchronize], I write a NSMutableDictionary(which keep the user settings) to a plist file, the code below shows synchronize of CCUserSettings omitting some trivial details.

- (BOOL)synchronize {
    BOOL r = [_settings writeToFile:_filePath atomically:YES];
    return r;
}

I suppose NSUserDefaults should write to files when we call [[NSUserDefaults standardUserDefaults] synchronize], but it runs really fast, I write a demo to test, the key part is below, run 1000 times [[NSUserDefaults standardUserDefaults] synchronize] and [[CCUserSettings sharedUserSettings] synchronize] on my iPhone6, the result is 0.45 seconds vs 9.16 seconds.

NSDate *begin = [NSDate date];
for (NSInteger i = 0; i < 1000; ++i) {
    [[NSUserDefaults standardUserDefaults] setBool:(i%2==1) forKey:@"key"];
    [[NSUserDefaults standardUserDefaults] synchronize];
}
NSDate *end = [NSDate date];
NSLog(@"synchronize seconds:%f", [end timeIntervalSinceDate:begin]);


[[CCUserSettings sharedUserSettings] loadUserSettingsWithUserId:@"1000"];
NSDate *begin = [NSDate date];
for (NSInteger i = 0; i < 1000; ++i) {
    [[CCUserSettings sharedUserSettings] setBool:(i%2==1) forKey:@"_boolKey"];
    [[CCUserSettings sharedUserSettings] synchronize];
}
NSDate *end = [NSDate date];
NSLog(@"CCUserSettings modified synchronize seconds:%f", [end timeIntervalSinceDate:begin]);

As the result shows, NSUserDefaults is almost 20 times faster than my CCUserSettings. Now I start to wonder that "Does NSUserDefaults really write to the plist files every time we call synchronize method?", but if it doesn't, how can it guarantee the data write back to file before the process exits(as the process may be killed at any time)?

These days I come up with an idea to improve my CCUserSettings, it is mmap Memory-mapped I/O. I can map a virtual memory to a file and every time user calls synchronize, I create a NSData with NSPropertyListSerialization dataWithPropertyList:format:options:error: method and copy the data into that memory, operating system will write memory back to file when process exits. But I may not get a good performance because the file size is not fixed, every time the length of data increases, I have to remmap a virtual memory, I believe the operation is time consuming.

Sorry for my redundant details, I just want to know how NSUserDefaults works to achieve so good performance, or can anyone have some good advices to improve my CCUserSettings ?

KudoCC
  • 6,912
  • 1
  • 24
  • 53
  • 1
    Here is a nice article about `NSUserDefaults` : http://dscoder.com/defaults.html . Its author is an engineer at Apple, so its quite safe to assume he knows what he is talking about :) – Losiowaty Dec 15 '16 at 14:48
  • @Losiowaty Thanks for your link, but I think it talks about the implementation in mac ox, because it says "Setting a value will (eventually, it's asynchronous and occurs some time later in another process) write out the entire plist to disk, no matter how small the change was.". If you modify the NSUserDefaults and kill you app without `synchronize`, the settings won't write to file, so I don't think there is another process writing the file in iOS. – KudoCC Dec 16 '16 at 01:45
  • If you modify NSUserDefaults and kill the app, you will have to be *very* fast on the kill button to lose data. A few milliseconds or so. This changed in iOS 8; before then, it was much easier to lose data that way. – Catfish_Man Dec 19 '16 at 06:47

2 Answers2

4

On modern operating systems (iOS 8+, macOS 10.10+), NSUserDefaults does not write the file when you call synchronize. When you call -set* methods, it sends an async message to a process called cfprefsd, which stores the new values, sends a reply, and then at some later time writes the file out. All -synchronize does is wait for all outstanding messages to cfprefsd to receive replies.

(edit: you can verify this, if you like, by setting a symbolic breakpoint on xpc_connection_send_message_with_reply and then setting a user default)

Catfish_Man
  • 41,261
  • 11
  • 67
  • 84
  • Thanks for your answer! But if the IO operation happens on the other process, why did I lose the settings which set just before I call `exit`? – KudoCC Dec 19 '16 at 06:56
  • The fact that I lose the settings which set just before I call `exit` doesn't mean you're false, maybe the cfprefsd process need our process to send some other message. I'm just curious to know : ) BTW how did you know that, can you post some references. – KudoCC Dec 19 '16 at 07:04
  • There's two reasons that can occur. The first is the simplest one: the message sending is asynchronous, so your exit() call may happen before the message has actually made it out of your process. The second is subtler: cfprefsd must verify your sandbox permissions, to make sure you're allowed to get to those preferences. Checking sandbox permissions requires your process to still be running. – Catfish_Man Dec 19 '16 at 07:08
  • Just checked `xpc_connection_send_message_with_reply`, you're right. – KudoCC Dec 19 '16 at 07:19
3

Finally I come up with an solution to improve the performance of my CCUserSettings with mmap, I call it CCMmapUserSettings.

Prerequisite

The synchronize in CCUserSettings or NSUserDefaults method writes the plist file back to the disk, it costs notable time, but we must call it in some situations like when app goes into background. Even so we take the risk of losing the settings: we apps may be killed by system because it runs out of memory or accesses a address which it hasn't permission to, at that time the settings we set after the latest synchronize may lose.

If there is a way that we can write the file to the disk when the process exits, we can modify the settings in memory all the time, it's pretty fast. But is there a way to achieve that ?

Well, I find one, it is mmap, mmap maps a file to a region of memory. When this is done, the file can be accessed just like an array in the program. So we can modify the memory as if we write the file. When process exits, the memory will write back to the file.

There are two links supporting me:

Does the OS (POSIX) flush a memory-mapped file if the process is SIGKILLed?

mmap, msync and linux process termination

Problem of using mmap

As I mentioned in my question:

These days I come up with an idea to improve my CCUserSettings, it is mmap Memory-mapped I/O. I can map a virtual memory to a file and every time user calls synchronize, I create a NSData with NSPropertyListSerialization dataWithPropertyList:format:options:error: method and copy the data into that memory, operating system will write memory back to file when process exits. But I may not get a good performance because the file size is not fixed, every time the length of data increases, I have to remmap a virtual memory, I believe the operation is time consuming.

The problem is: every time the length of data increases, I have to remmap a virtual memory, it is time consuming operation.

Solution

Now I have a solution: always create a bigger size than we need and keep the real file size in the beginning 4 bytes of the file and write the real data after the 4 bytes. As the file is bigger than what we need, when the data is increasing smoothly, we don't need to remmap memory at every call of synchronize. There is another restriction on file size : file size is always multiple of MEM_PAGE_SIZE(defined as 4096 in my app).

The synchronize method:

- (BOOL)synchronize {
    if (!_changed) {
        return YES;
    }
    NSData *data = [NSPropertyListSerialization dataWithPropertyList:_settings format:NSPropertyListXMLFormat_v1_0 options:0 error:nil];
    // even if data.length + sizeof(_memoryLength) is a multiple of MEM_PAGE_SIZE, we need one more page.
    unsigned int pageCount = (unsigned int)(data.length + sizeof(_memoryLength)) / MEM_PAGE_SIZE + 1;
    unsigned int fileSize = pageCount * MEM_PAGE_SIZE;
    if (fileSize != _memoryLength) {
        if (_memory) {
            munmap(_memory, _memoryLength);
            _memory = NULL;
            _memoryLength = 0;
        }

        int res = ftruncate(fileno(_file), fileSize);
        if (res == -1) {
            // truncate file error
            fclose(_file);
            _file = NULL;
            return NO;
        }
        // re-map the file
        _memory = (unsigned char *)mmap(NULL, fileSize, PROT_READ|PROT_WRITE, MAP_SHARED, fileno(_file), 0);
        _memoryLength = (unsigned int)fileSize;
        if (_memory == MAP_FAILED) {
            _memory = NULL;
            fclose(_file);
            _file = NULL;
            return NO;
        }
#ifdef DEBUG
        NSLog(@"memory map file success, size is %@", @(_memoryLength));
#endif
    }

    if (_memory) {
        unsigned int length = (unsigned int)data.length;
        length += sizeof(length);
        memcpy(_memory, &length, sizeof(length));
        memcpy(_memory+sizeof(length), data.bytes, data.length);
    }
    return YES;
}

An example will help to describe my thought: suppose the plist data size is 5000 bytes, the total bytes I need to write is 4 + 5000 = 5004. I write 4 bytes unsigned integer which value is 5004 first then write the 5000 bytes data. The total file size should be 8192(2*MEM_PAGE_SIZE). The reason I create a bigger file is I need a big buffer to reduce the time to re-mmap memory.

Performance

{
    [[CCMmapUserSettings sharedUserSettings] loadUserSettingsWithUserId:@"1000"];
    NSDate *begin = [NSDate date];
    for (NSInteger i = 0; i < 1000; ++i) {
        [[CCMmapUserSettings sharedUserSettings] setBool:(i%2==1) forKey:@"_boolKey"];
        [[CCMmapUserSettings sharedUserSettings] synchronize];
    }
    NSDate *end = [NSDate date];
    NSLog(@"CCMmapUserSettings modified synchronize seconds:%f", [end timeIntervalSinceDate:begin]);
}

{
    [[NSUserDefaults standardUserDefaults] setBool:YES forKey:@"key"];
    NSDate *begin = [NSDate date];
    for (NSInteger i = 0; i < 1000; ++i) {
        [[NSUserDefaults standardUserDefaults] setBool:YES forKey:@"key"];
        [[NSUserDefaults standardUserDefaults] synchronize];
    }
    NSDate *end = [NSDate date];
    NSLog(@"NSUserDefaults not modified synchronize seconds:%f", [end timeIntervalSinceDate:begin]);
}

{
    NSDate *begin = [NSDate date];
    for (NSInteger i = 0; i < 1000; ++i) {
        [[NSUserDefaults standardUserDefaults] setBool:(i%2==1) forKey:@"key"];
        [[NSUserDefaults standardUserDefaults] synchronize];
    }
    NSDate *end = [NSDate date];
    NSLog(@"NSUserDefaults modified synchronize (memory not change) seconds:%f", [end timeIntervalSinceDate:begin]);
}

The output is:

CCMmapUserSettings modified synchronize seconds:0.037747
NSUserDefaults not modified synchronize seconds:0.479931
NSUserDefaults modified synchronize (memory not change) seconds:0.402940

It shows that CCMmapUserSettings runs faster than NSUserDefaults!!!

I'm not sure

CCMmapUserSettings passes the unit settings on my iPhone6 (iOS 10.1.1), but I really not sure if it works on all iOS versions because I haven't gotten a official document to make sure the memory used to map the file will be written back to disk immediately when the process exits, if it's not, will it be written to disk before the device shuts down ?

I think I have to study the system behavior about mmap, if anyone of you knows that, please share. Thanks very much.

KudoCC
  • 6,912
  • 1
  • 24
  • 53
  • FWIW, calling -synchronize on NSUserDefaults is generally unnecessary on iOS 8 and later. Doing so will only slow your program down (sometimes not a bad thing, if you're about to call exit() and need to wait for the data to be safely out of process). Using mmap() is dangerous if the underlying volume can be unmounted, if atomicity is needed, if a kernel panic occurs, or if other processes can mess with the file. Use care :) – Catfish_Man Dec 19 '16 at 06:49
  • If only one process can access the file as it only runs in iOS platform and only one thread does the `mmap` and modifies the memory, is it still dangerous? I really like it because it runs so fast!!! – KudoCC Dec 19 '16 at 07:12
  • You still have some risks if the system crashes in the middle of writing the data, but yes, if you can control the usage carefully like that, the risks of using mmap are much much smaller. The typical pattern to make writes robust against system crashes is to use mkstemp() to make a temporary file, write to that, fsync, and then rename() it over the original file. Sadly as you've seen, that's slower. – Catfish_Man Dec 19 '16 at 07:18